Rec.stylerefhrefITU-RBS.1534 PAGE17RECOMMENDATIONITU-RBS.1534Methodforthesubjectiveassessmentofintermediatequalitylevel

of

coding

systems(QuestionITU-R220/10)TheITURadiocommunicationAssembly,consideringa) thatRecommendationsITU-RBS.1116,ITU-RBS.1284,ITU-RBT.500,ITU-RBT.710andITU-RBT.811aswellasITU-TRecommendationsP.800,P.810andP.830,haveestablishedmethodsforassessingsubjectivequalityofaudio,videoandspeechsystems;b) thatnewkindsofdeliveryservicessuchasstreamingaudioontheInternetorsolidstateplayers,digitalsatelliteservices,digitalshortandmediumwavesystemsormobilemultimediaapplicationsmayoperateatintermediateaudioquality;c) thatRecommendationITU-RBS.1116isintendedfortheassessmentofsmallimpairmentsandisnotsuitableforassessingsystemswithintermediateaudioquality;d) thatRecommendationITU-RBS.1284givesnoabsolutescoringfortheassessmentofintermediateaudioquality;e) thatITU-TRecommendationsP.800,P.810andP.830arefocusedonspeechsignalsina

telephoneenvironmentandprovedtobenotsufficientfortheevaluationofaudiosignalsina

broadcastingenvironment;f) thattheuseofstandardizedsubjectivetestmethodsisimportantfortheexchange,compatibilityandcorrectevaluationofthetestdata;g) thatnewmultimediaservicesmayrequirecombinedassessmentofaudioandvideoquality,recommends1 thatthetestingandevaluationproceduresgiveninAnnex1ofthisRecommendationbeusedforthesubjectiveassessmentofintermediateaudioquality.ANNEX11 IntroductionThisRecommendationdescribesanewmethodforthesubjectiveassessmentofintermediateaudioquality.ThismethodmirrorsmanyaspectsofRecommendationITU-RBS.1116andusesthesamegradingscaleasisusedfortheevaluationofpicturequality(i.e.RecommendationITUR

BT.500).

Themethod,called“MUltiStimulustestwithHiddenReferenceandAnchor(MUSHRA)”,hasbeensuccessfullytested.ThesetestshavedemonstratedthattheMUSHRAmethodissuitableforevaluationofintermediateaudioqualityandgivesaccurateandreliableresults,[EBU,2000a;SoulodreandLavoie,1999;EBU,2000b].ThisRecommendationincludesthefollowingsectionsandAppendix:Section1: IntroductionSection2: Scope,testmotivationandpurposeofnewmethodSection3: ExperimentaldesignSection4: SelectionofsubjectsSection5: TestmethodSection6: AttributesSection7: TestmaterialSection8: ListeningconditionsSection9: StatisticalanalysisSection10: TestreportandpresentationofresultsAppendix1: Instructionstobegiventosubjects.2 Scope,testmotivationandpurposeofnewmethodSubjectivelisteningtestsarerecognizedasstillbeingthemostreliablewayofmeasuringthequalityofaudiosystems.Therearewelldescribedandprovenmethodsforassessingaudioqualityatthetopandthebottomqualityrange.RecommendationITU-RBS.1116–Methodsforthesubjectiveassesmentofsmallimpairmentsinaudiosystemsincludingmultichannelsoundsystems,isusedfortheevaluationofhighqualityaudiosystemshavingsmallimpairments.However,thereareapplicationswherelowerqualityaudioisacceptableorunavoidable.RapiddevelopmentsintheuseoftheInternetfordistributionandbroadcastofaudiomaterial,wherethedatarateislimited,haveledtoacompromiseinaudioquality.OtherapplicationsthatmaycontainintermediateaudioqualityaredigitalAM(i.e.digitalradiomondiale(DRM),digitalsatellitebroadcasting,commentarycircuitsinradioandTV,audioondemandservicesandaudioondial-uplines.ThetestmethoddefinedinRecommendationITUR

BS.1116isnotentirelysuitableforevaluatingtheselowerqualityaudiosystems[SoulodreandLavoie,1999]becauseitispooratdiscriminatingbetweensmalldifferencesinqualityatthebottomofthescale.RecommendationITU-RBS.1284givesonlymethodswhicharededicatedeithertothehighqualityaudiorangeorgivesnoabsolutescoringofaudioquality.OtherRecommendations,likeITU-TRecommendationsP.800,P.810orP.830,arefocusedonsubjectiveassessmentofspeechsignalsinatelephoneenvironment.TheEuropeanBroadcastingUnion(EBU)ProjectGroupB/AIMhasdoneexperimentswithtypicalaudiomaterialasusedinabroadcastingenvironmentusingtheseITU-Tmethods.Noneofthesemethodsfulfilsthe

requirementforanabsolutescale,comparisonwithareferencesignalandsmallconfidenceintervalswithareasonablenumberofsubjectsatthesametime.Thereforetheevaluationofaudiosignalsinabroadcastingenvironmentcannotbedoneproperlybyusingoneofthesemethods.ThenewtestmethoddescribedinthisRecommendationisintendedtogiveareliableandrepeatablemeasureofsystemshavingaudioqualitywhichwouldnormallyfallinthelowerhalfoftheimpairmentscaleusedbyRecommendationITURBS.1116[EBU,2000a;SoulodreandLavoie,1999;EBU,2000b].IntheMUSHRAtestmethod,ahighqualityreferencesignalisusedandthesystemsundertestareexpectedtointroducesignificantimpairments.Ifthesystemsundertestcanimprovethesubjectivequalityofasignalthenothertestmethodsshouldbeused.3 ExperimentaldesignManydifferentkindsofresearchstrategiesareusedingatheringreliableinformationinadomainofscientificinterest.Inthesubjectiveassessmentofimpairmentsinaudiosystems,themostformalexperimentalmethodsshallbeused.Subjectiveexperimentsarecharacterizedfirstlybyactualcontrolandmanipulationoftheexperimentalconditions,andsecondlybycollectionandanalysisofstatisticaldatafromlisteners.Carefulexperimentaldesignandplanningisneededtoensurethatuncontrolledfactorswhichcancauseambiguityintestresultsareminimized.Asanexample,iftheactualsequenceofaudioitemswereidenticalforallthesubjectsinalisteningtest,thenonecouldnotbesurewhetherthejudgementsmadebythesubjectswereduetothatsequenceratherthantothedifferentlevelsofimpairmentsthatwerepresented.Accordingly,thetestconditionsmustbearrangedinawaythatrevealstheeffectsoftheindependentfactors,andonlyofthesefactors.Insituationswhereitcanbeexpectedthatthepotentialimpairmentsandothercharacteristicswillbedistributedhomogeneouslythroughoutthelisteningtest,atruerandomizationcanbeappliedtothepresentationofthetestconditions.Wherenon-homogeneityisexpectedthismustbetakenintoaccountinthepresentationofthetestconditions.Forexample,wherematerialtobeassessedvariesinlevelofdifficulty,theorderofpresentationofstimulimustbedistributedrandomly,bothwithinandbetweensessions.Listeningtestsneedtobedesignedsothatsubjectsarenotoverloadedtothepointoflessenedaccuracyofjudgement.Exceptincaseswheretherelationshipbetweensoundandvisionisimportant,itispreferredthattheassessmentofaudiosystemsiscarriedoutwithoutaccompanyingpictures.Amajorconsiderationistheinclusionofappropriatecontrolconditions.Typically,controlconditionsincludethepresentationofunimpairedaudiomaterials,introducedinwaysthatareunpredictabletothesubjects.Itisthedifferencesbetweenjudgementofthesecontrolstimuliandthepotentiallyimpairedonesthatallowsonetoconcludethatthegradesareactualassessmentsoftheimpairments.Someoftheseconsiderationswillbedescribedlater.Itshouldbeunderstoodthatthetopicsofexperimentaldesign,experimentalexecution,andstatisticalanalysisarecomplex,andthatnotalldetailscanbegiveninaRecommendationsuchasthis.Itisrecommendedthatprofessionalswithexpertiseinexperimentaldesignandstatisticsshouldbeconsultedorbroughtinatthebeginningoftheplanningforthelisteningtest.

4 SelectionofsubjectsDatafromlisteningtestsassessingsmallimpairmentsinaudiosystems,asinRecommendationITURBS.1116,shouldcomefromsubjectswhohaveexperienceindetectingthesesmallimpairments.Thehigherthequalityreachedbythesystemstobetested,themoreimportantitistohaveexperiencedlisteners.4.1 CriteriaforselectingsubjectsAlthoughtheMUSHRAtestmethodisnotintendedtobeappliedtosmallimpairments,itisstillrecommendedthatexperiencedlistenersshouldbeused.Theselistenersshouldhaveexperienceinlisteningtosoundinacriticalway.Suchlistenerswillgiveamorereliableresultmorequicklythannon-experiencedlisteners.Itisalsoimportanttonotethatmostnon-experiencedlistenerstendtobecomemoresensitivetothevarioustypesofartefactsafterfrequentexposure.Thereissometimesareasonforintroducingarejectiontechniqueeitherbefore(pre-screening)orafter(postscreening)therealtest.Insomecasesbothtypesofrejectionsmightbeused.Here,rejectionisaprocesswherealljudgementsfromaparticularsubjectareomitted.Anytypeofrejectiontechnique,notcarefullyanalysedandapplied,mayleadtoabiasedresult.Itisthusextremelyimportantthat,whenevereliminationofdatahasbeenmade,thetestreportclearlydescribesthecriterionapplied.4.1.1 Pre-screeningofsubjectsThelisteningpanelshouldbecomposedofexperiencedlisteners,inotherwords,peoplewhounderstandandhavebeenproperlytrainedinthedescribedmethodofsubjectivequalityevaluation.Theselistenersshould: haveexperienceinlisteningtosoundinacriticalway; havenormalhearing(ISOStandard389shouldbeusedasaguideline).Thetrainingproceduremightbeusedasatoolforpre-screening.Themajorargumentforintroducingapre-screeningtechniqueistoincreasetheefficiencyofthelisteningtest.Thismusthoweverbebalancedagainsttheriskoflimitingtherelevanceoftheresulttoomuch.4.1.2 Post-screeningofsubjectsPost-screeningmethodscanberoughlyseparatedintoatleasttwoclasses: oneisbasedontheabilityofthesubjecttomakeconsistentrepeatedgradings; theotherreliesoninconsistenciesofanindividualgradingcomparedwiththemeanresultofallsubjectsforagivenitem.Itisrecommendedtolooktotheindividualspreadandtothedeviationfromthemeangradingofallsubjects.Theaimofthisistogetafairassessmentofthequalityofthetestitems.Iffewsubjectsuseeitherextremeendofthescale(excellent,bad)andthemajorityareconcentratedatanotherpointonthescale,thesesubjectscouldberecognizedasoutliersandmightberejected.

Duetothefactthat“intermediatequality”istested,asubjectshouldbeabletoidentifythecodedversionveryeasilyandthereforefindagradewhichisintherangeofthemajorityofthesubjects.Subjectswithgradesattheupperendofthescalearelikelytobelesscriticalandsubjectswhohavegradesonlyatthelowestendofthescalearelikelytobetoocritical.Byrejectingtheseextremesubjectsamorerealisticqualityassessmentisexpected.Themethodsareprimarilyusedtoeliminatesubjectswhocannotmaketheappropriatediscriminations.Theapplicationofapost-screeningmethodmayclarifythetendenciesinatestresult.However,bearinginmindthevariabilityofsubjects’sensitivitiestodifferentartefacts,cautionshouldbeexercised.Byincreasingthesizeofthelisteningpanel,theeffectsofanyindividualsubject’sgradeswillbereducedandsotheneedtorejectasubject’sdataisgreatlydiminished.4.2 SizeoflisteningpanelTheadequatesizeforalisteningpanelcanbedeterminedifthevarianceofgradesgivenbydifferentsubjectscanbeestimatedandtherequiredresolutionoftheexperimentisknown.Wheretheconditionsofalisteningtestaretightlycontrolledonboththetechnicalandbehaviouralside,experiencehasshownthatdatafromnomorethan20subjectsareoftensufficientfordrawingappropriateconclusionsfromthetest.Ifanalysiscanbecarriedoutasthetestproceeds,thennofurthersubjectsneedtobeprocessedwhenanadequatelevelofstatisticalsignificancefordrawingappropriateconclusionsfromthetesthasbeenreached.If,foranyreason,tightexperimentalcontrolcannotbeachieved,thenlargernumbersofsubjectsmightbeneededtoattaintherequiredresolution.Thesizeofalisteningpanelisnotsolelyaconsiderationofthedesiredresolution.TheresultfromthetypeofexperimentdealtwithinthisRecommendationis,inprinciple,onlyvalidforpreciselythatgroupofexperiencedlistenersactuallyinvolvedinthetest.Thus,byincreasingthesizeofthelisteningpaneltheresultcanbeclaimedtoholdforamoregeneralgroupofexperiencedlistenersandmaythereforesometimesbeconsideredmoreconvincing.Thesizeofthelisteningpanelmayalsoneedtobeincreasedtoallowfortheprobabilitythatsubjectsvaryintheirsensitivitytodifferentartefacts.5 TestmethodTheMUSHRAtestmethodusestheoriginalunprocessedprogrammematerialwithfullbandwidthasthereferencesignal(whichisalsousedasahiddenreference)aswellasatleastonehiddenanchor.Additionalwell-definedanchorscanbeusedasdescribedinSection DescriptionoftestsignalsThelengthofthesequencesshouldtypicallynotexceed20stoavoidfatiguingoflistenersandtoreducethetotaldurationofthelisteningtest.

Thesetofprocessedsignalsconsistsofallthesignalsundertestandatleastoneadditionalsignal(anchor)beingalow-passfilteredversionoftheunprocessedsignal.Thebandwidthofthisadditionalsignalshouldbe3.5kHz.Dependingonthecontextofthetest,additionalanchorscanbeusedoptionally.Othertypesofanchorsshowingsimilartypesofimpairmentsasthesystemsundertestcanbeused.Thesetypesofimpairmentscaninclude: bandwidthlimitationof7kHzor10

kHz; reducedstereoimage; additionalnoise; dropouts; packetlosses; andothers.NOTE

1

–

ThebandwidthsoftheanchorscorrespondtotheRecommendationsforcontrolcircuits(3.5

kHz),usedforsupervisionandcoordinationpurposeinbroadcasting,commentarycircuits(7

kHz)andoccasionalcircuits(10

kHz),accordingtoITUTRecommendationsG.711,G.712,G.722andJ.21,respectively.Thecharacteristicofthe3.5kHzlow-passfiltershouldbeasfollows:fc3.5kHzMaximumpassbandripple0.1dBMinimumattenuationat4

kHz=25dBMinimumattenuationat4.5kHz=50dB.Theadditionalanchorsareintendedtoprovideanindicationofhowthesystemsundertestcomparetowell-knownaudioqualitylevelsandshouldnotbeusedforrescalingresultsbetweendifferenttests.5.2 TrainingphaseInordertogetreliableresults,itismandatorytotrainthesubjectsinspecialtrainingsessionsinadvanceofthetest.Thistraininghasbeenfoundtobeimportantforobtainingreliableresults.Thetrainingshouldatleastexposethesubjecttothefullrangeandnatureofimpairmentsandalltestsignalsthatwillbeexperiencedduringthetest.Thismaybeachievedusingseveralmethods:a

simpletapeplaybacksystemoraninteractivecomputer-controlledsystem.InstructionsaregiveninAppendix1.5.3 PresentationofstimuliMUSHRAisadouble-blindmulti-stimulustestmethodwithhiddenreferenceandhiddenanchor(s),whereasRecommendationITU-RBS.1116usesa“doubleblindtriple-stimuluswithhiddenreference”testmethod.TheMUSHRAapproachisfelttobemoreappropriateforevaluatingmediumandlargeimpairments

[SoulodreandLavoie,1999].Inatestinvolvingsmallimpairments,thedifficulttaskforthesubjectistodetectanyartefactswhichmightbepresentinthesignal.Inthissituationahiddenreferencesignalisnecessaryinthetestinordertoallowtheexperimentertoevaluatethesubject’sabilitytosuccessfullydetectthese

artefacts.Conversely,inatestwithmediumandlargeimpairments,thesubjecthasnodifficultyindetectingtheartefactsandthereforeahiddenreferenceisnotnecessaryforthispurpose.Rather,thedifficultyariseswhenthesubjectmustgradetherelativeannoyancesofthevariousartefacts.Herethesubjectmustweighhispreferenceforonetypeofartefactversussomeothertypeofartefact.Theuseofahighqualityreferenceintroducesaninterestingproblem.Sincethenewmethodologyistobeusedforevaluatingmediumandlargeimpairments,theperceptualdifferencefromthereferencesignaltothetestitemsisexpectedtoberelativelylarge.Conversely,theperceptualdifferencesbetweenthetestitemsbelongingtodifferentsystemsmaybequitesmall.Asaresult,ifamulti-trialtestmethod(suchasisusedinRecommendationITU-RBS.1116)isused,itmaybeverydifficultforsubjectstoaccuratelydiscriminatebetweenthevariousimpairedsignals.Forexample,inadirectpairedcomparisontestsubjectsmightagreethatSystemAisbetterthanSystemB.However,inasituationwhereeachsystemisonlycomparedwiththereferencesignal(i.e.SystemAandSystemBarenotdirectlycomparedtoeachother),thedifferencesbetweenthetwosystemsmaybelost.Toovercomethisdifficulty,intheMUSHRAtestmethod,thesubjectcanswitchatwillbetweenthereferencesignalandanyofthesystemsundertest,typicallyusingacomputer-controlledreplaysystem,althoughothermechanismsusingmultipleCDortapemachinescanbeused.Thesubjectispresentedwithasequenceoftrials.Ineachtrialthesubjectispresentedwiththereferenceversionaswellasallversionsofthetestsignalprocessedbythesystemsundertest.Forexample,ifatestcontains8audiosystems,thenthesubjectisallowedtoswitchinstantlyamongthe11signals(1

reference8impaired1hiddenreference1hiddenanchor).Becausethesubjectcandirectlycomparetheimpairedsignals,thismethodprovidesthebenefitsofafullpairedcomparisontestinthatthesubjectcanmoreeasilydetectdifferencesbetweentheimpairedsignalsandgradethemaccordingly.Thisfeaturepermitsahighdegreeofresolutioninthegradesgiventothesystems.Itisimportanttonotehowever,thatsubjectswillderivetheirgradeforagivensystembycomparingthatsystemtothereferencesignal,aswellastotheothersignalsineachtrial.Itisrecommendedthatnomorethan15signals(e.g.12systemsundertest,1knownreference,1

hiddenanchor,and1hiddenreference)shouldbeincludedinanytrial.InaRecommendationITU-RBS.1116test,subjectstendtoapproachagiventrialbystartingwithadetectionprocess,followedbyagradingprocess.TheexperiencefromconductingtestsaccordingtotheMUSHRAmethodshows,thatsubjectstendtobeginasessionwitharoughestimationofthequality.Thisisfollowedbyasortingorrankingprocess.Afterthatthesubjectperformsthegradingprocess.Sincetherankingisdoneinadirectfashion,theresultsforintermediateaudioqualityarelikelytobemoreconsistentandreliablethaniftheRecommendationITU-RBS.1116methodhadbeenused.5.4 GradingprocessThesubjectsarerequiredtoscorethestimuliaccordingtothecontinuousqualityscale(CQS).TheCQSconsistsofidenticalgraphicalscales(typically10cmlongormore)whicharedividedintofiveequalintervalswiththeadjectivesasgiveninFig.1fromtoptobottom.

Thisscaleisalsousedforevaluationofpicturequality(RecommendationITU-RBT.500–Methodologyforthesubjectiveassessmentofthequalityattelevisionpictures).Thelistenerrecordshis/herassessmentofthequalityinasuitableform,forexample,withtheuseofslidersonanelectronicdisplay(seeFig.2),orusingapenandpaperscale.Thesubjectisaskedtoassessthequalityofallstimuli,accordingtothefiveintervalCQS.

ComparedtoRecommendationITU-RBS.1116,theMUSHRAmethodhastheadvantageofdisplayingallstimuliatthesametimesothatthesubjectisabletocarryoutanycomparisonbetweenthemdirectly.Theresultsbecomemoreconsistent,leadingtosmallerconfidenceintervals.ThetimetakentoperformthetestusingtheMUSHRAmethodcanbesignificantlylessthanwhenusingtheRecommendationITU-RBS.1116method.6 AttributesListedbelowareattributesspecifictomonophonic,stereophonicandmultichannelevaluations.Itispreferredthattheattribute“basicaudioquality”beevaluatedineachcase.Experimentersmaychoosetodefineandevaluateotherattributes.Onlyoneattributeshouldbegradedduringonetrial.Whensubjectsareaskedtoassessmorethanoneattributeineachtrialtheycanbecomeoverburdenedorconfused,orboth,bytryingtoanswermultiplequestionsaboutagivenstimulus.Thismightproduceunreliablegradingsforallthequestions.6.1 MonophonicsystemBasicaudioquality:Thissingle,globalattributeisusedtojudgeanyandalldetecteddifferencesbetweenthereferenceandtheobject.6.2 StereophonicsystemBasicaudioquality:Thissingle,globalattributeisusedtojudgeanyandalldetecteddifferencesbetweenthereferenceandtheobject.Thefollowingadditionalattributemaybeofinterest:Stereophonicimagequality:Thisattributeisrelatedtodifferencesbetweenthereferenceandtheobjectintermsofsoundimagelocationsandsensationsofdepthandrealityoftheaudioevent.Althoughsomestudieshaveshownthatstereophonicimagequalitycanbeimpaired,sufficientresearchhasnotyetbeendonetoindicatewhetheraseparateratingforstereophonicimagequalityasdistinctfrombasicaudioqualityiswarranted.NOTE

1

–

Upto1993,mostsmallimpairmentsubjectiveevaluationstudiesofstereophonicsystemshaveusedtheattributebasicaudioqualityexclusively.Thustheattributestereophonicimagequalitywaseitherimplicitlyorexplicitlyincludedwithinbasicaudioqualityasaglobalattributeinthosestudies.6.3 MultichannelsystemBasicaudioquality:Thissingle,globalattributeisusedtojudgeanyandalldetecteddifferencesbetweenthereferenceandtheobject.Thefollowingadditionalattributesmaybeofinterest:Frontimagequality:Thisattributeisrelatedtothelocalizationofthefrontalsoundsources.Itincludesstereophonicimagequalityandlossesofdefinition.Impressionofsurroundquality:Thisattributeisrelatedtospatialimpression,ambience,orspecialdirectionalsurroundeffects.

7 TestmaterialCriticalmaterialwhichrepresentsthetypicalbroadcastprogrammeforthedesiredapplicationshallbeusedinordertorevealdifferencesamongsystemsundertest.Criticalmaterialisthatwhichstressesthesystemsundertest.Thereisnouniversallysuitableprogrammematerialthatcanbeusedtoassessallsystemsunderallconditions.Accordingly,criticalprogrammematerialmustbesoughtexplicitlyforeachsystemtobetestedineachexperiment.Thesearchforsuitablematerialisusuallytime-consuming;however,unlesstrulycriticalmaterialisfoundforeachsystem,experimentswillfailtorevealdifferencesamongsystemsandwillbeinconclusive.Itmustbeempiricallyandstatisticallyshownthatanyfailuretofinddifferencesamongsystemsisnotduetoexperimentalinsensitivitywhichmaybecausedbypoorchoicesofaudiomaterial,oranyotherweakaspectsoftheexperiment.Otherwisethis“null”findingcannotbeacceptedasvalid.Inthesearchforcriticalmaterial,anystimulusthatcanbeconsideredaspotentialbroadcastmaterialshallbeallowed.Syntheticsignalsdeliberatelydesignedtobreakaspecificsystemshouldnotbeincluded.Theartisticorintellectualcontentofaprogrammesequenceshouldbeneithersoattractivenorsodisagreeableorwearisomethatthesubjectisdistractedfromfocusingonthedetectionofimpairments.Theexpectedfrequencyofoccurrenceofeachtypeofprogrammematerialinactualbroadcastsshouldbetakenintoaccount.However,itshouldbeunderstoodthatthenatureofbroadcastmaterialmightchangeintimewithfuturechangesinmusicalstylesandpreferences.Whenselectingtheprogrammematerial,itisimportantthattheattributeswhicharetobeassessedarepreciselydefined.Theresponsibilityofselectingmaterialshallbedelegatedtoagroupofskilledsubjectswithabasicknowledgeoftheimpairmentstobeexpected.Theirstartingpointshallbebasedonaverybroadrangeofmaterial.Therangecanbeextendedbydedicatedrecordings.Forthepurposeofpreparingfortheformalsubjectivetest,theloudnessofeachexcerptneedstobeadjustedsubjectivelybythegroupofskilledsubjectspriortorecordingitonthetestmedia.Thiswillallowsubsequentuseofthetestmediaatafixedgainsettingforallprogrammeitemswithina

testtrial.Foralltestsequencesthegroupofskilledsubjectsshallconveneandcometoaconsensusontherelativesoundlevelsoftheindividualtestexcerpts.Inaddition,theexpertsshouldcometoa

consensusontheabsolutereproducedsoundpressurelevelforthesequenceasawholerelativetothealignmentlevel.Atoneburst(forexample1kHz,300ms,–18dBFS)atalignmentsignallevelmaybeincludedattheheadofeachrecordingtoenableitsoutputalignmentleveltobeadjustedtotheinputalignmentlevelrequiredbythereproductionchannel,accordingtoEBURecommendationR

68(seeRecommendationITU-RBS.1116,§

8.4.1).Thetoneburstisonlyforalignmentpurposes:itshouldnotbereplayedduringthetest.ThesoundprogrammesignalshouldbecontrolledsothattheamplitudesofthepeaksonlyrarelyexceedthepeakamplitudeofthepermittedmaximumsignaldefinedinRecommendationITU-R

BS.645(asinewave9dBabovethealignmentlevel).Thefeasiblenumberofexcerptstoincludeinatestvaries:itshallbeequalforeachsystemundertest.Areasonableestimateis1.5timesthenumberofsystemsundertest,subjecttoaminimumvalueof5excerpts.Audioexcerptswillbetypically10sto20slong.Duetothecomplexityofthe

task,thesystemsundertestshouldbeavailable.Asuccessfulselectioncanonlybeachievedifanappropriatetimescheduleisdefined.Theperformanceofamultichannelsystemundertheconditionsoftwochannelplaybackshallbetestedusingareferencedown-mix.Althoughtheuseofafixeddown-mixmaybeconsideredtoberestrictinginsomecircumstances,itisundoubtedlythemostsensibleoptionforusebybroadcastersinthelongrun.Theequationsforthereferencedown-mix(seeRecommendationITU-RBS.775)are: Thepre-selectionofsuitabletestexcerptsforthecriticalevaluationoftheperformanceofreferencetwo-channeldown-mixshouldbebasedonthereproductionoftwo-channeldown-mixedprogrammematerial.8 ListeningconditionsMethodsfor