1、Shanghai Jiao Tong UniversityM.A. Thesisproduces significantly less correct answers, with the other two procedures B (QSP) andC (AOP) falling between. Questionnaire responses suggest that subjects believe thatquestion preview can supply extra information and thus facilitate their listeningcomprehens

2、ion. The second experiment, an immediate retrospection study, attempts toprovide qualitative data by examining other 12 Chinese EFL learners listeningstrategies and tactics in completing the listening test. It is found that there are nosignificant differences between the four test procedures in test

3、 validity.This paper concludes that the choice of different test procedures leads to differencesin test difficulty, but not in test validity. Test takers can benefit from previewingquestion stems since the previewed question stems provide extra useful information;whereas most test takers suffer from

4、 the absence of both question stems and answeroptions before listening. The researcher recommends that the procedure allowing onlyquestion stem preview should be widely adopted.KEY WORDS: question preview, multiple choice question, listening comprehension,difficulty， validityIIIShanghai Jiao Tong Un

5、iversityM.A. ThesisList of AcronymsCET-4: College English Test Band 4CET-6: College English Test Band 6TOEFL: Test of English as a Foreign LanguageIELTS: International English Language Testing SystemIBT: Internet-Based TestsEFL: English as a foreign languageSLA: Second language acquisitionVOA: Voice

6、 of AmericaMC: Multiple choiceSAQ: Short answer questionMCQ: Multiple choice questionFQP: Full question previewAOP: Answer option previewQSP: Question stem previewNP: No previewLUS: language understanding systemGPS: general problem-solverVShanghai Jiao Tong UniversityM.A. ThesisList of TablesTable 1

7、.1 Test procedures adopted in the four versions of listening testTable 3.1 Previous listening comprehension scores of the four groups in experiment 1Table 3.2 Description of four VOA news reports in the listening testTable 3.3 Previous listening test scores of the 12 participants in experiment 2Tabl

8、e 4.1 Test scores of the four groups of students on the 4 textsTable 4.2 Multiple comparisons of the total scores in the MCQ listening test across thefour groupsTable 4.3 The descriptive statistics of the test scores of the four textsTable 4.4 Test scores across the four test procedures and textsTab

9、le 4.5 The mean and standard deviation of the four MCQ test proceduresTable 4.6 Multiple comparisons of test difficulty of the four MCQ test proceduresTable 4.7 Item analysis result of the listening comprehension testTable 4.8 Studentssubjective judgment on their familiarity with four test procedure

10、sTable 4.9 Comparison of familiarity level with the four test proceduresTable 4.10 Studentssubjective judgment on the validity of four test proceduresTable 4.11 Comparison of studentsjudgment on the validity of the four test procedures Studentsrating of the relative difficulty of four test procedure

11、sTable 4.13 Comparison of students rating of relative difficulty of four test proceduresTable 4.14 Rank order of studentsjudgment on four test proceduresTable 4.15 Test scores of participants across the four groups and textsTable 4.16 Taxonomy of listening strategies used for codingTable 4.17 Furthe

12、r observations on frequencies in four test proceduresTable 4.18 Observed frequencies of valid and invalid test items in four test proceduresVI上海交通大學(xué)學(xué)位論文原創(chuàng)性聲明本人鄭重聲明：所呈交的學(xué)位論文，是本人在導(dǎo)師的指導(dǎo)下，獨立進行研究工作所取得的成果。除文中已經(jīng)注明引用的內(nèi)容外，本論文不包含任何其他個人或集體已經(jīng)發(fā)表或撰寫過的作品成果。對本文的研究做出重要奉獻的個人和集體，均已在文中以明確方式標明。本人完全意識到本聲明的法律結(jié)果由本人承當。學(xué)位論文作

13、者簽名：劉淑敏日期：年月日上海交通大學(xué)學(xué)位論文版權(quán)使用授權(quán)書本學(xué)位論文作者完全了解學(xué)校有關(guān)保存、使用學(xué)位論文的規(guī)定，同意學(xué)校保存并向國家有關(guān)部門或機構(gòu)送交論文的復(fù)印件和電子版，允許論文被查閱和借閱。本人授權(quán)上海交通大學(xué)可以將本學(xué)位論文的全部或局部內(nèi)容編入有關(guān)數(shù)據(jù)庫進行檢索，可以采用影印、縮印或掃描等復(fù)制手段保存和匯編本學(xué)位論文。學(xué)位論文作者簽名：劉淑指導(dǎo)教師簽名：潘之欣日期：年月日日期：年月日Shanghai Jiao Tong UniversityM.A. ThesisAcknowledgementsFirst and foremost, I cannot sufficiently expr

14、ess my most sincere gratitude to mysupervisor, Professor Pan Zhixin, for her solicitude and thoughtprovoking talksthroughout my postgraduate study and especially for her enlightening instructions in mythesis writing. Her inspiration, encouragement, and guidance will have remarkableinfluence on my en

15、tire career in the future.I would like to express my gratitude to all the professors who have given thelectures and guided me into the investigation of English language during mypostgraduate study in Shanghai Jiao Tong University.I also wish to extend my thanks to all the teachers and students who t

16、ook thelistening comprehension test in this study with great collaboration.My sincere thanks also go to my classmates and my friends who have offered theirselfless assistance and generous encouragement, especially Zhao Lulu, Wang Yanyan,who helped me a lot in data collection in the present study.Las

17、t but not least, a special note of appreciation goes to my boyfriend, whosesupport has always been the source of my strength.IShanghai Jiao Tong UniversityM.A. ThesisChapter 1 Introduction1.1 Background of the present studyAs is known to all, College English Test (CET), Test of English as a ForeignL

18、anguage (TOEFL) and International English Language Testing System (IELTS), havealmost the largest population of test takers around the world. All the three language testsadopt Multiple Choice Question (MCQ) test format in the part of or the whole listeningcomprehension test; however, test procedures

19、 of MCQ vary one from another.For a long time, only two forms of MCQ test are adopted, which include theanswer option preview (AOP) form and full question preview form (FQP). Nowadays,with the increasingly widespread use of computer, internet-based test begins to beadministered in EFL testing, which

20、 provides more options for MCQ test.For example, in 2021, CET utilized the internet-based test in some pilot testingplaces, which adopts some different practices from that of the traditional CET. In theinternet-based CET test, test takers are able to preview the question stems and answeroptions on t

21、he computer screen first, and then listen to the passage and finish all thequestions. However, the traditional CET, which is a paper-and-pencil test, has all theanswer options printed out on the test paper. In the listening comprehension part, thetest takers finish listening to the whole passage fir

22、st, and then hear the questionspresented on the tape. During the 15 second pause after each question, they answerquestions by choosing from the options the one that is most appropriate.TOEFL and IELTS are widely used as international tests of English skills. Formore than 40 years, TOEFL has been reg

23、arded as the leading academic Englishproficiency test around the world and it measures the English proficiency of nonnativespeakers of North American English in reading, listening and writing. Most people whoare planning to study at colleges and universities where instruction is in English arerequir

24、ed to take the TOEFL test. The old TOEFL, which is paper-and-pencil test, isquite similar to the traditional CET; whereas the new TOEFL, which is internet-basedtest, is administered on computer. And test takers are allowed to preview neitherquestions nor answer options. They need to finish listening

25、 to the whole passage first,1Shanghai Jiao Tong UniversityM.A. Thesisand then read and finish the questions presented on the computer screen.IELTS is designed to assess the English language ability of non-native speakers inlistening, reading, writing and speaking. It is recognized by universities an

26、d employersin many countries, including Australia, Canada, New Zealand, the UK and the USA.Additionally, it is also recognized by professional institutions, immigration authoritiesand other government agencies. Practices in the old and new IELTS remain almost thesame in that full question preview is

27、 presented to test takers. That is, test takers areallowed to preview both the question stems and answer options first, and the listeningtext is presented afterwards. The difference is that in new IELTS, test takers are given20-30 seconds in advance to preview both the question stems and answer opti

28、ons.1.2 Research purposeThe diversity of MCQ test procedures in listening comprehension test leads us towonder: Do different orders in which question stems, answer options and listening textsare presented lead to any significant differences in test difficulty and test validity inMCQ listening compre

29、hension test?By systematically combining the three basic elements of MCQ, that is, questionstems, answer options and listening texts, four plausible MCQ test procedures arefinally obtained. The four MCQ test procedures constitute:Test procedure A (FQP) allowing test takers to preview the full questi

30、on, that is,both the question stems and answer options;Test procedure B (QSP) allowing test takers to preview only the question stems,but not the answer options;Test procedure C (AOP) allowing test takers to preview only answer options, butnot question stems;Test procedure D (NP) allowing test taker

31、s to preview neither question stems noranswer options.The present study conducts this investigation on the effects of the four differentMCQ test procedures on the difficulty and validity of a listening comprehension test.With the aim in mind, the study aims to answer the following two research quest

32、ions:Research question 1: Do the four different MCQ test procedures lead to anydifferences in test difficulty in MCQ listening comprehension test?Research question 2: Do the four different MCQ test procedures lead to any2Shanghai Jiao Tong UniversityM.A. Thesisdifferences in test validity in MCQ lis

33、tening comprehension test?1.3 Research methodologyThe present study is an experimental study in which two independent experimentsare carried out to collect both the quantitative and qualitative data respectively.In the first experiment, a MCQ listening comprehension test is conducted to 12randomly s

34、elected classes at Shanghai Jiao Tong University each of which has about 40students. There are in total 467 students who are non-English major freshmen. Based ontheir listening comprehension scores in the final exam of the previous semester, all the12 classes are assigned to four parallel groups. It

35、s indicated that there are no significantdifferences among the four groups. Their language proficiency level ranges fromlower-intermediate to higher-intermediate.First of all, a MCQ listening comprehension test paper is prepared, which entailsfour parts based on four different VOA news reports respe

36、ctively. There are five MCquestions in each part and each MC question is assigned one point. Therefore, the totalscore of the whole listening test paper is 20 points. Then, four versions of the samelistening comprehension test are prepared, each of which is used among one group ofstudents. Each one

37、of the four test versions entails the same four well-selected VOAnews reports respectively and these four VOA news reports appear in the same orderacross all the four test versions. In addition, each one of the four test versions includesall the four different MCQ test procedures (procedure A, B, C

38、and D); however, theorder in which the four test procedures arranged varies one test version from another.The purpose of doing this is to make sure that all the four VOA news reports are testedin different test procedures in the four groups of students and all the four test proceduresare tried by ea

39、ch one of the 467 students (See Table 1.1). The form of four test versionsis shown in Table 1.1.Table 1.1 Test procedures adopted in the four versions of listening test3Test VersionText1Test VersionText1Text2Text3Text4IABCDIIBADCIIICDABIVDCBAShanghai Jiao Tong UniversityM.A. ThesisIn addition, at th

40、e end of each part, a questionnaire is attached to collect studentscomments on the four test procedures. Question 1 is aimed at investigating studentsjudgment on their familiarity with the four different test procedures; question 2 and 3 isaimed at explore students judgment on test validity and test

41、 difficulty of the four testprocedures respectively. After the experiment, scores of the four groups will becompared and subjective judgments from questionnaires will be analyzed.The second experiment is an immediate retrospection study, which is designed toinvestigate the effects of four test proce

42、dures on test validity. This experiment isconducted among 12 participants who are, according to the final exam listening scoresof previous semester, assigned to four parallel groups. The language proficiency level ofthe 12 participants ranges from lower- intermediate to higher- intermediate. For eac

43、hgroup of participants, one different listening comprehension test version among the fourversions is adopted. All the participants take part in the listening comprehension testindividually. They report their thought processes of reaching the answer to eachquestion after they finish the 5 questions b

44、ased on each listening texts. All the recallprotocols are recorded, transcribed, coded and analyzed. Based on their recall protocols,listening strategies they use are classified into valid and invalid ones. Frequencycalculation is conducted to compare test validity of the four test procedures.1.4 Or

45、ganization of the thesisThe thesis is divided into five chapters.Chapter one gives a brief introduction to the research background, researchpurpose of the thesis, presenting a general layout. By reviewing the diverse practices inseveral large-scale standardized language tests, research questions of

46、the present studyare put forward.Chapter two is devoted to a comprehensive review of theories and practices in thetest of listening comprehension. In the former part, a theoretical foundation is given byintroducing theories such as the nature of listening comprehension, informationprocessing and thr

47、ee different models of listening comprehension. In addition, testprocedure effects on test takersperformance are reviewed and introduced, especiallyfactors affecting test difficulty and validity. In the latter part, summaries and briefcomments on previous related studies concerning MCQ test format e

48、ffect on test takers4Shanghai Jiao Tong UniversityM.A. Thesisperformance are provided. And studies on the question preview effects are the mainfocus.Chapter three aims to address the design of the present study, giving a descriptionof the experiments. Detailed information is provided about the subje

49、cts, materials andimplementation in the two experiments.Chapter four describes, analyzes and discusses the data collected from theexperiments. By using tables and figures, test results are, first of all, displayed andillustrated to answer the research questions posed at the very outset and to seek f

50、orexplanations for the major results.Chapter five is the conclusion, showing the implications and limitations of thepresent study. Practical suggestions are also given for future research.5Shanghai Jiao Tong UniversityM.A. ThesisChapter 2 Literature ReviewThis chapter lays the very theoretical found

51、ation for the later experimental research,including theory on the nature of listening comprehension, theory in language test andrelated studies on MCQ question preview effects.2.1 Understanding listening comprehension2.1.1 The nature of listening comprehensionThe importance of listening has led to a

52、bundant studies on the nature of listeningcomprehension. Traditionally, listening and reading are considered as sharing a numberof similarities. However, compared to reading, listening is less studied. As Weir (1993:9)claimed, “Much of the current thinking on the nature of listening comprehension is

53、based on earlier work of reading comprehension.For a long period of time, under the influence of behaviorism, listening wasregarded as a receptive language skill. Some researchers claimed that listening was “aninevitable by-product of speech(Temple and Gillet, 1989:53) and they believed thatlistenin

54、g was a passive and static receptive process, in which a listener functioned like atape recorder, whose main task while listening was simply to memorize what thespeaker had said. From their point of view, the main characteristic of listeners waspassiveness. Therefore, during listening activities, li

55、steners were regarded as passivelyassimilating the incoming messages presented by speakers. Under the guidance of thispoint of view, the teaching of listening paid more attention to the teachersstimuli andthe form rather than the meaning of listening materials and the role of listening wasmerely to

56、reinforce recognition of the words heard.In recent years, with the development of cognitive theories, people begin to gain abetter understanding of the nature of listening comprehension. In contrast withbehaviorists, cognitive researchers hold that listening comprehension involves more6Shanghai Jiao

57、 Tong UniversityM.A. Thesisthan simply perceiving aural stimuli, and listeners are considered as active rather thanpassive. More specifically, listening comprehension is now viewed as an active process,in which listeners focus on selected aspects of aural input, construct meaning frompassages, and r

58、elate what they hear to existing knowledge (OMalley & Chamot, 1990).According to Vandergrift (1999:168), “l(fā)istening comprehension is a complex,active process in which the listener must discriminate between sounds, understandvocabulary and grammatical structures, interpret stress and intonation, reta

59、in what isgathered in all of the above, and interpret it within the immediate as well as the largersocial structural context of the utterance. Coordinating all of this involves a great deal ofmental activity on the part of the listener. In the similar vein, Buck (1990:95) stated,“l(fā)istening comprehen

60、sion is a massively parallel interactive process taking advantage ofinformation from a large number of sources, both linguistic and non- linguistic.It isobvious that the cognitive point of view provides a more plausible explanation of thenature of listening comprehension. And it reveals that compreh