аЯрЁБс>ўџ '*џџџџ+џџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџьЅСq` С0 . Ѓ*******>>>>>8v$š ІюЊ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ ˜ h йюююююююююююююююююІММММММММММММММММММММММЙбббббббббббббббббййййййй                                                                 To: Administrative Committee From: Josef Glowa, FDAI I am sending you a summary of what we have discussed in our committee regarding the pros and cons of electronic student evaluations. Committee member Melanie Arthur deserves special credit for digging more deeply into this issue, and compiling and summarizing some crucial data (see below). Attached are the two articles that best illustrate the current unreliable state of online surveys of teaching. -------------------------------------------------------------------------------------------------- The first attached article compares modes of administration (paper vs online), finding a huge difference in response rates for in-class versus online evaluations (70% versus 29%, in the absence of special incentives for online evaluators). They included two additional groups of online evaluations, one in which the instructor provided an in-class demo of the online evaluation website and another in which a "modest" grade incentive was offered for completing the online evaluation. In the extra credit case, the response rates were comparable for paper and online evaluations. It should be noted, though, that no similar incentive was offered to the in-class students. Perhaps they could have gotten a response rate closer to 90% for the in-class evaluations if they had offered extra credit there? This was a single institution study of business classes and has not, to my knowledge, been replicated. The second attached article is a lengthier review of practices, emphasizing the quality of ratings by different modes of administration, and emphasizing the importance of response rates. It reviews studies that have compared in-person and online and finds a difference in response rates ranging from 37 to <1% lower, with an average difference of 23% (notably in the 1% case the in-person evaluation only had a 33% response and included only distance ed courses). The author describes strategies that have been used to raise response rates, but notes in summary that these have typically NOT been used. In their absence, online surveys can be expected to fall well below the acceptable response rate. (Of course, this does not engage the question of students' ability to effectively evaluate their instructors, which is outside the scope of this discussion) Acceptable response rates: Survey research methods texts (e.g. Babbie's Practice of Social Research) suggest that an absolute minimum response rate from which to draw valid conclusions is 50%, and 70% is desirable. There is precedent (described in the second attached article) for using a combination of paper and online within a single institution. I did not explore this explicitly on the IDEA website, but is worth looking into more deeply if the administration appears completely inflexible in their desire to switch to online evaluation. There is a lot more research out there on this topic, and both attached articles include lengthy bibliographies if you wish to dig deeper. These two pieces appeared to be good representatives of the available literature, in my judgment. Of note, UAA has not adopted an effective institution-wide strategy for improving response rates. Instead, they now write on their reports that the data cannot be considered valid for evaluative purposes in the case of low response rates. †œ@ёџ@NormalCJsH mH tH _HaJDAђџЁDDefault Paragraph FontdiѓџГd Table Normal.aіі4ж4жl4ж4жl (kєџС(No ListрўёџђрHeader & FooterM$$$ЄЄd№*$1$@& ]„^„`„$a$-DMЦ џџЦ$Y568:;<7S*CJOJPJQJ@ˆphsH mH H*>*KHEHfHqЪ џџЬўёџЬ Free FormE$$$ЄЄd№*$1$@& ]„^„`„$a$-DMЦ џџY568:;<7S*CJOJPJQJ@ˆphsH mH H*>*KHEHfHqЪ џџТўёџТBodyE$$$ЄЄd№*$1$@& ]„^„`„$a$-DMЦ џџY568:;<7S*CJOJPJQJ@ˆph) 0 00# $ АаА`Аа/ Ар=!А "А >rt` – ˜ ^ ` np"$а№раРА €p^N<,CJOJPJQJphsH mH "CJOJPJQJphsH mH EHCJOJPJQJphsH mH "CJOJPJQJphsH mH EHCJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH CJOJPJQJphsH mH  ав`b:<   юоЬМЊšˆCˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџ"CJOJPJQJphsH mH EHCJOJPJQJphsH mH "CJOJPJQJphsH mH EHCJOJPJQJphsH mH "CJOJPJQJphsH mH EHCJOJPJQJphsH mH "CJOJPJQJphsH mH EH  " $ & ( ЛКuК0Кˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџ( * . 0 ЛКuˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџˆ568:;<79XTS*CJOJPJQJ@ˆphџRHdsHmHtH_HH*>*KHEHwhџ *ehrЪџfHqЪ џџ>rt˜ ^ ` p$вb<   " $ & ( * . 0 §§§§§іччччччхусусусусх & F^„а`„ ўЦ єаЦмаsH mH H*>*KHEHfHqЪ џџ "џџџџ а ( 2 2 џџUnknownsшN‰џџџџџџџџџ^„є`„мЦєo(EH" ^„а`„Цаo(EH^„ `„Ц o(EH^„p`„Цpo(EH^„@ `„Ц@ o(EH^„`„Цo(EH^„р`„Црo(EH^„А`„ЦАo(EH^„€`„Ц€o(EHџџџџџџ `iTimes New RomanTimes New RomanESymbolSymbolA&ArialArialQHelveticaHelveticaaв0щ0Ў0Ю0в‰Д0 Pro W3в0щ0Ў0Ю0в‰Д0 Pro W3#ˆаhdхfdхf!€#@(0 Root Entryˆy /аКа'№_А_@#_`#P$0_`Р0^p^_р"џџџџџџџџ€>еоБ€OЛ4@ђЪ(@1TableР"РМюшАˆшан0јџџџџ€OЛ4@ђЪ€OЛ4@ђЪWordDocument   $Pg  џџџџ€OЛ4@ђЪ€OЛ4@ђЪ.Data@ рцрkрцшш ш џџџџџџџџџџџџ€OЛ4@ђЪ€OЛ4@ђЪўџеЭеœ.“—+,љЎDеЭеœ.“—+,љЎ|8 ( 0 ўџр…ŸђљOhЋ‘+'Гй0SummaryInformationр1@06(џџџџџџџџ€OЛ4@ђЪ€OЛ4@ђЪHDocumentSummaryInformation .p28џџџџџџџџџџџџ€OЛ4@ђЪ€OЛ4@ђЪObjectPoolРМ№џџџџџџџџџџџџ€OЛ4@ђЪ€OЛ4@ђЪџџџџФ@:№№џџџџџџџџџџџџ€>еоБ€>еоБџџџџўџџџўџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџ ўџџџўџџџ !"#$%&ўџџџ)ўџџџўџџџўџџџ§џџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџџ