Revising UO’s Teaching Evaluations

The Office of the Provost and the University Senate have been working together since spring 2017 to revise University of Oregon’s teaching evaluation system. Recent research indicates that student ratings may not accurately reflect teaching quality and may be inflected by bias [3456, 9, 10]. A research project at the University of Oregon similarly cast doubt on the reliability of numerical course evaluations [2].

The Association of American Universities (AAU) and other universities around the globe from University of Colorado, Boulder to University College London, England have argued that it is time for universities’ ideals regarding teaching excellence to align with their policies [178]. As such, the University of Oregon has developed a holistic new teaching evaluation system that does more than simply replace problematic evaluation instruments. The new system provides the path to definedevelopevaluate, and reward teaching excellence. The goals of the new system are to ensure teaching evaluation is fair and transparent, conducted against criteria aligned with the unit’s definition of teaching excellence, and includes input from students, peers and the faculty themselves.

In August 2019 a Memorandum of Understanding (MOU) between the faculty union and the Provost communicates an update to the Collective Bargaining Agreement (CBA) and enshrines the commitment to evaluating professional, inclusive, engaged and research-led teaching practices. A new Senate committee formed in Fall 2018, the Continuous Improvement and Evaluation of Teaching (CIET) committee, will oversee implementation of April 2019 Senate legislation which includes the elements described below.


Course Surveys

1. Midway Student Experience Survey (M-SES)

Term: Week 4 | Semester: Week 7
Opens: 8 a.m., Monday
Closes: 6 p.m. Friday


2. End-of-course Student Experience Survey (E-SES)

Term: Week 10 | Semester: Week 15
Opens: 8 a.m., Monday
Closes: 6 p.m. Friday
*Replaces previous Course Evaluations


3. Instructor Reflection

Opens: 8 a.m., Monday, of Week 10 for terms and Week 15 for semesters.
Closes: 6 p.m. Friday of the first week of the following term or semester.


The Continuous Improvement and Evaluation of Teaching System

There are three important inputs included in the evaluation of teaching system. These include student feedback, instructor reflection, and peer review. New tools are available in each of these categories to support quality teaching and ensure that instructors receive useful and constructive feedback.

Student Feedback

Student feedback is an important part of the Continuous Improvement and Evaluation of Teaching System because students are at the center of the teaching and learning endeavor and have valuable perspectives to share. Students also have the potential to provide instructors with useful feedback to clarify which teaching practices are most beneficial to their learning experience, and what could be improved to help them learn. 

Midway Student Experience Survey (M-SES)

The University Senate approved a centrally distributed Midway Student Experience Survey (M-SES) in which student feedback is only available to the instructor and is intended for mid-course adjustments or for clarifying goals and expectations with students. Campus-wide distribution begins fall 2019. 

Students will be invited to complete the M-SES during week 4 of the term (or week 7 for the School of Law semester) between Monday at 8 a.m. and Friday at 6 p.m. Instructors are encouraged to provide 10 minutes of class time for students to take the survey for best response rates and high-quality feedback. The results of the survey will be available to instructors the following week on Monday at noon; instructors are encouraged to debrief with their students about the feedback they received.

Instructors who have their own midway student feedback instruments and do not plan to read student feedback from the M-SES should communicate this clearly to students prior to week 4 of the term (or law semester week 7) so students know to skip that specific course when completing the survey.

End-of-course Student Experience Survey (E-SES)

The University Senate also approved a new End-of-course Student Experience Survey (E-SES) to replace the previous student Course Evaluation. After a year of pilot testing and modifications, campus-wide deployment begins fall 2019. The E-SES is focused on student learning, asks concrete questions about specific teaching practices, and inquires about student contributions to their own learning. In reports, student comments are collated by teaching practice making it easier to digest and interpret student feedback.

Analyses from E-SES pilots indicate this survey increases the number of student comments by 60% and drastically reduces the frequency of personal comments (from 21% to 1.5%). Additionally, most E-SES comments (61%), are positive, as students elaborate on teaching practices that are beneficial for their learning.

The E-SES will be available for students the last week of the term or semester prior to finals (Week 10 for the term, or Week 15 for the School of Law semester) from Monday at 8 a.m. to Friday at 6 p.m. For best response rates and high-quality feedback, instructors are encouraged to use the in-class protocol and provide 10-minutes of class time during Week 10 for the students to complete their E-SES. Instructors can view reports the Wednesday after grades are submitted.

Self Reflection

A crucial part of teaching evaluation is instructors’ reflection on their own teaching. The UO believes self-reflection should be a regular part of instructors’ course planning and development, and that the instructor’s own voice should be available at the course level to complement student feedback from the E-SES.

To facilitate on-going self-reflection, the Senate approved a new Instructor Reflection, which will be deployed campus-wide fall 2019. The tool makes it easy for instructors to archive what went well and what might be improved in the future as well as how their teaching aligns with UO’s definition of teaching excellence. The reflection also provides a new mechanism for the instructor’s own voice to inform evaluators’ interpretation of student feedback.

Peer Review

In order to improve upon peer review’s efficacy as a tool for continuous teaching improvement and evaluation, the 2018-19 CAIT on Teaching Excellence and Evaluation developed a tool for units to use to reflect on their own peer review systems and practices. The tool provides questions such as “Do you ever have to rush to get a peer review done to get it into a file for midterm review, promotion, tenure etc.?” or  “Does the peer review report feel like a creative writing assignment, or read like a love letter?” It then provides concrete solutions for each concern, all of which are best practices already successfully implemented in some UO departments.

This guide for unit-level self study is meant to support departments in ensuring their peer review practices are serving them well.


Evaluation of Teaching Criteria

Revising UO’s evaluation tools is only one step of the process. In order for evaluations to matter, there needs to be a set of standards against which teaching practices can be measured. Currently, many unit policies do not define teaching quality nor do they provide basic criteria that outline the behaviors or outcomes for meeting teaching quality expectations. The August 2019 Memorandum of Understanding between the university and faculty union clarifies the minimum definition of teaching quality to be used for evaluation of teaching beginning in Fall 2020. Units are invited to modify the standards outlined in the MOU with unit-specific standards.

The MOU describes standards for professional, inclusive, engaged and research-informed teaching, which are in alignment with the UO’s teaching excellence principles. The Provost’s Teaching Academy has been engaging with these principles since its inception in 2016, and members have posted their professional teaching profiles describing their inclusive, engaged and research-led teaching practices. As units are thinking about what teaching quality means to them, we offer this document to structure departmental reflection and conversation.


Timeline

This timeline displays our progress on these initiatives in reverse chronological order.

Fall 2019: Implementation of new tools campus-wide

Quarter System Dates:

Law Semester Dates:

Fall 2019: Meetings with UO Faculty and Leadership

Spring 2019

Winter 2019

  • March 20 - Updates on progress shared at All Unit Heads training.
  • February 18 - Around the O: Workplace story outlining reform efforts.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup continues to pilot new tools:
  • January 30 - University Senate approved the CIET senate committee’s work on a Warning and Guidance on Student Evaluations of Teaching document for faculty and personnel committees.
  • January 13 - Chronicle of Higher Education article highlights UO’s work on course evaluation.
  • Teaching Excellence and Evaluation CAIT Peer Review subgroup members each selects an observation tool to experiment with for Peer Review and completes 3 mock classroom observations.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup makes tweaks to the Midterm Student Experience Survey for week 5 and selects End of Term Student Experience Survey to Pilot in week 10. Pilot faculty will also complete Instructor Reflection.

Fall 2018

  • November 28 - Ginger Clark, assistant vice provost for academic and faculty affairs at the University of Southern California, hosted two discussions about USC's reform of their teaching evaluation process.
  • November 17 - Sierra Dawson, Austin Hocker, and Lee Rumbarger lead discussion about Teaching Evaluation reform efforts at Professional and Organization Developers Network (POD) conference in Portland, OR.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup includes English, Honors College, Human Physiology, Lundquist College of Business, and School of Planning, Public Policy and Management. From these units:
  • Teaching Excellence and Evaluation CAIT provided guidance for an update on the definition of Teaching Excellence.
  • Teaching Excellence and Evaluation CAIT (Community Accelerating the Impact of Teaching) formed with membership from each and every school and college, as well as the three divisions of the College of Arts and Sciences.
  • Continuous Improvement and Evaluation of Teaching Senate Committee formed based on spring Senate Legislation.
  • October 3 - University Senate updated with progress report and plan for the year.

Summer 2018

  • Faculty teaching courses in Business, English, Public Planning, Policy and Management, Writing, Romance Languages, Education Studies, Cinemas Studies, and Human Physiology are piloting the Midterm Student Experience Survey, End-of-Term Student Experience Survey, and 10-Minute Instructor Reflection in the CollegeNET system.

Spring 2018

  • Pilots of updated Midterm Student Experience Survey in Qualtrics, as well as End-of-Term Student Experience Survey and 10-Minute Instructor Reflection in CollegeNET
  • May 1 - Town Hall for undergraduate students, 6:00-7:00 p.m., EMU Redwood Auditorium
  • May 2 - Town Hall for unit heads, 8:30-9:30 a.m., EMU Redwood Auditorium
  • May 2 - Town Hall for graduate students, 10:00-11:00 a.m., EMU Redwood Auditorium
  • May 3 - Town Hall for faculty, 10:00-11:00 a.m., Ford Alumni Center Ballroom
  • May 9 - Senate discussion of motion
  • May 23 –Senate motion US17/18-19 legislates the following:
    • The Continuous Improvement and Evaluation of Teaching senate committee will be for
    • Student evaluations will be anonymous
    • Midterm Student Experience Surveys will become available via CollegeNET
    • 10-Minute Instructor Reflections will become available via CollegeNET

Winter 2018

Fall 2017

  • Meetings with stakeholders: Associate Dean’s Luncheon on Teaching Evaluation, Grad Council, ASUO Exec, SWAT, Graduate Student Advisory Board, Women’s Center, Dean of Student Life-Staff, Student Trustee and Senators, Women of Color Coalition, LCB Dean’s Undergraduate Student Advisory Council, Dean of Students Advisory Committee, Mujeres student group, IMPACT (meetings continue through winter and spring terms)

Spring 2017


Documents


References

To learn more about the research behind our initiative to revise the UO Teaching Evaluation System, please see the following documents.

  1. Aligning Practice to Policy: Changing the Culture to Recognize and Reward Teaching at Research Universities
  2. Teaching, Learning and Achievement: Are Course Evaluations Valid Measure of Instructional Quality at the University of Oregon
  3. Student Evaluations Can’t Be Used to Assess Professors
  4. Gender Bias in Teaching Evaluations
  5. Teaching Evaluations and Bias
  6. Is Gender Bias an Intended Features of Teaching Evaluations?
  7. Undergraduate STEM Education Initiative
  8. The Career Framework for University Teaching
  9. Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27. DOI: 10.1016/j.jpubeco.2016.11.006 at https://www.sciencedirect.com/science/article/abs/pii/S0047272716301591
  10. MacNell, L., Driscoll, A. & Hunt, A.N. (2015). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, 40, 291. DOI: 10.1007/s10755-014-9313-4

Updated August 2019