Revising UO’s Teaching Evaluations

The Office of the Provost and the University Senate have been working together since spring 2017 to revise University of Oregon’s teaching evaluation system. Recent research indicates that student ratings may not accurately reflect teaching quality and may be inflected by bias [3456, 9, 10]. A research project at the University of Oregon similarly cast doubt on the reliability of numerical course evaluations [2].

The Association of American Universities (AAU) and other universities around the globe from University of Colorado, Boulder to University College London, England have argued that it is time for universities’ ideals regarding teaching excellence to align with their policies [178]. As such, the University of Oregon has developed a holistic new teaching evaluation system that does more than simply replace problematic evaluation instruments. The new system provides the path to definedevelopevaluate, and reward teaching excellence. The goals of the new system are to ensure teaching evaluation is fair and transparent, is conducted against criteria aligned with the unit’s definition of teaching excellence, and includes input from students, peers and the faculty themselves.

In August 2019 a Memorandum of Understanding (MOU) between the faculty union and the Provost communicates an update to the Collective Bargaining Agreement (CBA) and enshrines the commitment to evaluating professional, inclusive, engaged and research-led teaching practices. A new Senate committee formed in Fall 2018, the Continuous Improvement and Evaluation of Teaching (CIET) committee, oversees implementation of the April 2019 Senate legislation which includes the elements described below.


Course Surveys

1. Midway Student Experience Survey (M-SES)

Term: Week 4 | Semester: Week 7
Opens: 8 a.m., Monday
Closes: 6 p.m. Friday


2. End-of-course Student Experience Survey (E-SES)

Term: Week 10 | Semester: Week 15
Opens: 8 a.m., Monday
Closes: 6 p.m. Friday
*Replaces previous Course Evaluations


3. Instructor Reflection

Opens: 8 a.m., Monday, of Week 10 for terms and Week 15 for semesters.
Closes: 6 p.m. Friday of the first week of the following term or semester.


The Continuous Improvement and Evaluation of Teaching System

There are three important inputs included in the evaluation of teaching system. These include student feedback, instructor reflection, and peer review. New tools are available in each of these categories to support quality teaching and ensure that instructors receive useful and constructive feedback.

Student Feedback

Student feedback is an important part of the Continuous Improvement and Evaluation of Teaching System because students are at the center of the teaching and learning endeavor and have valuable perspectives to share. Students also have the potential to provide instructors with useful feedback to clarify which teaching practices are most beneficial to their learning experience, and what could be improved to help them learn. 

Midway Student Experience Survey (M-SES)

The University Senate approved a centrally distributed Midway Student Experience Survey (M-SES) in which student feedback is only available to the instructor and is intended for mid-course adjustments or for clarifying goals and expectations with students. Campus-wide distribution began fall 2019. 

Students are invited to complete the M-SES during week 4 of the term (or week 7 for the School of Law semester) between Monday at 8 a.m. and Friday at 6 p.m. In order to improve response rates and receive high-quality feedback, instructors are encouraged to provide 10-minutes of class time for students to take the survey. The results of the survey will be available to instructors the following week on Monday at noon; instructors are encouraged to debrief with their students about the feedback they received.

Instructors who have their own midway student feedback instruments and do not plan to read student feedback from the M-SES should communicate this clearly to students prior to week 4 of the term (or law semester week 7) so students know to skip that specific course when completing their M-SES surveys.

End-of-course Student Experience Survey (E-SES)

The University Senate also approved a new End-of-course Student Experience Survey (E-SES) to replace the previous student Course Evaluation. After a year of pilot testing and modifications, campus-wide deployment began fall 2019. The E-SES is focused on student learning, asks concrete questions about specific teaching practices, and inquires about student contributions to their own learning. In reports, student comments are collated by teaching practice making it easier to digest and interpret student feedback.

Analyses from E-SES pilots indicated this survey increases the number of student comments by 60% and drastically reduces the frequency of personal comments (from 21% to 1.5%). Additionally, most E-SES comments (61%), are positive, as students elaborate on teaching practices that are beneficial for their learning.

The E-SES will be available for students the last week of the term or semester prior to finals (Week 10 for the term, or Week 15 for the School of Law semester) from Monday at 8 a.m. to Friday at 6 p.m. For best response rates and high-quality feedback, instructors are encouraged to use the in-class protocol and provide 10-minutes of class time during Week 10/Week 15 for the students to complete their E-SES. Instructors can view reports the Wednesday after grades are submitted.

Implemented Fall 2019, the CIET senate committee created a process to remove discriminatory, obscene, or demeaning SES comments committee as described by the protocol. Instructors can “flag” comments for possible redaction which will be reviewed by the CIET senate committee at least three times per year between October and May, and the upcoming review date will be listed on the main Course Surveys page in DuckWeb.

Self Reflection

A crucial part of teaching evaluation is an instructor’s reflection of their own teaching. The UO believes self-reflection should be a regular part of an instructor’s course planning and development, and that the instructor’s own voice should be available at the course level to complement student feedback from the E-SES.

To facilitate on-going self-reflection, the University Senate approved a new Instructor Reflection, which was deployed campus-wide fall 2019. The tool provides an opportunity for instructors to archive what went well and what might be improved in the future as well as how their teaching aligns with UO’s definition of teaching excellence. The reflection also provides a new mechanism for the instructor’s own voice to inform evaluators’ interpretation of student feedback.

Peer Review

In order to improve upon peer review’s efficacy as a tool for continuous teaching improvement and evaluation, the 2018-19 CAIT on Teaching Excellence and Evaluation developed a tool for units to use to reflect on their own peer review systems and practices. The tool provides questions such as “Do you ever have to rush to get a peer review done to get it into a file for midterm review, promotion, tenure etc.?” or “Are classroom observations done with a blank piece of paper and only the reviewer’s personal beliefs about teaching as a guide?” It then provides concrete solutions for each concern, all of which are best practices already successfully implemented in some UO departments.

This guide for unit-level self-study is meant to support departments in ensuring their peer review practices are serving them well.


Evaluation of Teaching Criteria

Revising UO’s evaluation tools is only one step of the process. In order for evaluations to matter, there needs to be a set of standards against which teaching practices can be measured. Currently, many unit policies do not define teaching quality nor do they provide basic criteria that outline the behaviors or outcomes for meeting teaching quality expectations. The August 2019 Memorandum of Understanding between the University and United Academics clarifies the minimum definition of teaching quality to be used for evaluation of teaching beginning in Fall 2020. Units are invited to modify the standards outlined in the MOU with unit-specific standards.

The MOU describes standards for professional, inclusive, engaged and research-informed teaching, which are in alignment with the UO’s teaching excellence principles. The Provost’s Teaching Academy has been engaging with these principles since its inception in 2016, and members have posted their professional teaching profiles describing their inclusive, engaged and research-led teaching practices. As units are thinking about what teaching quality means to them, we offer this document to structure departmental reflection and conversation.


Timeline

This timeline displays our progress on these initiatives in reverse chronological order.

Winter 2020: 

Winter Meetings with UO Faculty and Leadership

  • January – Department of Biology, faculty meeting
  • January – Department of Creative Writing, faculty meeting
  • January – Department of Political Science, faculty meeting
  • February – Department of Romance Languages, faculty meeting
  • February – Department of Physics, faculty meeting

Fall 2019

  • Campus-wide deployment of:
  • Continuous Improvement and Evaluation of Teaching (CIET) Senate Committee develops protocol for redaction of discriminatory, obscene or demeaning E-SES comments
  • 2019/20 Teaching Excellence and Evaluation Community Accelerating the Impact of Teaching (CAIT) group formed with representation from all schools and colleges with a focus on the Teaching Evaluation Criteria document and new data dashboard.

Fall Meetings with UO Faculty and Leadership

  • August - College of Education, leadership meeting
  • August – School of Law, faculty meeting
  • September - Summit for Academic Leaders
  • September – School of Music and Dance, faculty meeting
  • September – School of Architecture and the Environment, faculty meeting
  • September – Department of International Studies, faculty meeting
  • September – Professional Schools Associate Deans meeting
  • October – Department of Women, Gender & Sexuality Studies, faculty meeting
  • October – School of Journalism and Communication, faculty personal committee meeting
  • October – Department of Special Education, faculty meeting
  • October – Department of Philosophy, committee meeting
  • October – Portland Campus, zoom meeting for faculty
  • November – Department of Art, faculty meeting
  • November – Department of East Asian Languages & Literatures, faculty meeting
  • November – Department of Sociology, faculty meeting
  • November – Department of Dance, faculty meeting
  • November – School of Planning, Public Policy and Management, faculty meeting
  • November – Division of Humanities, heads & managers meeting
  • November – Getting More from Peer Review workshop, for faculty and heads
  • December – ‘First Cohort’ workshop, faculty up for review under new standards Fall 2020
  • December – Department of History, faculty meeting
  • December – Department of Educational Methodology, Policy and Leadership, faculty meeting
  • December – History of Art and Architecture, faculty meeting
  • December – Department of Comparative Literature, faculty meeting

Summer 2019

  • United Academics and University sign a Memorandum of Understanding (MOU) enshrining professional, inclusive, engaged and research-informed teaching standards and baseline conditions for use in formal teaching evaluation.  

Spring 2019

Winter 2019

  • March 20 - Updates on progress shared at All Unit Heads training.
  • February 18 - Around the O: Workplace story outlining reform efforts.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup continues to pilot new tools:
  • January 30 - University Senate approved the CIET senate committee’s work on a Warning and Guidance on Student Evaluations of Teaching document for faculty and personnel committees.
  • January 13 - Chronicle of Higher Education article highlights UO’s work on course evaluation.
  • Teaching Excellence and Evaluation CAIT Peer Review subgroup members each selects an observation tool to experiment with for Peer Review and completes 3 mock classroom observations.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup makes tweaks to the Midterm Student Experience Survey for week 5 and selects End of Term Student Experience Survey to Pilot in week 10. Pilot faculty will also complete Instructor Reflection.

Fall 2018

  • November 28 - Ginger Clark, assistant vice provost for academic and faculty affairs at the University of Southern California, hosted two discussions about USC's reform of their teaching evaluation process.
  • November 17 - Sierra Dawson, Austin Hocker, and Lee Rumbarger lead discussion about Teaching Evaluation reform efforts at Professional and Organization Developers Network (POD) conference in Portland, OR.
  • Teaching Excellence and Evaluation CAIT Pilot subgroup includes English, Honors College, Human Physiology, Lundquist College of Business, and School of Planning, Public Policy and Management. From these units:
  • Teaching Excellence and Evaluation CAIT provided guidance for an update on the definition of Teaching Excellence.
  • Teaching Excellence and Evaluation CAIT (Community Accelerating the Impact of Teaching) formed with membership from each and every school and college, as well as the three divisions of the College of Arts and Sciences.
  • Continuous Improvement and Evaluation of Teaching Senate Committee formed based on spring Senate Legislation.
  • October 3 - University Senate updated with progress report and plan for the year.

Summer 2018

  • Faculty teaching courses in Business, English, Public Planning, Policy and Management, Writing, Romance Languages, Education Studies, Cinemas Studies, and Human Physiology are piloting the Midterm Student Experience Survey, End-of-Term Student Experience Survey, and 10-Minute Instructor Reflection in the CollegeNET system.

Spring 2018

  • Pilots of updated Midterm Student Experience Survey in Qualtrics, as well as End-of-Term Student Experience Survey and 10-Minute Instructor Reflection in CollegeNET
  • May 1 - Town Hall for undergraduate students, 6:00-7:00 p.m., EMU Redwood Auditorium
  • May 2 - Town Hall for unit heads, 8:30-9:30 a.m., EMU Redwood Auditorium
  • May 2 - Town Hall for graduate students, 10:00-11:00 a.m., EMU Redwood Auditorium
  • May 3 - Town Hall for faculty, 10:00-11:00 a.m., Ford Alumni Center Ballroom
  • May 9 - Senate discussion of motion
  • May 23 –Senate motion US17/18-19 legislates the following:
    • The Continuous Improvement and Evaluation of Teaching senate committee will be for
    • Student evaluations will be anonymous
    • Midterm Student Experience Surveys will become available via CollegeNET
    • 10-Minute Instructor Reflections will become available via CollegeNET

Winter 2018

Fall 2017

  • Meetings with stakeholders: Associate Dean’s Luncheon on Teaching Evaluation, Grad Council, ASUO Exec, SWAT, Graduate Student Advisory Board, Women’s Center, Dean of Student Life-Staff, Student Trustee and Senators, Women of Color Coalition, LCB Dean’s Undergraduate Student Advisory Council, Dean of Students Advisory Committee, Mujeres student group, IMPACT (meetings continue through winter and spring terms)

Spring 2017


Documents


References

To learn more about the research behind our initiative to revise the UO Teaching Evaluation System, please see the following documents.

  1. Aligning Practice to Policy: Changing the Culture to Recognize and Reward Teaching at Research Universities
  2. Teaching, Learning and Achievement: Are Course Evaluations Valid Measure of Instructional Quality at the University of Oregon
  3. Student Evaluations Can’t Be Used to Assess Professors
  4. Gender Bias in Teaching Evaluations
  5. Teaching Evaluations and Bias
  6. Is Gender Bias an Intended Features of Teaching Evaluations?
  7. Undergraduate STEM Education Initiative
  8. The Career Framework for University Teaching
  9. Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27. DOI: 10.1016/j.jpubeco.2016.11.006 at https://www.sciencedirect.com/science/article/abs/pii/S0047272716301591
  10. MacNell, L., Driscoll, A. & Hunt, A.N. (2015). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, 40, 291. DOI: 10.1007/s10755-014-9313-4

Updated January 2020