Using Machine Learning to Understand Student Perspectives on Inclusiveness: An Analysis of the Student Experience Survey 

Austin Hocker, Charlotte Lloyd, Grant Crider-Phillips, & Sung-Woo Cho

Print-friendly PDF:

File Inclusiveness Practitioner Guide 

Table of Contents:

1. Exploring student comments 
2. Identifying themes in student comments 
3. What specifically about the inclusiveness of the course helped your learning? 
4. What specific change in the inclusiveness of the course would help your learning? 
5. Discussion 
6. Appendix A: Top Forty Distinguishing Words in “Inclusiveness” Responses* 

The new UO end of course Student Experience Survey (SES) asks students open-ended questions about 13 individual teaching practices. Each student selects the single teaching practice they consider most beneficial to their learning and a single teaching practice that needs improvement in order to support their learning, with prompts to leave textual comments about both answer choices. Therefore, student comments represent perspectives on when a teaching practice is either the most salient positive or negative teaching practice for their learning. One of the teaching practices students can choose and comment on is “ The inclusiveness of the course. ” From these comments, we can learn how students define inclusive teaching and which elements of their courses are the most inclusive or most in need of improvement to be more inclusive.

To contextualize the student comments on the most beneficial and most in need of improvement, a separate survey question asks all respondents to indicate if the inclusiveness of the courses is beneficial, neutral, or needing improvement for their learning. Across all SES responses from 2019-2021 (116,331 student responses), inclusiveness is rated as needing improvement by 4.0% of respondents, as neutral by 17.2% of respondents, and as beneficial for their learning by 78.8% of respondents.

Figure 1. Percent of Student Experience Survey comments about teaching practices selected as most beneficial and most in need of improvement.

From the 13 teaching practices, “ The inclusiveness of the course ” is selected as being “most beneficial for my learning” by 5.3% of respondents (5,427 student comments) and is selected as being “most in need of improvement for my learning” by 1.8% of respondents (1,642 student comments). As context, students can only choose the single element that is the most beneficial or most in need of improvement and over 30% of students responded that “none of these elements need improvement.” It is important to keep in mind that the set of responses considered in this analysis come from the small percentage of students who found inclusiveness to be either the most positive or negative experience relative to other teaching and learning elements. The types and topics of student responses about inclusiveness might be very different if, for instance, every student was prompted to write about the inclusiveness of the course.

Student comments on the inclusiveness learning element were relatively short: a median response of only 65 characters, or roughly 13 words (for reference, tweets were originally capped at 140 characters, or approximately 28 words). Compared to student responses to other SES questions, comments about inclusiveness placed great emphasis on feelings and environment , in particular if students felt comfortable, welcome, and safe . Another striking finding is how frequently students wrote about their experiences with group discussions , reflected in the high prevalence of words including speak, interact, discuss, share, participate, and opinion (see more in Appendix A).

To identify themes from student responses, we used a hybrid approach combining traditional close reading with a machine learning approach called Latent Dirichlet Allocation (LDA), a topic modeling technique that identifies underlying themes in text by looking for latent groupings of features, in our case individual words. We used LDA as an exploratory tool to present an expert researcher with topics of student comments, which the researcher analyzed and synthesized into the five themes presented here. The value of using LDA in this process comes from the neutrality of the algorithm’s topic groupings, which offers the researcher fresh ways of looking at the data that are relatively unconditioned by their prior expectation about what themes are present in student responses. With LDA categorizations as a starting point, the researcher’s human judgment and experience with UO student surveys are then crucial for integrating topic model results into a rich qualitative understanding of student experiences of inclusiveness.

Our approach identified five common themes. Below each theme is described alongside representative student comments.

1. Class communication: students speaking in class, participation encouraged, questions are welcomed and answered: 
2. Content includes diverse perspectives, different materials 
3. Instructors are accommodating, flexible, and understanding: 
4. Using inclusive language 
5. Student choice, relevance 

Class communication: students speaking in class, participation encouraged, questions are welcomed and answered:

Student perceptions of class communication and discussions is one of the most common themes. Students cite the feeling of class discussions and speaking in class and reflect that their instructor made them feel their voices and perspectives matter.

“I liked how we all were encouraged to speak in the class and that our opinions mattered.”

“Although I'm not super comfortable talking in front of a class they made me feel comfortable and able to contribute!”

Students also comment on specific facilitation practices that were beneficial for their engagement and learning. For example,

“Give students the opportunity to speak up by encouraging those with louder voices to have others heard.”

“ The instructor never attacks anyone's opinions, and they are great at hearing someone out and then condensing what they've said in an accurate way so everyone can better understand.”

“I like how the discussions are very open and you have the option to speak or not speak, and it helps me through learning about other perspectives and hearing different approaches to explaining or thinking about concepts. I am more of a small group person, so those opportunities to talk in small groups are valuable for me, but I appreciate the value of listening to large group discussions.”

“Though participation is an important aspect of this class, the professor offers an alternative to speaking in class by allowing us to email him with our thoughts and have that counted as participation points. As someone who gets anxious trying to speak in a discussion class like this, this goes above and beyond…”

Students comment about how it feels to ask questions as an inclusive teaching practice. Students either perceive that their questions are welcome or that instructors probe students to understand what questions they have. Students reference “questions” or “answers” in 4.5% of responses. For example,

“The professor is kind and wants to make sure we all understand by asking us questions, doing polls, and working to understand what we are having trouble with.”

Content includes diverse perspectives, different materials

Student comments frequently indicated the benefits of including diverse perspectives in the content of the course and in specifically addressing how course content impacts marginalized communities.

“The instructors of this course made a point to provide material from a broad range of perspectives that made the class feel inclusive.”

“As one of the two women of color in this class, I appreciate [the instructor’s] effort to address racial issues in [class content].”

“Being Native American, I felt more invested in this class when everyone discussed the impacts that [topic] has had on Indigenous people....”

Students also reference the benefits of different formats of materials,

“A really diverse and thorough list of materials for this class helped a lot in my understanding of the issues we discussed. I also really appreciated that all of these materials were provided for free, absolutely awesome and should be the standard.”

“[Instructor] does a lovely job of balancing out different perspectives and ideas, and always works to make students feel included even if they don't necessarily come from a STEM background.”

“Additionally, the instructor provided a variety of materials for class, which included different formats (videos, short articles, and textbook readings) as well as a variety of viewpoints on different topics, which really gave a full picture of the topic at hand.”

Instructors are accommodating, flexible, and understanding:

Students discuss flexibility, accommodations, and feeling their instructors understood their challenges as being beneficial to the inclusiveness of the course.

“[Instructor] is an ideal example of how faculty should manage their courses if they care about equity. They understood that our cohort was experiencing significant stress during this term and that the ongoing pandemic continued to take a toll on our energy levels, ability to concentrate and stay engaged. In response to our needs, they adapted the course syllabus and modified some course expectations to ensure that (a) we continue to receive a quality education, and (b) we could maximize our learning in our changing context.”

“[Instructor] understands that their students' learning needs are different and they did their best to give everyone a great learning experience in this difficult term. Their instructions are very clear and thorough, the classes are very accommodating, and the help and support to us are very personalized”

Using inclusive language

Comments referencing student’s appreciation for instructors use of inclusive language, particularly related to using appropriate pronouns and gender-inclusive terms. For example:

“very good at trying to use inclusive language and examples to help us understand the content from multiple perspectives.”

“as someone in the LGBTQ community, the instructor really worked hard on being inclusive and succeeded. You always made it clear when you were talking about biological sex versus gender, and never used terms like "normal" or other exclusive words.”

“Out of all my classes, I felt the most welcomed and respected here. I'm pretty sure everyone always used the correct pronouns (I'm trans) and I even had one instance of another student calling someone out for mis-gendering me.”

Student choice, relevance

Students cite the ability to choose ways to interact with content that aligns with their own interests as being a beneficial inclusive teaching practice. For example, modifying the content of the course to align with student interests:

“Material tried to apply to many types of individuals and their goals for future practice (i.e. ethics, job search).”

“The professor scales the course content to the background knowledge of the students”

Or being able to choose their own topics or explore content in a way that aligns with their own perspectives or skills:

“I liked being able to work with different mediums and explore our individual take on different prompts. I also like presenting my work and receiving feedback because although it may be scary it is good practice.”

“Because we each have to present on the readings as well as our projects, it really includes people and their interests in giving them the freedom to choose what they want to report on that interests them the most.”

Themes:

1. Class discussions, dominating students, student questions, and engagement 
2. Diversify content, authors, and speakers 
3. Being more understanding of students, flexibility 
4. Student interaction 
5. Inclusive language 

Class discussions, dominating students, student questions, and engagement

Again, the most prevalent theme in student responses relates to areas for improvement in the ways instructors facilitate discussions, especially around moderating students who dominate the conversation and creating more opportunities for every student to engage.

“I think there could've been more ways for more students to engage in the coursework. As the term continued, really the same people continued to participate with no new ones. Providing different opportunities for more engagement could've been improved.”

“I think if students were more inclusive and respectful of everyone inside and outside the class, it would be easier for everyone to contribute in class. I guess saying from the start that everyone's voice is valued and we want to include everyone in the class discussions will help everyone contribute in class.”

“I think the professor does try to make all discussion inclusive but ends up not always being the case when certain students dominate the conversation”

“The discussions were largely dominated by a few people and the teacher regularly called on them. Students have different ways in which they like to participate and encouraging students to speak their opinions would have be beneficial for my learning experience.”

“I think it would be helpful if the instructor moderated some in-class discussions a bit more. It's foreseeable that some of the topics covered could lead to sensitive issues, and I think it would be helpful if the instructor set some standards for constructive and respectful discussion, especially in terms of acknowledging the life experiences of underrepresented groups in class (instead of arguing with them).”

“I think it would be more beneficial if we did more small group discussions so we have the opportunity to discuss our own ideas with smaller groups then share them with the class”

“The instructor needs to select people more thoughtfully to participate in class, and they need to remember people’s names.”

Students also cite the feeling of being able to ask questions and feeling respected and answered by their instructors.

“I was terrified of asking questions because I was afraid of being ridiculed by even thinking of asking a question you didn't deem worthy.”

“The instructor comes across as very condescending and frankly rude when a student incorrectly answers a question, so I think a lot of students were intimidated by him and was afraid of being wrong, at least I can say I know I felt that way. It was almost embarrassing to answer questions if you didn't 100% understand the topic at hand or confident in your answer.”

Diversify content, authors, and speakers

Students frequently request more diverse materials, readings from diverse authors, and more diverse guest speakers.

“We need class materials written by black, indigenous and/or people of color. We need more guest speakers that come from intersectional identities.”

“I felt that the course could use more diversity in speakers.”

“Try to get some guest speakers of color.”

Being more understanding of students, flexibility

Students see opportunities for instructors to be more understanding of their experiences and to use appropriate accommodations.

“It would be beneficial to my learning if you said the exact same feedback, but in a more understanding way that acknowledges that you've been in our shoes at some point in your education, and sometimes reassure us that we'll get there some day.”

“With all of the stresses our community has experienced these past few months, sympathy/empathy from professors is greatly appreciated.”

“The instructor needs to spend time understanding and using AEC accommodations.”

And identify ways in which grading practices could be more flexible or equitable in relation to student needs.

“Accepting late assignments. Technically no one gets a free absence because if you miss a class you still miss the points that you would have gotten from an assignment. You should be allowed to make up the points.”

“The grading in this course is WAY too test heavy, students who are not strong test takers/need more hands-on learning are at a disadvantage and less grading weight should be based on tests”

Student interaction

Some students interpret inclusive teaching as how they are included in their own learning process. For example, students describe how they interact with peers, the instructor, or with the content as inclusive teaching practices.

“Literally zero student interaction (which is my favorite aspect of classes). No questions to students.”

“Maybe have small assignments like small pop quizzes on canvas to keep things interesting. The interactive/inclusive part does not necessarily have to be in class, but it would keep us interested between the start of the term and the midterm, etc.”

“I believe this class if possible can become more of a discussion with students rather than just slides”

Inclusive language

Students request more deliberate use of inclusive language, most often related to gender-inclusive language.

“Being a non-binary person it would have been nice to see more gender-inclusive language”

“lack of inclusiveness with pronouns (using he/him/his) was disappointing”

The themes presented above provide insight into how students define inclusive teaching and what inclusive teaching practices support their learning. In many ways, student’s perceptions of inclusive teaching align well with the definition of Inclusive teaching at University of Oregon.

“Instruction designed to ensure every student can participate fully and that their presence and participation is valued. The content of the course reflects the diversity of the field's practitioners, the contested and evolving status of knowledge, the value of academic questions beyond the academy and of lived experience as evidence, and/or other efforts to help students see themselves in the work of the course.”

For example, students indicate the importance of designing class discussions to include more student’s participation and how student interactivity with course content would benefit their learning. Students also comment on the feeling of their participation, voices, and questions being valued by the instructor. Similarly, students request course content from more diverse perspectives, acknowledgement of how content impacts marginalized communities, and more opportunities for student choice to find the relevance of content to their own lives and interests.

There is also clear overlap between the themes that emerge from the ways students talk about beneficial inclusive class elements and those elements that need improvement. From student comments in response to both open-ended questions, six common themes emerge relating to 1) class discussions, 2) diversifying content, 3) using inclusive language, 4) understanding student needs and flexibility, 5) student choice and the relevance of content, and 6) student interactivity in the course. These themes represent the most common themes in student responses, not an exhaustive list of the ways students interpret inclusiveness of courses.

These student perspectives on inclusive teaching align well with a framework for inclusive teaching presented by Marchesani and Adams. Their model includes four distinct categories of inclusive teaching intended to support instructors to develop more inclusive classes. The categories represent inclusive teaching broadly, but functionally for instructors regardless of discipline, modality, or class size.

Methods – “ What teaching methods will I employ?” In response to both questions, the most common responses comment on inclusive teaching methods, about the facilitation of class discussions, moderating dominating student voices, inviting other student voices and questions, and creating opportunities for student interactivity. Students also comment on the how they are included in the coursework and wanting more opportunities to engage with the content and interact with other students.

Content – “ What content will I convey?” Students commonly reflect a desire to read more diverse authors, to see representative examples in class, and to hear from more diverse guest speakers. Students cite the benefits of instructors adapting content to meet student’s goals or allowing opportunities for student choice in content or how they engage with content.

Students – “Who are my students and how will I find out?” Student comments indicate the impact of the instructor trying to know and understand their students, for example, using inclusive language (particularly around gender identity and sexuality) or being aware of student challenges and learning accommodations.

Instructor – “ Who am I and what are my assumptions?” Student comments about their instructor’s identity did not emerge as a common theme in student responses. However, there are less frequent comments about instructor’s beliefs, values, and assumptions impacting student experiences, especially when they contrast student beliefs and students feel their voices aren’t welcome.

Other student comments on inclusiveness

Student comments about inclusivity in the classroom are not limited to the two open-ended responses specifically linked to the selection of “inclusiveness” as either the most positive or negative learning element. In fact, 930 additional comments mention inclusivity as a response to other questions. Three topics in particular elicited the bulk of student mentions of inclusivity. Nearly 200 students mention inclusivity in their textual responses when selecting “instructor support” as the most beneficial element to their learning. On a similarly positive note, approximately 150 students who reported that no teaching practices were in need of improvement referred to inclusivity in their textual responses. Finally, roughly 150 students mention inclusivity in response to the open-ended question: “What else would you like to say about your learning experience in the course?” In these cases, students both praise and critique their experiences of inclusivity in the course.

Student demographics

The SES is an anonymous survey which limits our understanding of how identity impacts student perceptions of inclusive teaching. However, the survey does link student responses to demographic data about student academic level and gender. Among undergraduate students, male students were more likely to comment on the inclusiveness of the class, both about inclusiveness being beneficial (6.7% of male comments compared to 4.5% of female comments) and inclusiveness needing improvement (2.6% of male comments, 1.3% of female comments). Similarly, male graduate students were more likely to comment on the inclusiveness as a beneficial teaching practice (5.4% of male comments compared to 4.3% of female comments) but female graduate students were more likely to comment about inclusiveness needing improvement (1.7% of male comments 2.0% of female comments).

Differences by school and class size

Very large classes (over 200 students) are the most likely to receive comments about inclusiveness needing improvement, and the least likely to receive comments about inclusiveness being beneficial. There are relatively small differences in the frequency of student comments by school and college. Furthermore, differences may indicate that inclusiveness is a less salient feature to students rather than that there are meaningful differences in how courses are taught. Students in CAS Natural Science courses and College of Design courses are the least likely to leave comments about inclusiveness as a beneficial teaching practice. Students in Law, Journalism, and Music and Dance are the most likely to leave comments about inclusiveness needing improvement.

Impact of COVID

The shift in teaching and the challenges of the pandemic influenced student responses to the SES. The SES was launched in Fall of 2019 but was on pause during Spring 2020 due to the pandemic. Of the five terms of data included in this analysis, classes during the three terms of the 2020-21 academic year were significantly impacted by COVID. During the pandemic, students were slightly less likely to select “inclusiveness” as the most beneficial (5.7% in 2019-20, 4.7% in 2020-21) or as the most in need of improvement (2.0% in 2019-20, 1.5% in 2020-21) suggesting “inclusiveness” was less salient than other elements of their courses during COVID. Of all comments analyzed in this report, 37.5% of student comments on inclusiveness were from terms that were significantly impacted by COVID. However, the themes identified in this report are present in student comments both before and during the pandemic.

*This ordered list is created by looking at the most relevant words via tf-idf for inclusiveness responses and removing words that were relevant/common among all SES questions. Bolded words are chosen based on what the authors thought is interesting and/or actionable. More information can be found in an upcoming methods paper.