Date of Award
Spring 5-2025
Document Type
Thesis
Department
Electrical Engineering and Computer Science
First Advisor
LouAnne Boyd, Ph.D.
Second Advisor
Chelsea Parlett, Ph.D.
Third Advisor
Elizabeth Stevens, Ph.D.
Abstract
Closed-source large language models (LLMs) developed by large technology companies continue to grow in popularity. However, ethical conversations surrounding the safety of model outputs have been a prominent topic of discussion. This project aims to assess three leading closed-source LLMs: OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude, to analyze how their outputs perform when treated as a subject of several psychological evaluation scales measuring biased behaviors against women. The Ambivalent Sexism Index, Modern Sexism Scale, and Belief in Sexism Shift evaluations were used to get descriptions of how the LLMs respond to traditional and modern prompts involving sexism and gender bias. The three evaluations used Likert scale response scores, providing quantitative scoring data. Results from evaluation trials were obtained using each LLMs API, collecting Likert scores in response to the evaluation prompts. Free-response data was also collected to understand output reasoning. Ordinal regression modeling with mixed effects aim to further understand how certain variables affect scoring. To understand patterns in reasoning, thematic analysis of the free response data was completed. Three significant themes were found across all model responses: recognizing women’s challenges, understanding variation in gender experience, and feminism and progressive initiatives. These themes emphasize the ways in which LLMs respond to biased statements against women.
DOI
10.36837/chapman.000655
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Recommended Citation
S. T. Fieck, "An analysis of bias towards women in large language models using Likert scale evaluations," M. S. thesis, Chapman University, Orange, CA, 2025. https://doi.org/10.36837/chapman.000655
