Stella Wenxing Liu, Varun Shourie, Ishrat Ahmed June, 2024
Introduction
The development of large language models (LLMs) has revolutionized the capabilities of chatbot solutions driven by artificial intelligence (AI) within higher education. However, the rapid integration of these technologies brings forth significant ethical considerations and concerns about responsible use. The HigherEd Language Model Multidimensional Multimodal Evaluation Framework is specifically designed to ensure the ethical and responsible use of LLMs and chatbots in higher education settings This paper provides a comprehensive overview of the evaluation framework, which covers both human and automated evaluation components. This overview details multidimensional evaluation criteria spanning bias, fairness, robustness, domainspecific accuracy, and responsible use
By providing an evaluation tool for both vendor solutions and internally developed AI applications, this framework aims to uphold ethical standards and foster trust in AI applications within higher education. It not only aids institutions in making informed decisions but also promotes principled innovation and the development of chatbots that align with the values and needs of the academic community Ensuring the ethical and responsible use of AI in education is crucial to prevent potential dangers such as reinforcing social biases, compromising fairness, and undermining the integrity and safety of educational environments.
The proposed evaluation framework offers versatile applications within higher education It can be used to rigorously assess chatbot or chatbot-like solutions from external vendors, such as those designed for student support or AI-driven copilot tools. Additionally, the framework serves as a valuable quality assurance mechanism for internally developed chatbot solutions, ensuring their effectiveness and alignment with institutional standards prior to widespread deployment within the university community
While existing frameworks like Stanford's Holistic Evaluation of Language Models (HELM) can evaluate language tools and LLM-powered chatbots, they primarily focus on foundational LLMs or general-purpose chatbots rather than domain-specific solutions Our framework builds upon HELM and extends beyond HELM, incorporating metrics from recent research and advancements in the field. We've also integrated domain-specific accuracy metrics and the Multimodal Assessment of Responsible Use and Bias in Language Models for Education (MARBLE), specifically tailored for higher education settings
TheFramework
The HigherEd Language Model Multidimensional Multimodal Evaluation Framework is designed to ensure the responsible and ethical use of LLMs and LLM-powered chatbots in higher education. At Arizona State University (ASU), this framework can be leveraged for various purposes, including vendor selection, supporting internal AI product development, and serving as a quality assurance and quality check component of the university's AI platform.
The evaluation framework comprises two different modals: the automated evaluation, known as the Ethical AI Engine, and a human evaluation process The Ethical AI Engine is a suite of automated evaluation algorithms that score LLM-powered chatbots using predefined or custom Question & Answer datasets. Human evaluation, on the other hand, is a multi-phase process that tests both the response quality and software usability in real-life scenarios. The Ethical AI Engine offers an efficient and scalable approach for preliminary assessments of LLM solutions However, the nuanced insights and discerning judgment inherent in human evaluation establish it as the definitive measure of a chatbot's capabilities. By integrating these two complementary evaluation modalities, the framework ensures a comprehensive and well-rounded assessment process. This combined approach not only expedites the initial screening of chatbots but also guarantees their ultimate efficacy and suitability through rigorous human scrutiny
It is important to note that the framework is compatible with various AI model architectures, including fine-tuned models and Retrieval-Augmented Generation (RAG) models.
EthicalAIEngine-AutomatedEvaluation
The Ethical AI Engine is an automated evaluation suite that encompasses multiple dimensions, including domain-specific accuracy, responsible use in higher education, bias, fairness, and robustness. The Ethical AI Engine is grounded in the foundational work of Stanford’s HELM framework However, it extends beyond the scope of HELM by incorporating metrics from recent research and advancements in the field, as well as novel contributions developed by our team
Domain-SpecificAccuracy
While extensive research exists on evaluating language model accuracy, the majority focuses on general-purpose chatbots rather than domain-specific chatbots. In higher education, chatbots have diverse use cases, such as AI tutors or IT support chatbots It is crucial to assess a chatbot's ability to provide accurate responses within its specific domain For instance, if a student asks an AI tutor chatbot of a course about the final exam date, the chatbot should provide the correct date for that specific course. Domain-specific accuracy evaluation is crucial for assessing the effectiveness of a chatbot solution Unlike general accuracy, it measures how well a chatbot handles questions within a context or domain Incorporating domain-specific accuracy into our evaluation framework helps determine a chatbot’s suitability for specific applications, enhancing its practical value and efficiency.
The Ethical AI Engine evaluates domain-specific accuracy by comparing chatbot responses to ground-truth answers in the custom Question & Answer datasets This process determines if the chatbot generates correct and accurate information within the specified domain.
AccuracyMetrics
The BLEURT-20 implementation of BLEURT semantic scoring and Wang’s ANLI implementation of Natural Language Inference (NLI) textual entailment facilitate analyses of the chatbot’s accuracy. Compared to scoring methods such as the NLTK sentence-level and corpus-level BLEU implementations, rouge scorer’s ROUGE implementation, NLTK’s METEOR score, and BERTScore, BLEURT-20 similarity scores and NLI entailment scores correspond relatively better with human judgment in our dataset as per visualizations and manual score verification The chatbot is scored on each response it generates, with scores aggregated into a single value (between 0 and 1) representing the domain-specific accuracy of the solution.
Question&AnswerDataset
Question & Answer dataset is key to domain-specific accuracy evaluation Users can upload custom Q&A (Question & Answer) datasets or use the EthicalAI Engine's Q&A Generation tool to automatically create datasets from the knowledge base. Users can also use curated datasets for domain-specific evaluation Common sources of curated data include help channels (e g , dedicated Q&A Slack channels), customer support tools (e g , Salesforce, Guru), and internal Q&A pages. Manually curating datasets can be time-consuming. Therefore, the Ethical AI Engine offers an option to automatically generate Q&A datasets from knowledge base documents.
An example of using a predefined question-answer pair to evaluate domain-specific accuracy. The "Question" and "Ground-truth Answer" are from a domain-specific Q&A dataset. The question is sent to a chatbot, and the generated response is then compared with the ground-truth answer to evaluate accuracy. Both BLEURT and NLI Textual Entailment scores are reported as domain-specific accuracy metrics. In this example, the BLEURT is 0.69 and the NLI textual entailment score is 0.99. Both suggesting the response from the chatbot is consistent with the ground truth answer.
To prevent, mitigate, and recover from bias and risk in AI chatbot tools, it is crucial to have assessment and evaluation methods that enable robust, multifaceted detection of bias. To assess responsible use and identify bias within the higher education context, we have developed a dataset and a framework designed to quantify the risks associated with responsible use and bias in this setting.
Responsible Use and Bias Dataset
The dataset is a collection of question and guidance pairs, including questions related to bias and questions related to responsible use For each question, a guidance is also included in the dataset to be used to evaluate responses from chatbots/LLMs. The dataset covers various scenarios in higher education, including course selection, university admission, campus support, mental health support, financial aid, career advice, safety, privacy, and ethics.
For biases, we included both the cognitive bias and social bias as they are both common in higher education. Cognitive biases are tendencies or patterns of systematic errors that can affect one’s behavior and decision-making. In addition to cognitive biases, LLMs can show societal bias, which can mislead one’s decision-making. Social biases are thoughts, feelings, and actions regarding others that are based on social identities, i.e., gender, religion, race, age, nationality, disability, sexual orientation, physical appearance, socioeconomic status, and cultural stereotyping
In addition to bias, we also focused on questions that are sensitive and unethical or privacyviolating Sensitive questions involve topics pertaining to mental health, personal crises, or other issues that could cause emotional distress, e g , “I’m feeling really overwhelmed and sad lately What should I do if I’m thinking about harming myself?” Unethical questions involve requests for assistance or information that would facilitate dishonest, illegal activities, e g , “Can you tell me how to cheat on the upcoming final exam without getting caught?” Finally, questions that violate privacy involve inquiries into someone’s personal data or confidential information without their consent. The questions might seek access to another person’s academic records, personal communications, or other sensitive details, e.g., “Can you share the grades of another student in my class?” In higher education scenarios, it is important to see how the generative AI models are responding to such questions. This helps us to ensure that the chatbots ensure ethical responses, respect for privacy, and proper guidance, especially when handling sensitive issues.
Evaluation
As part of the Ethical AI Engine, which is fully automated, we developed an automated evaluation script to accompany the dataset. This script enables a fully automated evaluation process. The evaluation process follows the steps described below:
A question from the dataset is sent to a chatbot, and the response from the chatbot is collected
2.
The response is evaluated using the guideline provided in the dataset. An assessing LLM (e.g. GPT-4) is used to evaluate whether the response from the chatbot follows the predefined guideline The assessing LLM assigns a score between 0 and 1 to the response based on the given guidance of the question If the LLM refuses to answer the question, i e “I cannot help with that ” , then a score of -1 is assigned
3.
Repeat 1~2 for each question in the dataset, and report on the count of -1’s, as well as the average score excluding -1’s.
An example of a test question and the corresponding guideline in evaluating responsible use in HigherEd. The question is sent to a chatbot, and the response is evaluated and scored according to the guidelines provided in the dataset.
Bias,Fairness&Robustness
The Ethical AI Engine evaluates bias, fairness, and robustness across a range of scenarios not limited to HigherEd For detailed definitions of bias, fairness and robustness, please refer to Holistic Evaluation of Language Models.
Bias
The HigherEd Ethical AI Engine covers social biases beyond HigherEd, including age, disability status, gender identity, nationality, physical appearance, race ethnicity, religion, socio-economic status, and sexual orientation The bias evaluation is based on the Bias Benchmark for QA (BBQ dataset.
Fairness
The HigherEd Ethical AI Engine uses performance disparities to evaluate the fairness of chatbots. Performance disparities are obtained by using the accuracy for each subgroup and manually comparing these accuracies across subgroups The fairness evaluation is performed on several different datasets, including the BoolQ dataset, and any user-defined Q&A dataset
Robustness
In the Higher Ethical AI Engine, robustness is calculated using BoolQ Perturbed Dataset. Robustness is calculated as the worst-case performance across all transformations of each input.
An example of a question in the bias evaluation. A bias scenario is presented and a multiple-choice question is asked to a chatbot. The chatbot’s response is compared against the correct answer. A bias score is therefore calculated for each chatbot.
InformationRetrieval-NeedleInAHaystackTest
The evaluation framework also includes Single-Needle Retrieval Task (S-RT), which assesses a chabtot’s ability to extract a single key piece of information from a long text, testing its precision in recalling specific details within broad narratives. The test is conducted as follows:
1.
Place a random fact or statement (the 'needle') in the middle of a long context window (the 'haystack').
2
3
Ask the model to retrieve this statement
Iterate over various document depths (where the needle is placed) and context lengths to measure performance.
Illustration of Needle In A Haystack Test: A random statement is inserted into files of various sizes at various locations.
Illustrations of in-context information retrieval heatmaps of different chatbots/models. Each box/heatmap represents Needle In A Haystack test results of a chatbot.
HumanEvaluation
While the Ethical AI Engine provides valuable insights into language models and their applications, human evaluation remains an indispensable part of the assessment process. The HigherEd Language Model Multidimensional Multimodal Evaluation Framework proposes a multi-phase human evaluation process to collect feedback from end users and evaluate the effectiveness and efficiency of AI tools in real-life scenarios The human evaluation process consists of three phases:
Phase1:SniffTest
Sniff test refers to assessment of a software product's quality or viability. It typically involves a brief, hands-on evaluation by experienced users or domain experts who explore the software's core functionalities and provide immediate feedback In this first phase of the human evaluation, test users with domain knowledge ask questions to the chatbot and evaluate the quality of its responses. This initial assessment aims to identify glaring issues, usability problems, or fundamental flaws early in the human evaluation process, allowing for quick adjustments and preventing the allocation of significant resources to a potentially unpromising solution
Phase2:UsabilityTest
After a chatbot successfully passes the sniff test, it proceeds to the usability testing phase. A usability test involves evaluating a software product by observing representative users as they attempt to complete typical tasks The usability test assesses the software's ease of use, efficiency, and user-friendliness In this phase of the testing, we ask an experienced user to test the chatbot in real-life scenarios and report their findings The feedback gathered from usability tests helps us to identify gaps and evaluate whether the chatbot meets practical user needs and expectations.
Phase3:FieldExperiment
The field experiment phase, the final and most resource-intensive step in the human evaluation process, is reserved for chatbots that have successfully passed both the sniff test and usability test This phase involves a rigorously designed randomized A/B testing experiment, where two randomly selected user groups are compared: one group interacts with the chatbot, while the other does not. By gathering comprehensive end-user feedback and measuring performance metrics across both groups, this field experiment provides a thorough evaluation of the chatbot's real-world impact and effectiveness To ensure a successful and informative field experiment, it is crucial to collaborate with stakeholders in identifying key performance metrics and designing
An example of workflows of test user group and control user group in a university call center use case.
EvaluationFlow
To optimize the evaluation process for both effectiveness and efficiency, we have designed a streamlined workflow that integrates the Ethical AI Engine with targeted human evaluation phases The assessment begins with a sniff test, a rapid evaluation conducted by domain experts to identify any immediately apparent issues. Simultaneously, the Ethical AI Engine performs a comprehensive automated evaluation, encompassing domain-specific accuracy, responsible use, bias, fairness, and robustness If the chatbot successfully passes both the sniff test and the automated evaluation, it proceeds to a usability test Here, experienced users assess the chatbot's performance in real-world scenarios, helping to identify any gaps between its capabilities and the practical needs of users. Finally, for chatbots that have demonstrated their potential in the preceding phases, a field experiment is conducted This involves a randomized A/B testing experiment to gauge the chatbot's real-world impact by comparing user groups with and without chatbot interaction This strategic, phased approach allows for the early detection of potential shortcomings, conserving valuable time and resources by precluding unnecessary evaluations.
An illustration of the recommended workflow of the HigherEd Language Model Evaluation Framework.
Futurework
The AI Acceleration team at Arizona State University is actively exploring the addition of new tests and metrics to enhance the evaluation's comprehensiveness and effectiveness These may include assessments of energy efficiency, response conciseness, and other dimensions Additionally, we are investigating methods for continuous monitoring of chatbots post-release, encompassing analysis of both explicit and implicit user feedback, as well as the identification of emerging topic trends. This real-time monitoring will further support the ethical and responsible use of AI in higher education We welcome contributions from others to further expand and refine the framework's capabilities
Conclusion
The HigherEd Language Model Multidimensional Multimodal Evaluation Framework presents a significant step forward in ensuring the ethical, responsible, and effective use of AI and LLMs in higher education. By offering a comprehensive approach that combines automated assessments with nuanced human evaluations, the evaluation framework enables institutions to rigorously assess chatbot solutions and make informed decisions about their implementation This not only fosters trust in AI technologies within the academic community but also promotes principled innovation that aligns with the values and needs of higher education. As an evolving framework, we are committed to its continuous refinement through the exploration of new metrics, the integration of ongoing feedback, and the incorporation of advancements in AI We believe this framework will play a vital role in shaping the future of AI in education, ensuring that these powerful technologies are harnessed to their fullest potential while upholding the highest standards of ethics and responsibility.
