Office of the Provost
Handbook for Outcomes Assessment Using Learning Goals and Objectives Second Edition April, 2007
Compiled by Ralph S. Polimeni, Ph.D., C.P.A. Vice Provost for Accreditation and Assessment Tel 516-463-5685 E-mail: ralph.s.polimeni@hofstra.edu
This Handbook is intended for internal use only by Hofstra employees.
Introduction Much has been written on the need for colleges and universities to implement outcomes assessment programs, whether as part of the process to foster continuous improvement, or simply as a method to assure that students are learning what faculty expect them to learn. Most accreditation agencies now require that institutions accredited by them engage in assessment of their programs via learning goals (outcomes) and objectives. For example, the Middle States Commission on Higher Education revised its curriculum standards in 2002 to require that schools define learning goals and identify objectives which can be used to assess their realization. Learning goals represent the product of the education, not the process of educating. In order to serve students and faculty, learning goals, objectives and an assessment plan should be developed and implemented. Learning Goals What are learning goals? They are broad outcomes expected from an academic program, department or school. Ideally, they should flow from the mission statement of the program, department or school, and be aligned with the overall educational mission of the institution. More specifically, learning goals are the knowledge, skills, and attitudes that students should possess upon completion of an educational program. For example, some programs expect students to acquire specific knowledge or skills, while others strive at changing a student’s attitude by increasing their knowledge. Learning goals should focus on student and not teacher behavior, and describe a learning outcome, not the learning process (Suskie, 2004). Learning Objectives Once a program, department or school has agreed on a set of learning goals, more specific learning objectives need to be identified. Learning objectives describe what students must do to demonstrate proficiency in a given area, and they translate learning goals into descriptions of performance (Gainen & Locatelli, 1995). Whereas learning goals describe what a program aims to accomplish, objectives are the details that provide the operational plan for the attainment of these goals. Faculty will ultimately assess objectives in order to ascertain achievement of learning goals. Procedures for Assessment Using Learning Goals and Objectives The remainder of this handbook will outline steps to follow in the process of conducting assessment using learning goals and objectives. Before the process begins, an assessment plan must be developed. Assessment Plan It is commonly believed that the most effective assessment plans evaluate learning objectives at different stages as a student progresses toward the achievement of the learning goal(s). This approach is known as formative assessment. In that the student’s progress is measured at various points over time, the advantage to this method is that improvements can be made while the student is still in the program. Alternatively, a summative assessment evaluates the progress made by a student who is finishing the
2
program, and is usually done in the year the student is to graduate. Therefore, only those students who are still to complete the program benefit when the results of a summative assessment are implemented. Each learning goal and objective should be periodically reviewed and the assessment plan should state when and how goals and objectives will be assessed. Remember that not every learning objective needs to be assessed each year. Effective assessment plans include: • • • • • •
Multiple sources of assessment data. Objective review of goals, objectives, and assessment measures. Distribution of data to stakeholders (students, administrators, alumni, employers, etc.) as a basis for course and curricular development. A statement indicating how often formative or summative assessment data will be compared with program objectives. Systematic analysis of assessment data to understand what students are learning. A statement indicating the percentage of faculty participation in assessment activities (Calderon, Green, & Harkness, 2005).
The following assessment loop can be used as a blueprint for an assessment plan. Each stage of the loop will be a step to follow in the process. In order to monitor your progress, it is very important to establish a timeline at the beginning of this process for the completion of each step.
Assessment Loop The overall process can be viewed as a continuous assessment loop as follows: Develop a Mission Statement for the Program, Department or School (Step 1) Establish Learning Goals and Objectives (Step 2) Map Learning Goals and Objectives to Courses (Step 3) Determine Assessment Tools (Step 4) Gather Evidence and Interpret Findings (Step 5) Use Results for Improvement (Closing the Loop) (Step 6) Start Process Again (Step 7)
3
Step 1 – Develop a Mission Statement for the Program, Department or School The logical first step in this process is to develop a Mission Statement for the program, department or school. The Mission Statement: • is a broad statement of philosophy, role, scope, etc. • provides a general sense of identity for the program, department or school • states what we do and whom we do it for • is modified infrequently Note that the Mission Statement must be in agreement with the Hofstra University Mission Statement. The Mission Statement should be faculty driven and have input from as many stakeholders as possible. Suggestion: Many examples of Mission Statements in your discipline can be found on the Internet with a Google search entering: Mission Statement and your discipline. Step 2 - Establish Learning Goals and Objectives The development of learning goals and objectives must be faculty driven. Ideally, the faculty of each program, department or school should develop between five to fifteen major learning goals that include both general knowledge and content-specific learning goals. General knowledge learning goals may include critical thinking, written and verbal communication skills, information literacy skills, and organizational and interpersonal skills. Bloom’s Taxonomy of Educational Objectives (1956) and the newly created revised taxonomy (Anderson & Krathwohl, 2001), are excellent resources for general educational learning goals. Knowledge-specific learning goals will vary by discipline. Some accreditation agencies have developed specific learning goals that member institutions must assess. In many cases, learning goals and objectives will likely be introduced in a basic course and emphasized in one or more upper-level courses. Developing Learning Goals The following process may be helpful when developing learning goals: • • • •
Review Hofstra documents (bulletin, brochures, advertisements, etc.) that describe your program, department, or school. Identify what you expect a student will learn. Make sure that all learning goals are consistent with your mission statement. Each learning goal should be listed separately. Assessment will be difficult if they are combined.
The following table lists verbs that can be used in developing learning goals in accordance with Bloom’s Taxonomy.
4
Students will be able to: Knowledge Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline
Comprehension Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell
Application Apply Calculate Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Restructure Schedule Sketch Translate Use Write
Analysis Analyze Appraise Categorize Compare Debate Diagram Differentiate Distinguish Examine Experiment Identify Inspect Inventory Question Structure Separate Summarize Tabulate Test
Synthesis Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write
Evaluation Appraise Assess Choose Criticize Critique Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Validate
-Adapted from a presentation by Susan Hatfield at the Annual Conference of the Higher Learning Commission, April, 2004.
This process should result in learning goals that reflect what faculty believe are the skills, knowledge, experiences and values at the center of their curriculum. Developing Learning Objectives Objectives should answer the following questions: • • • •
Content – What do students need to know? Action – How do we want students to use that knowledge? Context – Under what circumstances will students be expected to demonstrate their knowledge? Performance Criteria – What are the standards that will be used to judge students’ performance? (Gainen & Locatelli, 1995).
Examples of Learning Goals and Objectives Objectives should be matched to and listed under the learning goal to be assessed. Note that, while learning goals should not be combined, it is common to have more than one objective for each learning goal.
5
The following are examples of learning goals and objectives that could be developed for accounting majors: 1. Learning Goal: Accounting students will develop an awareness and appreciation of ethical issues. Objectives: a. Demonstrate the ability to detect ethical issues in case assignments and problems. b. Identify individuals or entities involved in an ethical dilemma and identify the responsibility to shareholders and the general public. c. Offer a solution to an ethical problem posed in an assignment. 2. Learning Goal: Accounting students will demonstrate the ability to think critically and solve unstructured problems. Objectives: a. Identify issues in a problem or case and offer solutions, both quantitative (financial analysis), and qualitative (essay or memo). b. Draw upon prior knowledge and apply concepts to provide solutions to complex accounting problems. c. Be able to evaluate the overall financial health of a company by analyzing its financial statements and provide solutions for problem areas. 3. Learning Goal: Accounting students will develop proficiency in oral and written communication skills. Objectives: a. Be able to prepare and deliver an essay on a current accounting topic. b. Be able to write a memo to a client describing a tax issue and discuss the implications of alternative tax treatments. c. Be able to construct an audit program that can be followed by a first-year auditor. d. Demonstrate the ability to write an internal memorandum discussing budget variances for a particular accounting period and the reasons for the variances. The following are two examples of general education learning goals and objectives that are in the development stage by HCLAS faculty: 1. Learning Goal: Students will demonstrate the ability to think critically and creatively. Objectives: a. Clearly and concisely summarize and evaluate the facts, presumptions, viewpoints, and arguments presented in a text or creative work. b. Gather and assess relevant information, and apply appropriate
6
cognitive methods in solving problems or answering questions raised in a text or creative work. Construct well-reasoned solutions or conclusions; test and defend conclusions against relevant criteria and standards. Critically analyze one’s own thinking, by identifying one’s presumptions and viewpoints, as well as problems, inconsistencies, and unanswered questions. Imagine and defend alternative hypotheses and viewpoints; explain the rational reasons that led one to reject or provisionally retain these alternatives.
c. d.
e.
2. Learning Goal: Students will apply analytical reasoning across academic disciplines. Objectives: a. Read with comprehension, and apply critical reasoning to an understanding of literature. b. Critically and aesthetically analyze works in literature and the fine or performing arts. c. Engage in quantitative and deductive reasoning d. Apply the scientific method to investigate and analyze the natural world. e. Apply the scientific method to investigate and analyze human social behavior. f. Describe, comprehend, and analyze the role of philosophical ideas or historical movements in the development of civilization. Step 3 – Map Learning Goals and Objectives to Courses Once the learning goals and objectives are established, they should be linked to the individual courses offered in the curriculum. The purpose of this “mapping” is to assure that all learning goals and objectives are covered in the curriculum. Usually a simple spreadsheet is all that is required to complete the mapping process. For example:
Accounting Learning Goal 1: Course Obj. 1 Obj. 2 Obj. 3 etc. 101 102 123 Etc.
0 1 2
2 1 2
Learning Goal 2: Obj. 1 Obj. 2 Obj. 3 etc.
1 0 1
1 3 1
Coverage: 0 = none; 1 Minimal; 2 Moderate; 3 Extensive
7
0 1 2
2 1 2
Etc.
Most courses will not cover all the learning goals and objectives; however at least one learning goal and objective should be linked to each course, otherwise the value of the course should be addressed. All learning goals and objectives that are linked to a specific course should be listed on the course syllabus. Step 4 – Determine Assessment Tools There are many types of tools available to assess learning objectives in order to ascertain whether learning goals have been met. The type of learning objective should dictate the assessment tool. For example, if a learning objective refers to communication skills, a random sampling of student oral presentations using a rubric (to be discussed later) might be an effective tool. If a learning objective states that a student will improve in a specific skill, then a pre-test and a post-test are commonly performed. It should be emphasized that learning objectives are assessed in order to ascertain achievement of learning goals. A common question often asked by faculty and administrators is “Why aren’t course grades an effective assessment tool?” Course grades are determined by many factors which do not always directly relate to a learning objective. For example, in addition to course examinations, an instructor may consider attendance, class participation, group projects, student behavior in class, late assignments, etc. While all these factors are appropriate when determining a final grade in a course, they may not directly link to the learning goal(s) and objective(s) that were established for the course. It should also be emphasized that an assessment technique must be designed to measure whether or not a learning objective is being achieved. A very important point to remember is that the purpose of the assessment plan is to assess the program and not the performance of an individual instructor. Indirect versus Direct Assessment Techniques Indirect assessment techniques reveal only characteristics associated with learning and are usually less effective in determining whether or not a learning goal has been achieved (MCHE, 2003). Examples include surveys, questionnaires, and course evaluations. Direct assessment techniques of a student’s work usually give the best indication of student learning, since these methods provide evidence of command of a subject. The following are examples of tools that may be used for direct assessment of learning objectives: •
• •
Test questions – A very effective and popular method of assessing learning objectives. Full tests can be developed or selected questions embedded into a class examination. The questions must be directly related to the learning objective(s) that you desire to assess. If embedded questions are used, the results must be separated from the regular examination for analysis. Analysis of written assignments that relate to a specific learning objective. Analysis of capstone projects or research theses.
8
• •
E Portfolio* – Online collections of students’ academic work. Clickers* – Hand-held devices used to answer multiple choice questions. Results are immediately available to the instructor in graph form. Blackboard* - Offers a wide variety of question types that can be included in online course examinations. Review of student oral presentations. Selected analysis of student papers for proficiency in written communications. Educational Testing Service (ETS) – Used to assess learning objectives in a student’s major field. Major Field Achievement Tests (MFATs) may be taken in: Biology, Business, Chemistry, Computer Science, Economics, Education, History, Literature in English, Mathematics, Music, Physics, Political Science, Psychology, Sociology, etc. (Note: Hofstra is currently using the Biology, Political Science, and Psychology tests to ascertain the achievement of learning goals in those disciplines).
• • • •
ETS also offers a Measure of Academic Proficiency and Progress (MAPP) test. These tests measure college-level reading, writing, mathematics, and critical thinking skills. They are commonly used to assess general education objectives. •
Area Concentration Achievement Test (ACAT) – A standardized test bank is used to assess a program’s curriculum. ACAT tests may be taken in: Agriculture, Art, Biology, Criminal Justice, Geology, History, Literature, Political Science, Psychology, Social Work, etc.
•
Licensure Examinations – A good means of assessment. However, the results normally do not provide enough detail to make improvements to the program. Examples of licensure examinations: Bar, Certified Public Accountant, Counseling, Medical, National Teacher, Nursing, Social Work, etc.
If an assessment tool does not provide enough information to make improvements, then it has limited value. Often more than one assessment tool is selected to assess the same learning objective in order to validate the results. Remember: whatever tool is selected, the intention is to assess the program, department, or school in order to determine where and how improvements can be made. *More information on e-Portfolios, clickers and Blackboard can be found on Hofstra’s assessment website: login to the portal at my.hofstra.edu. Once there, click on the 'Support' tab, in the left column look under 'Faculty Resources' (second section from the top).
9
Using Rubrics Rubrics are commonly used as a tool to evaluate a student’s performance on case studies, term papers, oral presentations, etc. “A rubric is a scoring guide: a simple list, chart, or guide that describes the criteria that you and perhaps your colleagues will use to score or grade an assignment. At minimum, a rubric lists the things you re looking for when you evaluate a student assignment.” (Suskie, 2004). Steps in the Development and Use of Scoring Rubrics for Assessment 1 Select the learning goal you want to evaluate. 2 Choose the objectives of the learning goal you wish to assess. 3 Develop the performance criteria for scoring. 4 Determine the minimum score that will be acceptable before you begin the assessment. 5 Perform a small sample test using at least two faculty members to assess the same material to determine if results are similar. 6 Instruct faculty on use of the rubric. Example using a Rubric to Assess Achievement of a Learning Goal Assume a case study was selected by accounting faculty in order to ascertain achievement of the following learning goal that was previously presented: Accounting students will develop an awareness and appreciation of ethical issues. Students in a senior level accounting course were asked to provide a written analysis of the case, specifically addressing the three previously presented objectives related to the learning goal. The following is an example of a rubric that could be developed to ascertain the achievement of the above learning goal: Performance Outstanding Good Fair Unsatisfactory
Score 3 2 1 0
Learning Goal Accounting students will develop an awareness and appreciation of ethical issues. Objectives to Assess a. Demonstrate the ability to detect ethical issues in case assignments and problems………………………. b. Identify individuals or entities involved in an ethical dilemma and identify the responsibility to shareholders and the general public……………………………. c. Offer a solution to an ethical problem posed in an assignment……………………………………………….. Total Score
10
Score_ ______
______ ______
Faculty decided that they would consider a learning goal achieved if a minimum average score of 2 for each objective and an average total score of at least 6 was obtained for all objectives. (The example is continued in the appendix, which presents the results in a report form). Suggestion: Many examples of rubrics in your discipline can be found on the Internet with a Google search entering: Rubrics and your discipline. Step 5 – Gather Evidence and Interpret Findings When assessing a small group, all members of the group should be assessed. However, a sample is sufficient when dealing with larger groups. When samples are used, be careful to assure that the sample is representative of the entire group. The interpretation of the findings need not involve a sophisticated statistical analysis. For example, if a desired outcome is that 70% of the students assessed achieve a specific learning objective, then a simple average is sufficient. Faculty should determine the desired outcome and method of analysis. Step 6 – Use Results for Improvement (Closing the Loop) Closing the loop is perceived by many to be the most important step in the assessment process. A major objective of assessing a program using learning objectives is to find ways to improve the program. If the results are not used for improvement, then the process is of limited use. It is rare to have a situation where “everything is perfect,” with no improvements necessary. The following points should be considered during this step: • Procedures should be in place to facilitate and encourage change (for example, the results could be sent directly to the curriculum committee for action). • Improvements made should be responsive to the assessment findings. • Recommended improvements should be monitored to ensure implementation. • Sometimes assessment may result in the learning goals and objectives being modified or another assessment tool being selected if further validation is desired. Not every change needs to be significant; sometimes continuous improvement is best accomplished in small steps. Step 7 – Start Process Again Assessment is a continuous process and is never finished. Once improvements are made, and sufficient time has elapsed in order for the changes to take effect, a reassessment should be performed in order to ascertain achievement of a learning goal.
11
Reporting Assessment Activities It is important, especially for accreditation purposes, to keep track of all your assessment activities. The following form may be used to report the learning goals and objectives assessment activities of your program:
OUTCOMES ASSESSMENT REPORT Using Learning Goals and Objectives School: _____________________________ Program/Department/Major:____________ Academic Year Covered: ______________ Date of Report:_______________________ Step 1: Step 2: Mission Learning Goal Statement and Objectives Prog./Dept.
Step 3: Map to Courses
Step 4: Assessment Tool(s)
Step 5: Interpret Findings
Step 6: Use Results (Closing the loop)
SEE THE APPENDIX FOR A HYPOTHETICAL EXAMPLE OF THE ASSESSMENT LOOP USING THE ABOVE OUTCOMES ASSESSMENT REPORT
12
APPENDIX – HYPOTHETICAL EXAMPLE OF ASSESSMENT LOOP
OUTCOMES ASSESSMENT REPORT Using Learning Goals and Objectives School: Business Program/Department/Major: Accounting Majors Academic Year Covered: 2006/07 Date of Report: May 2007 Step 1: Step 2: Mission Learning Goal and Objectives Statement Prog./Dept. Throughout each level of instruction, the department recognizes the importance of high ethical standards in the practice of accounting
Step 3: Map to Courses
Step 4: Assessment Tool(s)
Step 5: Interpret Findings
Learning Goal: Accounting students will Accounting: Seniors were asked to Students scored an develop an awareness and appreciation of 101,102,123, analyze a case study average of 2.3 on objective (a) and a 124, on ethics. Faculty ethical issues. 2.2 on objective 125,131,134, decided that they Objectives: (b). However, a. Demonstrate the ability to detect ethical 143 and 144. would consider a students scored learning goal issues in case assignments and problems. only a 1.8 on achieved if a b. Identify individuals or entities involved objective (c). The minimum average in an ethical dilemma and identify the average total score score of 2 for each responsibility to shareholders and the was 6.3. objective and an general public. average total score of c. Offer a solution to an ethical problem at least 6 was posed in an assignment. obtained for all objectives.
The average scores for the completed rubric are presented below: Performance Outstanding Good Fair Unsatisfactory
Score 3 2 1 0
Learning Goal: Accounting students will develop an awareness and appreciation of ethical issues. Objectives to Assess Score_ a. Demonstrate the ability to detect ethical issues in case assignments and problems…………………………….. 2.3 b. Identify individuals or entities involved in an ethical dilemma and identify the responsibility to shareholders and the general public………………………………….. 2.2 c. Offer a solution to an ethical problem posed in an assignment……………………………………………………… 1.8 Total Score 6.3
13
Step 6: Use Results (Closing the Loop) To address the weakness found in objective (c), the curriculum for courses 125 and above will be enhanced to place a greater emphasis on solutions to ethical problems.
This Handbook is intended for internal use only by Hofstra employees. Portions of this Handbook were taken from a working paper titled “Continuous Improvement for an Undergraduate Accounting Program – an Assessment Model” by Sheila A. Handy, Lafayette College and Ralph S. Polimeni, Hofstra University which is currently under review by a journal.
REFERENCES Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom s taxonomy of educational objectives. New York: Longman. Angelo, T., & Cross, K.P. (1993). Classroom assessment techniques. San Francisco: Jossey-Bass. Bloom, B., ed., (1956). Taxonomy of educational objectives: Book 1: Cognitive domain. New York: David McKay. Calderon, T., Green, B.P., & Harkness, M. (2005). Best practices in accounting program assessment; Executive summary, AAA Teaching and Curriculum on best practices in accounting program assessment. Advances in Accounting Education 7 277-306. Gainen, J., & Locatelli, P. (1995). Assessment for the new curriculum: A guide for professional accounting programs. Accounting Education Series, Volume No. 11. Sarasota: American Accounting Association. Herring, H.C., & Izard, C.D. (1992). Outcomes assessment of accounting majors. Issues in Accounting Education 13(4) 1-17. Middle States Commission on Higher Education (MCHE). (2003). Student learning assessment options and resources. Philadelphia, MCHE. Norton, S.M., & Dudycha, A.L. (2001). Accountability and integration in assessment: identifying learning goals. Academic Exchange Quarterly 5 (1) 38-42. Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton: Anker.
14