COTE NOTE
The Center for Online Teaching Excellence What I know about Development and Implementation of an Enterprise Assessment Service
Thomas Slomka Thomas Slomka is a staff member of the Center for Educational Innovation at the University at Buffalo. His expertise is in communication and information design, with particular interests in the intersection of traditional, hybrid and electronic communication patterns. Thom has a goal of advancing teaching, learning and research at UB through the effective use of information technology. At UB he develops specialized software to enhance teaching and research; supports the universities ability to support, track, assess, respond to and manage teaching and research goals. Areas of Interest include: Social Collaborative Networks; Shared Semantic Workspaces; Modularized Access to Digital Repositories; Course, Program and University Assessment; Learning Space Design; and the Intersection of Traditional and Digital Experience Design
“The service took on a
life of its own, building a small culture of sharing and a desire for selfimprovement and that was seen as a very good outcome.
”
I would like to share what I know about developing and implementing an enterprise assessment service The University at Buffalo’s Annual Academic Program Assessment is a web application designed to help programs provide assessment reports, program assessment coordinators to share best practices and resources, faculty and staff to compare assessment efforts and administration to track and report on program learning outcomes and assessment based program planning. The Online Annual Academic Program Assessment grew out of the Middle States Commission on Higher Education’s requirements for accreditation and efforts to build a culture of assessment on campus. In an effort to make the program reporting process as productive as possible a report viewing website was created. The University response to the report viewing web site was quite positive resulting in many programs choosing to refine their program reports and share assessment expertise with others on campus. With that success in mind a redesign effort was begun to look at ways to make the reports web site even more productive. The result of the redesign effort is the Online Annual Academic Program Assessment web application.
What is it The Online Annual Academic Program Assessment has a primary goal of creating a longitudinal archive of program development, assessment, and planning tools to provide the university community and administration access to the ongoing effort of university programs assessment of student learning. The University has an additional goal of providing direct support to programs through the use of the web applications reporting tools, reporting resources, interactions between assessment program coordinators, and interactions with program assessment reviewers and assessment support staff.
How it works The Annual Academic Program Assessment is a role based web application which provides a record of annual reporting on the student learning outcomes for the programs at UB and documentation of the programs design, assessment, and planning. The primary roles in the application are: Assessment Coordinator, Assessment Evaluator, and Public and Member Viewers. • The Assessment Coordinator is responsible for building the annual report by adding: program goals; accreditation details, learning outcomes, assessment details for each outcome, program planning for each cycle, and supporting documentation. • The Assessment Evaluator is responsible for reviewing a program report and, through an interactive rubric, providing feedback to the assessment coordinator on nature of the report. The Evaluator can also leave comments for the assessment coordinator. • The Public viewer can only see the basic program details, the list of program learning outcomes, and the service resources. The member viewers, faculty and staff at UB, can see the complete report but cannot see the report evaluation or evaluation comments.
What I did The application was designed to provide the following capabilities: • Web Based Template: The templates give program reports a common structure, which enhances the ability to review and compare assessment and planning efforts across programs. • Role Based Access: The assignment of responsible parties involved in the creation, maintenance, and assessment of the reports and the reporting process. Similar to the web tempting making the
The Open SUNY Center for Online Teaching Excellence
September 23, 2015 • Volume 4 • Issue 1
COTE NOTE Staff The COTE Community Team: Alexandra M. Pickett, Associate Director, Open SUNY; Martie Dixon, Assistant Academic Dean, Distance Learning & Alternate Programs, Erie Community College; Patricia Aceves, Director of the Faculty Center in Teaching, Learning & Technology, Stony Brook University; Lisa Dubuc, Coordinator of Electronic Learning, Niagara County Community College; Christine Kroll, Assistant Dean for Online Education, Graduate School of Education, University at Buffalo; Deborah Spiro, Assistant Vice President for Distance Education, Nassau Community College; Vicky Sloan, Distance Learning Coordinator, Clinton Community College; Erin Maney, Senior Instructional Designer, Open SUNY; Lisa Raposo, Assistant Director, SUNY Center for Professional Development This publication is produced by the Open SUNY Center for Online Teaching Excellence under the SUNY Office of the Provost.
Contact/Questions State University Plaza Albany, New York 12246 ContactCOTE@suny.edu
How to Submit Material This publication is produced in conjunction with the COTE “Fellow Chat” speaker series. Please submit a proposal at http://bit.ly/COTEproposal for consideration. Visit http://commons.suny.edu/cote for more information. To join COTE, visit http://bit.ly/joinCOTE
finished reporting more accessible the program assessment coordinators become transparent to the university community making inquiry and knowledge sharing more easily accessed. • Review Of Reports: An online peer review process, which applies an assessment rubric to each aspect of the report, intended to help cultivate and nurture an ongoing cycle of assessment and planning. • Resource Sharing: Create a wealth of available resources for the campus community to review and benefit from. Administrative Review: Provide university administration a window onto program assessment and reporting activities. • Enterprise Planning: Additional design and development plans include connecting data services from the application to other services on campus such as the course catalog, course syllabus tools, curriculum mapping tools, and accreditation services.
How I did it The web application is built with HTML5 and CSS3 and Java Script for client side display, runtime interaction, and connectivity to server side processing. Application logic is written in Perl Script and PHP. File and data management is taken care of by Apache web services and MySQL data services. The interface is designed to be responsive to user desktop, laptop, and tablet computing spaces and function in MacOS, WinPC, Android, and iOS operating environments.
Why I did it In 2012 the University at Buffalo began preparing for it’s Decennial Re-accreditation with the Middle States Commission on Higher Education. Three of the requirements for accreditation focused on assessment of student learning, building a culture of assessment on campus, and demonstrating the use of assessment results in program and university planning. Committees were formed and efforts were organized to both write narratives and collect supporting evidence for the university self study. Two web applications were created: An interactive road map which linked the primary requirements to the self study and associated supporting evidence with the statements of compliance and which was inter-linked to the second web application; An interactive appendices, provided a means of finding evidence and other supporting documents in the self-study. In preliminary review of the self-study, these two web applications were very well received by the UB community and were reported to be very helpful and easy to access by the review team. It was decided that we would use a similar approach to collect program assessment reports and make them more visible through a web interface. Like the previous two applications the focus was to allow users to easily find program learning outcomes, assessment reports and supporting documentation such as, curriculum maps, rubrics and other evidence.
What happened when I did it The University’s response to the program report web site was quite positive resulting in many programs choosing to refining their program reports once they were displayed in the interface, and prompt a sharing of assessment expertise with others on campus. When we took the project on there was no expectation that the service would be anything more than a finding aid for the Middle States review team and would have a limited life. The interface was really a simple finding aid, making visible the hundreds of programs within the university hierarchy and linking to them the thousands of outcomes and supporting documents. The service took on a life of its own, building a small culture of sharing and a desire for self-improvement and that was seen as a very good outcome. With that success in mind a redesign effort was begun to look at ways to make the reports web site even more productive. The result of the redesign effort is the online Annual Academic Program Assessment web application.
This publication is disseminated under the creative commons license AttributionNoncommercial-Share Alike 3.0.
The Open SUNY Center for Online Teaching Excellence
September 23, 2015 • Volume 4 • Issue 1
COTE NOTE Links
What I learned
The Center for Educational Innovation @UB
We are in the third cycle of reporting: the first was a largely traditional collection process with a web based display; the second was an implementation of the first cycles reports as a web finding aid; and the third a fully supported application with, administrative reminders, training in use of the service, and promotional and operational meetings with department heads and related campus services.
Annual Program Assessment & Program Evaluation
• Primarily we have learned that a key component in building a culture of continuous assessment is providing direct evidence that the work put into the reporting process is visible, is desired, is used by the campus, will reduce future work by providing a resource, and will enhance the university through shared access with other campus services.
Annual Academic Program Assessment Portal For more information go to
• We learned a key to continuous assessment is planning. The assessment process is a life cycle that renews with planning–as a plan is executed, assessed, reviewed, and reported on a new plan is generated. Our focus is to continue to refine the program reporting application so the process of reporting becomes a meaningful and enabling piece of the assessment life cycle.
http://www.buffalo.edu/ubcei/ services/assessment.html
“The assessment process is a life cycle that renews with planning - as a plan is executed, assessed, reviewed, and reported on a new plan is generated.
”
• We also learned that we need to continue to seek partnerships with programs and faculty to continue the process of sharing and hope to do that through an evaluation committee and peer assessment of reports and begin to link course level activity with program level success.
How others can use it The Center for Educational Innovation has produced significant impact in the areas of student learning assessment at UB (university-wide, department, program and course) and is available to external audiences through a coordinated set of fee based consultation services in the following concentrations: • Accreditation Consultation: Middle States Commission on Higher Education (MSCHE) standards of accreditation: preparing your campus for reviews; building and training teams to address accreditation standards; identifying, organizing and presenting evidence of compliance. • Assessment Consultation: Consultation on student learning assessment at all levels (universitywide, department, program and course) from organizing efforts to selection and use of in-house and third party assessment solutions. • Program Planning Consultation: Consultation on program planning and annual reporting in compliance with the Middle States Commission on Higher Education (MSCHE), specialized accreditation and comprehensive program review. • Presentations, team facilitation, and workshops are provided for half day and full day events. Materials are developed expressly to meet your needs.
This publication is disseminated under the creative commons license AttributionNoncommercial-Share Alike 3.0.
The Open SUNY Center for Online Teaching Excellence
September 23, 2015 • Volume 4 • Issue 1