Framing the Review

JASON BERGERON

As campuses increase efforts to enhance and transform the fraternity and sorority experience, external reviews increasingly emerge as mechanisms to identify gaps and inform solutions. By simplification (or one could argue, oversimplification), an external review often consists of assembling a team of content experts that, through a combination of virtual and on-site data collection, evaluates the current effectiveness of a fraternity and sorority program. This often leads to identifying key areas for improvement and heightened effectiveness.

External reviews, however, are more complex than that, as they hold a unique space within student affairs and higher education. Specifically, that complexity rests at the intersection of research, assessment, and evaluation, including important elements of all three concepts. There is no “guide book” for external reviews. Campuses are often left to their own devices to figure out how to best navigate the planning, implementation, and post-review expectations of an external review. While certainly not an expert on the topic of external reviews, I have been privileged to manage external reviews from both sides: as an external reviewer and coordinator for campus-based external reviews. Those experiences — coupled with a deep, almost unexplainable affinity for assessment work — inform the following makeshift “roadmap” that can help campuses determine if an external review is the right decision and if so, how to effectively navigate it.

Clarifying Reason & Intent

An external review can often be the first strategy thrown on the table in several different instances:

1. When confusion or challenge about the fraternity and sorority experience and/or effectiveness of fraternity/sorority advising and operations exists

2. When a change in departmental/divisional/ institutional leadership occurs and new leadership attempts to better understand operative effectiveness

3. As a tool in a larger planning or change initiative effort

While value exists for each of those rationales, it is important to have clarity on the proposed review’s purpose and intended impact. Understanding who commissioned the review and their desired expectations for the process’ outcomes helps provide direction for the purpose. A review commissioned by divisional/university leadership to better understand the successes and challenges within the fraternity and sorority community may yield different outcomes than an external review considered as part of a larger assessment strategy.

The reality of external reviews is they utilize multiple limited resources: time, financial, and attention from others. External reviews can be an important component of a larger assessment and evaluation strategy. An external review should be designed to complement, but not replace or supersede, current campus assessment and evaluation practices. In the same way a campus may establish a timeline for other important assessment activities, establishing a regular timeline for external reviews allows it to become an ongoing part of larger assessment and evaluation strategies as opposed to something more episodic. A five to seven year cycle can be an effective timeline for conducting a review, utilizing the review for long-term enhancement planning, and resting/maintaining.

Framing the Review

Framing the review by clearly articulating the lens through which external reviewers should approach the process can be one of the single greatest strategies for increasing review effectiveness and usefulness. Effectively framing the review prevents it from sliding into a “let’s bring people together so they can tell us what they think” methodology. While that approach might be valuable and honors reviewers’ individual perspectives, it does not always translate into a final product grounded in relevant fraternity/sorority/student affairs/higher education literature. It also may not be usable by or useful for audiences beyond the direct fraternity and sorority advising operation.

Here are the most common ways external reviews are framed:

1. Council for the Advancement of Standards in Higher Education (CAS): CAS was designed to guide program review in higher education. It is “a consortium of professional associations in higher education that promotes the use of its professional standards for the development, assessment, and improvement of quality student learning, programs, and services.” 1 CAS provides a common language and roadmap for functional areas to evaluate current programmatic efforts, policies, and organizational structures. It is the gold-standard and provides incredible resources to successfully execute a review.

2. “Big Questions”/”Problem Areas”: While CAS provides a common language widely understood across multiple functional and programmatic areas within higher education, it may not provide a review that meets the needs of campuses engaging an external review team to provide insight toward specific questions. The “big questions” or “problem areas” approach engages an external review team to provide insights and recommendations for targeted areas of concern. While there are less resources available with this approach, it can provide practical recommendations that may have deep relevance to specific challenges within a community.

In my experience most external reviews are framed through elements of both. Utilizing CAS ensures credibility and inter-divisional relevance, while the “big questions” framework ensures the review meets immediate needs of the campus and community.

Building the External Review Team

External reviewers are qualitative researchers. They are charged with entering an unfamiliar space, quickly gaining entry into that space, and intentionally engaging with members of the community in ways that allow themes to emerge. While being a skilled fraternity and sorority professional can certainly inform work as an external reviewer, it does not solely define external review work. Effective external reviewers lean on skill sets in research and assessment, often complemented or enhanced by experiences in the fraternity and sorority advising profession. Care should go into ensuring an external review team has the research, evaluation, and assessment skills necessary to conduct the review. Simply put, this is not a time to assemble your friends.

Additionally, effective review teams exist at the intersection of skill and diversity. External review teams are tasked with gaining access quickly upon entering a campus, and the quality of access can impact what information is available to the reviewer. Campus community members — specifically students and volunteers — want to see a team as representative of their lived experiences as possible. A team that fails to value their experiences through representation can impact the quality of access gained by reviewers. On an external review team, representation matters.

The Self-Study

A self-study is likely the most neglected component of an external review. This is a campus effort to evaluate perceived effectiveness in the proposed review areas prior to evaluation by external reviewers. The self-study is noteworthy for several reasons:

1. It engages the campus in its own review. Perceptions of those who exist within the institutional culture and “do the work” everyday hold equal importance to the “expert opinion”

2. It prepares and primes the review team with data that allows the team to build an effective data collection strategy on campus. By nature of name, external reviewers are “external.” As such, a selfstudy provides insights about what interventions currently exist and perceived effectiveness of those interventions.

While a full self-study is recommended, any efforts to gather and collect artifacts to help the review team build its strategy are helpful to ensure time on-site is maximized. There is a difference, however, between gathering documents and conducting a self-study. Simply compiling policies, procedures, and other items forces reviewers to make sense of the information provided through their own lenses. A self-study provides those artifacts in a way that has already been evaluated for perceived effectiveness. For example, if a specific policy, practice, or intervention strategy may be ineffective and you KNOW it is ineffective, it changes the way it is evaluated within the context of the review. A strong self-study effort is grounded in the same framework as the external review. One of the many benefits of using the CAS framework is the self-study guides allow for easy translation within and between functional areas.

The On-Site Visit

While only part of the external review process, the on-site visit is often perceived as the most critical component of the review. It is where preparation from the campus and external review team is more fully realized, as it places reviewers in front of community members to directly collect data from those experiencing or interacting with the fraternity and sorority community on campus. While this is the case, there often seems to be a “kitchen-sink” approach to developing visit schedules, where JJ’s cousin’s best friend’s babysitter has an opinion and wants to share that with the external review team. This can present a challenge for the campus in crafting a schedule that is comprehensive, gives voice to all relevant constituent groups, and values external reviewers’ time and intellectual energy. The focus is often only on creating a schedule that meets with all the appropriate “constituent groups.” It can be more helpful, however, to focus on creating “theme groups” in cooperation with “constituent groups.” Theme groups may emerge as those who are involved and have a vested interest in student safety and student conduct, intellectual development, volunteer engagement, etc. This can also allow campuses to see if there are important stakeholder communities based on topic area, in addition to constituent groups, that can lead to a more comprehensive schedule.

Managing External Review Outcomes and Expectations

The intellectual labor of the external review team is usually captured in one specific form: the final report. The final report is intended to accurately describe methods used by the review team to collect and understand data, how that helped the team most accurately understand and communicate the issue(s), and provide a recommendation based upon relevant literature and best practice. It is important a final report communicates the high-quality evaluative work of the review team, however it is even more vital the report is usable. A final report is ineffective if it cannot accurately communicate the challenges in a way that is easily understood across constituent groups. Campuses can provide guidance to the external review team in organizing and framing the report for maximum effectiveness. I often ask clients in advance how the report can best be understood by campus community members. This includes how content can be grouped and organized, along with language that is colloquial to the institution.

In external reviews, expectations of what the report should recommend can be placed upon the review team before the work even begins. This most often comes in the area of staffing and resources. Clients may expect an external review to tell them they “need more staff” or “need a bigger budget.” These expectations are often grounded in feelings about the ways fraternity and sorority advising is valued within the context of an institution or within higher education. While these expectations are understandable, recommendations for increasing staff and budget are generally an oversimplified — and sometimes inaccurate —

solution to the problems at hand. More staff and larger budgets may in fact be recommendations from an external review. However, I frequently find recommendations come in the form of repositioning and reallocating time, focus areas, and resources by the university and fraternity and sorority community members. If more (insert resource here) is needed, it will most certainly be recommended. However, it will be done within the context of maximizing current resources and awareness of the diminishing resources with which many student affairs divisions currently operate.

When summarizing the usefulness and impact of external reviews, the Principles of Good Practice for Assessing Student Learning 3 serve as an effective guide and reminder for how to frame reviews:

• Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes: The same should be said for external reviews. What these processes aim to achieve and how they are framed should be clear across multiple communities

• Assessment works best when it is ongoing and not episodic: Establish a cycle of external reviews as part of a broader assessment program. This ensures it is not only seen as a strategy when things get “out of control”

• Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change: An external review can be a great resource that might help facilitate change, but the conditions that exist around and as a result of it are more likely to facilitate change

External reviews can be a valuable tool to aid in the growth and development of a fraternity and sorority community. An external review should not, however, be a casual proposition, nor conducted just because other institutions or peers are doing one. An effective review that will provide a useful report must be constructed with intentionality, direction, and preparation. Designing a review process the right way sets the review team up for success and ensures a high-quality product that will help the institution move forward.

“While these expectations are understandable, recommendations for increasing staff and budget are generally an oversimplified— and sometimes inaccurate— solution to the problems at hand.”

Jason Bergeron became invested in higher education assessment work through his leadership of the now defunct AFA Assessment Committee. In addition to his 14 years of professional experience in fraternity/sorority advising, he has both coordinated external reviews on his home campuses in addition to serving on multiple external review teams. He is currently a full-time director and part-time Ph.D. student at the University of Houston studying student learning and institutional effectiveness. Jason often speaks about assessment work to anyone who will give him the time of day.