Data-Driven Resource Allocation

The e-Advocate Quarterly Magazine Proverbs 18: 13 | Proverbs 18: 17

Data-Driven Resource Allocation

“Helping Individuals, Organizations & Communities Achieve Their Full Potential”

Vol. V, Issue XXII – Q-* Bonus-2 2019

The Advocacy Foundation, Inc. Preparing Individuals, Organizations & Communities to Achieve Their Full Potential

Data-Driven Resource Allocation

1735 Market Street, Suite 3750 Philadelphia, PA 19102

| 100 Edgewood Avenue, Suite 1690 Atlanta, GA 30303

John C Johnson III, Esq. Founder & CEO ______

(855) ADVOC8.0 (855) 238-6280 ยง (215) 486-2120 www.TheAdv ocacyFoundation.org Page 2 of 50

Biblical Authority ______ Proverbs 18:13 (AMP) 13

He who answers a matter before he hears the facts—it is folly and shame to him.

Proverbs 18:17 (AMP) 17

He who states his case first seems right, until his rival comes and cross-examines him.

Page 3 of 50

Page 4 of 50

Table of Contents ______ Biblical Authority I.

Introduction

II.

Data Gathering

III. Community Needs Assessments IV. Crunching the Numbers – Big Data V.

Strategic Resource Allocation

VI. Project Implementation VII. Change Management VIII. Evaluation ______ Attachment A: Data-Driven Decision Making: A Powerful Tool for School Improvement

Attachment B: 10 Things You Always Wanted To Know About DataDriven Decision Making

Page 5 of 50

Page 6 of 50

Introduction Data-Informed DecisionMaking (DIDM)

DDDM refers to the collection and analysis of data to guide decisions that improve success. DIDM is used in education communities (where data is used with the goal of helping students) but is also applicable to (and thus also used in) other fields in which data is used to inform decisions. While data-driven decision-making is a more common term, data-informed decision-making is a preferable term since decisions should not be based solely on quantitative data. Most educators have access to a data system for the purpose of analyzing student data. These data systems present data to educators in an over-the-counter data format (embedding labels, supplemental documentation, and a help system, making key package/display and content decisions) to improve the success of educators’ data-informed decision-making.

In a recently published article, “Data Science and its Relationship to Big Data and Data-Driven Decision Making,” Foster Provost and Tom Fawcett define data-driven decision making as “the practice of basing decisions on the analysis of data rather than purely on intuition.” Equally succinctly, they view data science “as the connective tissue between dataprocessing technologies (including those for b ig data) and data-driven decision making.”

http://en.wikipedia.org/wiki/Datainformed_decision-making

One of the biggest challenges in leveraging data science to help make complex strategic decisions is to mistakenly assume that an unordered, unpredictable, complex context is in fact an ordered, predicable complicated one.

“This assumption, grounded in the Newtonian science that underlies scientific management, encourages simplifications that are useful in ordered circumstances. Circumstances change, however, and as they become more complex, the simplifications can fail.

Good leadership is not a one-size-fits-all proposition.”

Decision making has long been a subject of study and given the explosive growth of Big Data over the past decade, it’s not surprising that data-driven decision making is one of the most promising applications in the emerging discipline of data science.

Neither is good data-driven decision making. With operational decisions, we have to learn to distinguish between those situations when decisions can be embedded in automated processes, and those that require human intervention.

With strategic decisions we have to learn the difference between Page 7 of 50

complicated but predictable contexts, and complex and intrinsically unpredictable ones. This is all part of

what makes data science such an important and exciting discipline.

-

Irving Wladawsky-Berger

Page 8 of 50

Page 9 of 50

Data Gathering Looking deeper at data-driven decision making, it’s important to understand not only the promise but also the limits.

When can we embed decisions into well understood, automated processes?

When does automation run into limits, and should we view data-driven decision making as a tool to help people make smarter, more effective decisions? And what are the prospects for its future, as technology, Big Data and data science continue to advance?

In an online conversation, “Reinventing Society in the Wake of Big Data,” MIT Media Lab Professor Sandy Pentland talks about the promise of becoming a datadriven society: *** “I believe that the power of Big Data is that it’s information about people’s behavior –it’s about customers, e mployees, and prospects for your new business . . .This Big Data comes fro m location data from your cell phone and transaction data about the things you buy with your credit card. It’s the little data breadcrumbs that you leave behind you as you move around in the world. . . Big Data is increasingly about real behavior, and by analyzing this sort of data, scientists can tell an enormous amount about you.

They can tell whether you are the sort of person who will pay back loans. They can tell you if you’re likely to get diabetes.”

Privacy Issues These decision making applications require access to vast amounts of

personal information, which leads to very serious concerns about privacy, data ownership and data control. It’s important that individuals are aware of and have final say about the use of the data collected about them. For example, we will likely be quite happy with the use of personal data in identity management applications, but might want to be more selective in how our data is used to send us marketing offers. Considerable research is needed as we learn how to strike the right balance between such data-driven decision making and privacy.

Data Mining Data mining and similar analytical methods are most applicable in the ordered contexts, whether simple – having a clear cause-and-effect relationship that the analysis can uncover – or complicated – which unlike simple contexts may have multiple options and answers which can be analyzed, evaluated and compared prior to making a decision.

A complex context is quite different. The right decision cannot be ferreted out from the available information. The article points out that comparing a complicated and a complex context is like comparing a Ferrari with the Brazilian rainforest. “Ferraris are complicated machines, but an expert mechanic can take one apart and reassemble it without changing a thing.

The car is static and the whole is the sum of its parts. The rainforest, on the other hand, is in constant flux – a species becomes extinct, weather patterns change, an agricultural project Page 10 of 50

reroutes a water source – and the whole is far more than the sum of its parts.”

“Most situations and decisions in organizations are complex because some major change–a bad quarter, a shift in management, a merger or acquisition–introduces unpredictability and flux. In this domain we can understand why things happen only in

retrospect. Instructive patterns, however, can emerge if the leader conducts experiments that are safe to fail. That is why, instead of attempting to impose a course of action, leaders must patiently allow the path forward to reveal itself. They need to probe first, then sense, then respond”.

Page 11 of 50

Community Needs Assessments A Community Needs Assessment is a combination of information gathering, community engagement and focused action with the goal of community improvement. A community needs assessment identifies the strengths and weaknesses (needs) within a community. A community needs assessment is also unique and specific to the needs within a community and is usually an extension of a community's strategic planning process.

The community needs assessment places great emphasis on the abilities of the people in the community, and on the agencies and organizations within that community that provides services to the children and families. Community leaders, local government, advocacy groups or a combination of these then address these identified needs through policy change or development.

A community needs assessment can be broadly categorized into three types based on their respective starting points: First, needs assessments which aim to discover weaknesses within the community and create a solution (Community Needs Assessment I).

Second, needs assessments which are structured around and seek to address an already known problem or potential problem facing the community (Community Needs Assessment II).

Third, needs assessments of an organization which serves the community (domestic violence centers, community health clinics etc.)

(Community Needs Assessment III).

Community needs assessments are generally executed in four steps: planning and organizing, data collection, coding and summarizing the needs assessment results, and sharing the results with the community to facilitate action planning. During the planning and organizing phase stakeholders are identified, local organizations and/or local government begin to collaborate.

Depending on the type of needs assessment being conducted one can tailor their approach.

Types of Community Needs Assessment – Strategies for Planning and Organizing Community Needs Assessment I – This type of needs assessment seeks to evaluate the strengths and weaknesses within a community and create or improve services based on the identified weaknesses. Organizing this type of needs assessment is primarily structured around how to best obtain information, opinions, and input from the community and then what to do with that information. This process may be broken into targeted questions which can direct the project overall. The following are sample questions taken from “A Community Needs Assessment Guide” from The Center for Urban Research & Learning: •

Define goals assessment.

for

the

needs

•

What is the specific purpose of the needs assessment?

Page 12 of 50

•

How will the data from the community be used; to set a new agenda, support a new program or support new changes in service delivery or policies?

•

What is the timeline for the needs assessment?

•

If applicable, identify the target population. How will a sample from the population be chosen?

Are there any special considerations which need to be considered in the most effective way to approach/obtain information and cooperation from said population?

Community Needs Assessment II – This type of needs assessment is constructed around a known problem or potential problem facing the community for example, disaster preparedness, how to address an increase in violent crime etc. This type of community needs assessment centers less around the direct involvement of the community but rather the governing entities, stakeholders, businesses, advocacy groups and organizations which will be potentially affected or can contribute to the community need.

Potential organization questions could include: •

Identifying relevant stakeholders.

This includes stakeholders affected by the problem or stakeholders of the program/or solution being addressed. The program staff, the funders, and the consumers of the program.

•

Review already existing material regarding the community problem or potential problem.

•

Sharing expectations, goals, and approach regarding the needs assessment with the other partners.

•

Discuss and identify potential users of the agenda/solution likely to be generated by the needs assessment process.

Community Needs Assessment III This final type of needs assessment is based within an organization which either serves the community at large, is currently addressing a need within the community, or is dedicated to an underserved population within the community.

This type of needs assessment centers around improving the efficiency or effectiveness of such organizations.

Potential organization questions could include: •

• •

Learn more about the community and its residents.

Learn about the organizational culture and its philosophy by interviewing staff, including the executive director.

Review existing materials regarding the community need and the organization.

Page 13 of 50

•

•

•

•

•

Tour the community and learn more about the target population or problem the organization serves.

Conduct a literature review to see what the recent research has to offer, review relevant archival information and what previous needs assessments by the organization have found.

Where is the program in terms of the implementation and development of service delivery?

What current resources do the organization and its programs offer?

Identify and learn about the program that would most benefit from a needs assessment.

Implementing a Community Needs Assessment – The exact methodology to implementing a community needs assessment is partially determined by the type of assessment that is being performed (discussed above). However, general guidelines can be proposed.

population, or demographic which will structure the community needs assessment. This information guides the selection process for a focus group. The principle of the focus group is to select members who are diverse yet share a degree of commonality.

This may sound paradoxical yet it isn’t necessarily.

Generally speaking the commonality between focus group members is a vested interest and stake in their community. Thus, focus group members might include: “local politicians, business owners, block club leaders and community activists.

Another focus group would consist of adult resident of the community; and a third consisting of youth residents of the community”.

Focus groups solicit input from community members on broad, openended questions such as: • • • •

1. Use of focus groups 2. Creating a needs assessment survey 3. Collecting and analyzing data 4. Community public forums 5. Producing a final report and planning action committees Selecting members of a focus group first requires choosing a target community,

What do you like about your community?

What concerns you within your community?

How would you improve your community?

What changes do you foresee/fear/want to see in your community within the next 10 years?

Questions such as these can help target potential strengths, weaknesses, opportunities and needs for change or growth.

Page 14 of 50

With the targeted objectives discovered in the focus group, the community needs assessment survey can be created and dispersed.

Leaders of the community needs assessment can then summarize the data through computer analysis programs such as Access or SPSS. The results are then brought to the community through a public forum.

Public forums are the place where the information collected through the survey, the identified strengths, weaknesses, and concerns of the community are presented for open public discussion.

Finally, the results of the focus groups, survey, and public forum present a direction which the final report can detail. Action groups are formed and solutions and guidelines are enacted to ensure the changes desire are realized.

Local Government’s Community Needs Assessment - Local city governments have a department dedicated to the sole purpose of funding nonprofit organizations that see about the current needs of the children and families who reside in that city. The purpose of these departments is to ensure that nonprofit organizations that receive funding from the Children, Families Department will provide families with children with the necessary services that are essential to children growing up healthy, have access to a quality education, and thrive in safe homes and neighborhoods. Example: The Department of Children, Youth and Their Families in San Francisco, California. This specific city department conducts a Needs Assessment every

three years to develop a strategic plan to guide the department during their funding cycle when they send out Request for Proposal (RFP) for organizations to apply for grants which will enable these community organizations to continue to provide services to the children and families in their community.

Conducting A Community-Level Needs Assessment According to Sharma, Lanum and Saurez-Balcazar (2000) “the goals of a 'needs assessment' is to identify the assets of a community and determine potential concerns that it faces� (p. 1). A needs assessment therefore becomes crucial in the initial stages of an intervention. A needs analysis is focused on identifying the possible barriers to successful program intervention in a community and possibly finding solutions to these challenges.

Service providers in Monitoring and Evaluation (M&E) work are also concerned with assessment and provision of services to different stakeholders. Such services may include an assessment closely related to a needs assessment that focuses on whether current services are effective or not, and if not, identifying the gaps in implementation; or an assessment of whether potential services are likely to be effective once they have been implemented (Rossi, Lipsey & Freeman, 2004). These assessments highlight the close relationship between needs assessment, monitoring, and evaluation; while each applies similar tools, each also has independent objectives and requires unique skills.

Page 15 of 50

In community development work practitioners are concerned with identifying barriers that stand in the ways of positive community progress. In many cases, an organization or community is faced by challenges with regards to some social issue, provision or access to services and it is the job of the practitioner, in consultation with stakeholders, to decide about how best to go about finding helpful interventions and implementing solutions to this.

of which are aimed at gathering data that will answer what the practitioner needs to know and inform the decisions that he or she makes. According to the National Consumer Supporter Technical Assistance Center (www.ncstac.org.) the following are crucial components of a community level needs assessment.

A community level needs assessment is beneficial and crucial to any planned intervention on behalf of communities facing difficulties with regard to some community issue. A community level needs assessment will assist the practitioner to determine the nature and scope of a problem at which an intervention might be aimed, with the aim of finding out what possible interventions might be successful in alleviating the problem (Rossi, Lipsey & Freeman, 2004). A community needs assessment will also uncover which members of the community are most likely to benefit from a planned intervention and who might not be.

Community level needs assessment will also give direction to planners in terms of where resources need to be allocated for the intervention so that they are not wasted.

Community level needs assessments should include the community at all stages of planning, and should consider all people that might be affected by the planned intervention, including children, the elderly and the mentally ill.

Community demographics assist the practitioner to get a feel of the field that they are working in. Demographics include things like age ranges, the number of people living in a certain area within the community, the number or percentage of people within a certain socio economic status and gender characteristics <"www.ncstac.org">/<www.ncstac.org/c ontent/materials/CommunityNeedsAsse ssment.pdf >. Demographic information about certain population groups can be found online at such official websites as www.statssa.gov in South Africa.

Tools for an Effective CLNA Project There are a number of components in a community level needs assessment, all

Assessment Community Demographics :

Consumer Leadership Consumer leadership assessment is an assessment of the frequency with which community members use or are likely to use an existing or planned service.1 This assessment is meant to give an indication of the need for the existing or proposed intervention or service.

Consumer leadership assessment is meant to give an indication of the different types of leadership activities and roles that are related to transformation in relation to some health or social issue that is being addressed.

This may give an indication as to the 1

www.ncstac.org Page 16 of 50

degree of the need for an intervention or not.

Service Gaps An assessment of service gaps is meant to give an indication of the types of services that are needed the most at the particular point of time in which the assessment is being conducted (www.ncstac.org). A scale measuring the availability, accessibility, provider choice and cultural responsiveness of services, rated on a scale from 0-no availability/non-existent, to 3outstanding and responsive is provided by the National Consumer Supporter Technical Assistance Center. The scale also assesses the availability of other services in the community such as support groups, education and employment services that may be of interest to the practitioner.

Methodology and data collection: how to get information for the assessment The following are the actual tools that can be involved in the process of gathering data to be used in the community needs assessment.

Community/ Social Survey Surveys can be used especially in relation to the gathering of community demographics where a large number of people may be involved, and also in which multiple variables such socioeconomic status, education levels and employment are being measured in

relation to the planned intervention.

Large scale surveys involving many people can reveal useful information, while smaller surveys may be less generalizable and used only in the context within which they are conducted.

Survey design will vary depending on context, such as internet and phone surveys for well resourced communities or face to face surveys for less resourced communities.

Community Mapping Often, a practitioner may be wanting to implement a service that they think will be of benefit to the community. The problem facing the practitioner will be where and how to place the service at a particular point in the community, and whether that service is likely to be used.

Community mapping is where the practitioner gets people in the community to draw a map of the community of the places that they visit the most and how often they go there.

This will give an indication of where to locate a service so that it is conveniently placed and accessible to community participants whom it is intended to service. The problem may arise where there are differences between the places that people visit.

Seasonal Calendar A seasonal calendar allows the practitioner to look at seasonal trends that may affect the use or implementation of a service in a

Page 17 of 50

community.2 Seasonal trends may reveal decreases in the supply of labor, periods of hunger that may affect for example school childrenâ&#x20AC;&#x2122;s performance at school and so on. Seasonal calendars may reveal important reasons for the gaps between service utilization and intervention outcomes. This will allow the practitioner to plan for other things that may not have been considered as part of the intervention but which will greatly improve the quality of the intervention and make life better for the community members. To use the seasonal calendar as a data collection tool, the practitioner gets community members to write a list of the things that they have to do throughout the year.

These things are related to work, cultural activities, certain times of the year in which participants are unavailable at all and so on, and to plot how they share them with other members of the community.

Focus Group Sessions Focus groups are sessions in which community members can answer questions about different aspects of the intervention in the form of planned discussions. This is a good opportunity to actually find out about the needs and concerns of the community. It is also a good opportunity for addressing service gaps and what needs to be done about them.

Example of a Community Needs Assessment Bridging the Gaps : Toward an Efficient Social Service Delivery in Bayview Hunters Point

This is a good example as needs are identified in several different ways, such as research, survey analysis, and current gaps in service provision. All of this information can be used as analysis towards future policy implementation or as a focal point for discussion.

The author examined significant statistics that showed a need within the community of Bayview Hunters Point in order to â&#x20AC;&#x153;identify gaps in service delivery system to create a road map for improving neighborhood conditions by rationalizing the allocation of city dollars to social service programsâ&#x20AC;? (Burke, 7).

For example, in 2003, 174 children were removed from family homes in the Bayview; this is more than 18% of all children removed from their family in San Francisco. Such numbers could signify a need within the foster care system or family resources. The author also looked at the broad-based survey, Project Connect, which gathered data from 10,330 households specifically about their needs for services and current service utilization practices in the summer of 2004. The analysis from 1,551 Bayview households showed that their priorities, in order, are 1) childcare services, 2) health services, 3) tutoring/educational services, 4) immigrant services, 5) foodbank/meal services. According to the Child Care Planning and Advisory Council, in 2002 the unmet need for subsidized care in Bayview Hunters Point included 2,379 slots for children 0-13. Such needs were gathered from identifying how many slots exist, and whether families can pay for those slots. http://en.wi kipedia.org/wi ki/Needs_assess ment

2

http://www.rotary.org/ridocuments/en_pdf/605c _en.pdf Page 18 of 50

Crunching the Numbers - Big Data Big data is an all-encompassing term for any collection of data sets so large and complex that it becomes difficult to process using traditional data processing applications. It usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time. Big data "size" is a constantly moving target, as of 2012 ranging from a few dozen terabytes to many Petabytes of data.

visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers". What is considered "big data" varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain.

Big Data is a moving target; what is considered to be "Big" today will not be so years ahead.

"For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."

The Three â&#x20AC;&#x153;Vâ&#x20AC;&#x2122;sâ&#x20AC;? Model

The challenges include analysis, capture, curation, search, sharing, storage, transfer, visualization, and privacy violations. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, prevent diseases, combat crime and so on."

Big data is difficult to work with using most relational database management systems and desktop statistics and

In a 2001 research report and related lectures, META Group (now Gartner) analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional, i.e. increasing Volume (amount of data), Velocity (speed of data in and out), and Variety (range of data types and sources).

Gartner, and now much of the industry, continue to use this "3Vs" Model for describing big data. In 2012, Gartner updated its definition as follows: "Big data is high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization." Additionally, a new V "Veracity" is added by some organizations to describe it.

Page 19 of 50

The growing maturity of the concept fosters a more sound difference between big data and Business Intelligence, regarding data and their use: •

Business Intelligence uses descriptive statistics with data with high information density to measure things, detect trends etc.;

•

Big data uses inductive statistics and concepts from nonlinear system identification to infer laws (regressions, nonlinear relationships, and causal effects) from large sets of data with low information density to re veal relationships, dependencies and perform predictions of outcomes and behaviors.

Big data can also be defined as "Big data is a large volume unstructured data which cannot be handled by standard database management systems like DBMS, RDBMS or ORDBMS".

Big Data in Government In 2012, the Obama administration announced the Big Data Research and Development Initiative, to explore how big data could be used to address important problems faced by the government. The initiative is composed of 84 different big data programs spread across six departments.

•

Big data analysis played a large role in Barack Obama's successful 2012 re-election campaign.

•

The United States Federal Government owns six of the ten most powerful supercomputers in the world.

•

The Utah Data Center is a data center currently being constructed by the United States National Security Agency. When finished, the facility will be able to handle a large amount of information collected by the NSA over the Internet.

The exact amount of storage space is unknown, but more recent sources claim it will be on the order of a few Exabytes.

International development Research on the effective usage of information and communication technologies for development (also known as ICT4D) suggests that big data technology can make important contributions but also present unique challenges to International development.

Ad vancements in big data analysis offer cost-effective opportunities to improve decision-making in critical development areas such as health care, employment, economic productivity, crime, security, and natural disaster and resource management.

However, longstanding challenges for developing regions such as inadequate technological infrastructure and economic and human resource scarcity exacerbate existing concerns with big data such as privacy, imperfect Page 20 of 50

methodology, issues.

and

interoperability

Characteristics of Big Data Big data can be described by the following characteristics: Volume – The quantity of data that is generated is very important in this context. It is the size of the data which determines the value and potential of the data under consideration and whether it can actually be considered as Big Data or not. The name ‘Big Data’ itself contains a term which is related to size and hence the characteristic.

Variety - The next aspect of Big Data is its variety. This means that the category to which Big Data belongs to is also a very essential fact that needs to be known by the data analysts. This helps the people, who are closely analyzing the data and are associated with it, to effectively use the data to their advantage and thus upholding the importance of the Big Data.

Critique of Paradigm

The

Big

Data

"A crucial problem is that we do not know much about the underlying empirical microprocesses that lead to the emergence of the[se] typical network characteristics of Big Data".

Chief critics of The Big Data Paradigm point out that often very strong assumptions are made about mathematical properties that may not at all reflect what is really going on at the level of micro-processes.

Velocity - The term ‘velocity’ in the context refers to the speed of generation of data or how fast the data is generated and processed to meet the demands and the challenges which lie ahead in the path of growth and development.

Variability - This is a factor which can be a problem for those who analyze the data. This refers to the inconsistency which can be shown by the data at times, thus hampering the process of being able to handle and manage the data effectively.

Complexity - Data management can become a very complex process, especially when large volumes of data come from multiple sources. These data need to be linked, connected and correlated in order to be able to grasp the information that is supposed to be conveyed by these data. This situation, is therefore, termed as the ‘complexity’ of Big Data.

Mark Graham3 has leveled broad critiques at Chris Anderson's assertion that big data will spell the end of theory: focusing in particular on the notion that big data will always need to be contextualized in their social, economic and political contexts. Even as companies invest eight- and nine-figure sums to derive insight from information streaming in from suppliers and customers, less than 40% of employees have sufficiently mature processes and skills to do so. To overcome this insight deficit, "big data", no matter how comprehensive or well analyzed, needs to be complemented by "Big 3

Graham M. ( 9 March 2012). "Big data and the end of theory?". The Guardian (London).

Page 21 of 50

Judgment", according to an article in the Harvard Business Review.

Big data is a buzzword and a "vague term", but at the same time an "obsession" with entrepreneurs, consultants, scientists and the media.

Big data showcases such as Google Flu Trends failed to deliver good predictions in recent years, overstating the flu outbreaks by a factor of two. Similarly, Academy awards and election predictions solely based on Twitter were more often off than on target. Big data often poses the same challenges as small data; and adding more data does not solve problems of bias, but may emphasize other problems. In particular data sources such as Twitter are not representative of the overall population, and results drawn from such sources may then lead to wrong conclusions.

Google Translate - which is based on big data statistical analysis of text - does a remarkably good job at translating web pages, but for specialized domains the results may be badly off. On the other hand, big data may also introduce new problems, such as the multiple comparisons problem: simultaneously testing a large set of hypotheses is likely to produce many false results that mistakenly appear to be significant.

Ioannidis argued that "most published research findings are false" due to essentially the same effect: when many scientific teams and researchers each perform many experiments (i.e. process a big amount of scientific data; although not with big data technology), the likelihood of a "significant" result being actually false grows fast - even more so, when only positive results are published.

Page 22 of 50

Strategic Resource Allocation To explore these questions, letâ&#x20AC;&#x2122;s take a closer look at different kinds of decisions. We can think of decision making as lying across a broad spectrum. At one end of the spectrum are Operational Decisions, which are generally highly structured, routine, short-term oriented and increasingly embodied in sophisticated software applications.

processes, and tactical decisions dealing with the necessary adjustments required to implement longer term strategies.

Given their structured nature, IT and data analysis have long been applied to automate routine, day-to-day operational decisions, such as logistics and inventory management, personalized marketing offers and recommendations, and fraud detection in financial transactions.

For example, on a recent trip to a Latin America city a gentleman had not been to before, when checking into the hotel, his credit card was not accepted. A few moments later, he received an automated call in his mobile phone as well as an e-mail from the credit card company asking him to verify that he was indeed trying to use my credit card in that particular city.

At the other end of the spectrum are Strategic Decisions. These are usually taken by high levels of management as they set the long-term directions and policies of a business, government or other organizations.

They tend to be complex, and unstructured because of the uncertainty and risks that generally accompany longer term decisions.

In between are many kinds of decisions, including non-routine ones in response to new or unforeseen circumstances beyond the scope of operational

Similarly, when he logged into his e-mail account, he was first asked to verify his identity to make sure that he was the person logging in from a new location.

These are concrete examples of operational, data-driven decision making that have been built into automated authentication and fraud detection processes.

The more data we gather, and the more sophisticated the analysis, the more such decisions can be made with little or no human intervention. Over time, Big Data and advanced data science applications will enable us to take Page 23 of 50

operational decision making to a whole new level in a wide variety of disciplines. ______ And beyond automated operational decisions, there are many situations where human intervention is required for a variety of reasons.

Strategic decisions are aimed at setting the long term directions and policies of an organization. Making sound strategic decisions is one of the most important qualities of a great leader, and is thus a major component of leadership courses and seminars. But, the use of Big Data and data science to help with strategic decisions is in its early stages and requires quite a bit more research to understand how to use them under different contexts.

Executives need a framework that allows them to see things from new viewpoints and assimilate complex concepts. A framework can be defined, the authors suggest, with examples from the organization’s history.

“This enhances communication and helps

executives rapidly understand the context in which they are operating.”

Dave Snowden and Mary Boone: “A Leader ’s Framework for Decision Making,” (a 2007 Harvard Business Review paper).

A very good framework, they write, is designed to help leaders determine the overall context for making their strategic decisions, in particular whether it is ordered and complicated, or unordered and complex.

“Each domain requires different actions”.

Simple and complicated contexts assume an ordered universe, where cause-and-effect relationships are perceptible, and right answers can be determined based on the fact. Complex and chaotic contexts are unordered – there is no immediate relationship between cause and effect, and the way forward is determined based on emerging patterns. The ordered world is the world of facts-based management; the unordered world represents patternbased management.

Page 24 of 50

Page 25 of 50

Project Implementation Project Management is the process and activity of planning, organizing, motivating, and controlling resources, procedures and protocols to achieve specific goals in scientific or daily problems. A project is a temporary endeavor designed to produce a unique product, service or result with a defined beginning and end (usually timeconstrained, and often constrained by funding or deliverables), undertaken to meet unique goals and objectives, typically to bring about beneficial change or added value, [in this case to an entire community of people]. The temporary nature of projects stands in contrast with business as usual (or operations), which are repetitive, permanent, or semi-permanent functional activities to produce products or services [in this case services]. In practice, the management of these two systems is often quite different, and as such requires the development of distinct technical skills and management strategies.

The primary challenge of project management is to achieve all of the project goals and objectives while honoring [certain] preconceived constraints. The primary constraints are [service deliverables, timetables, quality, budget and compliance]. The secondary — and more ambitious — challenge is to optimize the allocation of necessary [resources] and integrate them to meet pre-defined [goals and] objectives.

There are a number of approaches to managing project activities including lean, iterative, incremental, and phased approaches.

Regardless of the methodology employed, careful consideration must be given to the overall project objectives, timeline, and cost, as well as the roles and responsibilities of all participants and stakeholders.

The Traditional Approach A traditional phased approach identifies a sequence of steps to be completed. In the "traditional approach", five developmental components of a project can be distinguished (four stages plus control): 1.

2.

3.

4.

5.

Initiation [Mobilization] Execution Monitoring and Quality Control Evaluation

Not all projects will have every stage, as projects can be terminated before they reach completion. Some projects do not follow a structured planning and/or monitoring process. And some projects will go through steps 2, 3 and 4 multiple times.

Many industries use variations of these project stages.

[A]

“Cone

of

Uncertainty”

explains some of this as the planning made on the initial phase of the project suffers from a high degree of uncertainty.

This becomes especially true as software development is often the realization of a new or novel product. In projects where requirements have not Page 26 of 50

been finalized and can change, requirements management is used to develop an accurate and complete definition of the behavior of software that can serve as the basis for software development. While the terms may differ from industry to industry, the actual stages typically follow common steps to problem solving—"defining the problem, weighing options, choosing a path, implementation and evaluation."

• • • •

•

analyzing needs in measurable goals reviewing of the current operations financial analysis of the costs and benefits including a budget stakeholder analysis, including users, and support personnel for the project project charter including costs, tasks, deliverables, and schedule

The Process

Project Execution

Traditionally, project management includes a number of elements: four to five process groups, and a control system. Regardless of the methodology or terminology used, the same basic project management processes will be used. Major process groups generally include:

Executing Process Group Processes

• • • • •

Initiation [Mobilization] Execution Monitoring and Quality Control Evaluation

The initiating processes determine the nature and scope of the project. If this stage is not performed well, it is unlikely that the project will be successful in meeting [organizational] needs. The key project controls needed here are an understanding of the environment and making sure that all necessary controls are incorporated into the [project or program]. Any deficiencies should be reported and [change] recommendations should be made to fix them.

The initiating stage should include a plan that encompasses the following areas:

Executing consists of the processes used to complete the work defined in the project plan to accomplish the project's requirements.

Execution process involves coordinating people and resources, as well as integrating and performing the activities of the project in accordance with the Project Management Plan . The deliverables are produced as [outcomes] from the processes performed as defined in the Project Management Plan and other frameworks that might be applicable to the type of project at hand.

Execution process group include:

Page 27 of 50

• • •

Direct and manage project execution Quality assurance of deliverables Acquire, develop and manage Project team

• • • •

Distribute information Manage stakeholder expectations Conduct procurement Test the deliverables against the initial design

Quality Control Monitoring and Controlling Monitoring and controlling consists of those processes performed to observe project execution so that potential problems can be identified in a timely manner and corrective action can be taken, when necessary, to control the execution of the project.

The key benefit is that project performance is observed and measured regularly to identify variances from the project management plan.

Monitoring and controlling includes: • • • •

Measuring the ongoing project activities ('where we are'); Monitoring the project variables (cost, effort, scope, etc.) against the project management plan and the project performance baseline (where we should b e); Identify corrective actions to address issues and risks properly (How can we get on track again); Influencing the factors that could circumvent integrated change control so only approved changes are implemented.

In multi-phase projects, the monitoring and control process also provides feedback between project phases, in order to implement corrective or preventive actions to bring the project into compliance with the project management plan.

Project maintenance is an ongoing process, and it includes: • • •

Continuing support of end-users Correction of errors [Continuous] updates of tracking software

Monitoring and Controlling Cycle In this stage, auditors should pay attention to how effectively and quickly user problems are resolved.

Page 28 of 50

Over the course of any [program] or project, the work scope may change. Change is a normal and expected part of the process. Changes can be the result of necessary programmatic modifications, differing site conditions, collateral resource availability, [sub-]contractorrequested [revisions] and impacts from third parties, to name a few. Beyond executing [a] change in the field, change[s] normally need to be documented to show what was actually [accomplished]. This is referred to as Change Management.

Hence, the [funders will] requires a final record to show [any] changes or, more specifically, any change that modifies the [Scope of Services]. The record is made on the contract documents – usually, but not necessarily limited to, the [Program Narratives].

When changes are introduced to the [program], the viability of the project has to be reassessed. It is important not to lose sight of the initial goals and [objectives] of the projects. When the changes accumulate, the forecasted result may not justify the original proposed investment in the project.

Project Controlling and Project Control Systems Project Controlling should be established as an independent function in [Program] Management. It implements verification and controlling function during the processing of a project in order to reinforce the defined performance and formal goals. The tasks of project controlling are also: • • • • • •

the creation of infrastructure for the supply of the right information and its update the establishment of a way to communicate disparities of project parameters the development of project information technology based on an intranet or the determination of a project key performance index system (KPI) divergence analyses and generation of proposals for potential project regulations the establishment of methods to accomplish an appropriate project structure, project workflow organization, project control and governance creation of transparency among the project parameters

Fulfillment and implementation of these tasks can be achieved by applying specific methods and instruments of project controlling. The following methods of project controlling can be applied: • • • • • • •

Investment Analysis Cost–Benefit Analyses Value Benefit Analysis Expert Surveys Simulation Calculations Risk-Profile Analyses Surcharge Calculations Page 29 of 50

• • •

Milestone Trend Analysis Cost Trend Analysis Target/Actual-Comparison

Project control is [the] element of a project that keeps it on-track, on-time and within budget. Project control begins early in the project with planning and ends late in the project with post-implementation review, having a thorough involvement of each step in the process. Projects may be audited or reviewed while the project is in progress.

Formal audits are generally risk or compliance-based and management will direct the objectives of the audit. An examination may include a comparison of approved Project Management Processes with how the project is actually being managed. Each project should be assessed for the appropriate level of control needed: too much control is too time consuming, too little control is very risky. If project control is not implemented correctly, the cost to the [budget] should be clarified in terms of Errors And Revisions.

Control systems are needed for cost, risk, quality, communication, time, change, procurement, and human resources. In addition, auditors should consider how important the projects are to the financial statements, how reliant the stakeholders are on controls, and how many controls exist. Auditors should review the development process and procedures for how they are implemented.

The process of development and the quality of the final product may also be assessed if needed or requested. A business may want the auditing firm to be involved throughout the process to catch problems earlier on so that they can be [re vised] more easily. An auditor can serve as a controls consultant as part of the development team or as an Independent Auditor as part of an audit.

[Organizations should] [utilize] formal systems development processes. These help assure that systems are successfully [developed]. A formal process is more effective in creating strong controls, and auditors should review this process to confirm that it is well designed and is followed in practice. A good Formal Systems Development Plan outlines: • • • • •

A Strategy to Align Development with the Organization’s Broader Objectives Standards for New Systems [Program] Management Policies for Timing and Budgeting Procedures Describing the Process Evaluation of Quality of Change http://en.wikipedia.org/wiki/Project_management

Page 30 of 50

Page 31 of 50

Change Management Hardly would any [program or] project be completed without some changes being asked of [management]. The changes can stem from changes in the environment in which the finished product is envisaged to be used, business changes, regulation changes, errors in the original definition of requirements, limitations in technology, changes in the security environment and so on. The activities of Requirements Change Management include receiving the change requests from the stakeholders, recording the received change requests, analyzing and determining the desirability and process of implementation, implementation of the change request, quality assurance for the implementation and [evaluating] the change request. Then the data of change requests [can] be compiled, analyzed and appropriate metrics derived and dovetailed into the organizational knowledge repository.

Requirements Management is the process of documenting, analyzing, tracing, prioritizing and agreeing on requirements and then controlling change and communicating to [Program team members and] relevant stakeholders. It is a continuous process throughout a project. A requirement is a capability to which a project outcome (product or service) should conform.

The purpose of requirements management is to ensure that an organization documents, verifies, and meets the needs and expectations of its internal [and] external stakeholders.

Requirements management begins with the analysis and elicitation of the objectives and constraints of the organization.

Requirements management further includes supporting planning for requirements, integrating requirements and the organization for working with them (attributes for requirements), as well as relationships with other information delivering against requirements, and changes for these.

The Traceability thus established is used in managing requirements to report back fulfillment of company and stakeholder interests in terms of compliance, completeness, coverage, and consistency. Traceabilities also support Change Management as

Part of Requirements Management in understanding the impacts of changes through requirements or other related elements (e.g., functional impacts through relations to functional architecture), and facilitating introducing these changes.

Requirements management involves communication between the project Page 32 of 50

team members and stakeholders, and adjustment to requirements changes throughout the course of the [program]. To prevent one class of requirements from overriding another,

constant communication among members of the development team is critical. http://en.wi kipedia.org/wi ki/Requirements _management

Page 33 of 50

Page 34 of 50

Evaluation Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. So evaluation can be Formative , that is taking place during the development of a concept or proposal, project or organization, with

the intention of improving the value or effectiveness of the proposal, project, or organization.

It can also be Assumptive , drawing lessons from a completed action or project or an organization at a later point in time or circumstance.

Evaluation is inherently a theoretically informed approach (whether explicitly or not), and consequently any particular definition of evaluation would have be tailored to its context – the theory, needs, purpose, and methodology of the evaluation process itself.

Having said this, evaluation has been defined as: •

A systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement, or outcomes of a program. It is a resourceintensive process, frequently requiring resources, such as, evaluation expertise, labor, time, and a sizable budget;

•

"The critical assessment, in as objective a manner as possible, of the degree to which a service or its component parts fulfills stated goals" (St Leger and Wordsworth-Bell). The focus of this definition is on attaining objective knowledge, and scientifically or quantitatively measuring predetermined and external concepts;

•

"A study designed to assist some audience to assess an object's merit and worth" (Shuffleboard).

In this definition the focus is on facts as well as value laden judgments of the programs outcomes and worth.

Purpose The main purpose of a program evaluation can be to "determine the quality of a program by formulating a judgment" -

Marthe Hurteau, Sylvain Houle, Stéphanie Mongiat (2009).

Page 35 of 50

An alternative view is that "projects, evaluators, and other stakeholders (including funders) will all have potentially different ideas about how best to evaluate a project since each may have a different definition of 'merit'.

The core of the problem is thus about defining what is of value." From this perspective, evaluation "is a contested term", as "evaluators" use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research.

There are two functions considering to the evaluation purpose Formative Evaluations provide the information on the improving a product or a process Summative Evaluations provide information of short-term effectiveness or long-term impact to deciding the adoption of a product or process.

Not all evaluations serve the same purpose some evaluations serve a monitoring function rather than focusing solely on measurable program outcomes or evaluation findings. [A] full list of types of evaluations would be difficult to compile. This is because evaluation is not part of a unified theoretical framework, drawing on a number of disciplines, which include management and organizational theory, policy analysis, education, sociology, social anthropology, and social change.

Classification of Approaches Two classifications of evaluation approaches by House and Stufflebeam and Webster can be combined into a manageable number of approaches in terms of their unique and important underlying principles.

House considers all major evaluation approaches to be based on a common ideology entitled liberal democracy.

Important principles of this ideology include freedom of choice, the uniqueness of the individual and empirical inquiry grounded in objectivity.

He also contends that they are all based on subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is Utilitarian , in which "the good" is determined by what maximizes a single, explicit interpretation of happiness for society as a whole. Another form of Subjectivist Ethics is intuitionist/ pluralist, in which no single interpretation of "the good" is assumed and such interpretations need not be explicitly stated nor justified.

These ethical positions have corresponding epistemologiesâ&#x20AC;&#x201D; philosophies for obtaining knowledge.

The objectivist epistemology is associated with the utilitarian ethic; in general, it is used to acquire knowledge that can be externally verified (intersubjective agreement) through publicly exposed methods and data.

The subjectivist epistemology is associated with the intuitionist/ pluralist ethic and is used to acquire new knowledge based on existing personal knowledge, as well as experiences that are (explicit) or are not (tacit) available Page 36 of 50

for public inspection. House then divides each epistemological approach into two main political perspectives. Firstly, approaches can take an elite perspective, focusing on the interests of managers and professionals; or they also can take a mass perspective, focusing on consumers and participatory approaches.

Stufflebeam and Webster place approaches into one of three groups, according to their orientation toward the role of values and ethical consideration.

The political orientation promotes a positive or negative view of an object regardless of what its value actually is and might be—they call this PseudoEvaluation . The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object—they call this Quasi-Evaluation . The values orientation includes approaches primarily intended to determine the value of an object—they call this True Evaluation .

When the above concepts are considered simultaneously, Fifteen Evaluation Approaches can be identified in terms of epistemology, major perspective (from House), and orientation. Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six Quasi-Evaluation Approaches use an objectivist epistemology.

Five of them—

experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective.

Accountability takes a mass perspective.

Seven True Evaluation Approaches are included. Two approaches, decisionoriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective.

Two approaches—accreditation/ certification and connoisseur studies— are based on a subjectivist epistemology from an elite perspective.

Finally, Adversary And ClientCentered Studies are based on a subjectivist epistemology from a mass perspective.

Summary of approaches The following table is used to summarize each approach in terms of four attributes—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together. http://en.wikipedia.org/wiki/Evaluation

Page 37 of 50

Summary of Approaches for Conducting Evaluations Attribute Approach

Key strengths

Key weaknesses

Get, keep or increase influence, power or money.

Secure evidence advantageous to the client in a conflict.

Violates the principle of full & frank disclosure.

Propaganda Needs

Create positive public image.

Secure evidence most likely to bolster public support.

Violates the principles of balanced reporting, justified conclusions, & objectivity.

Causal Relationships

Determine causal relationships between variables.

Strongest paradigm for determining causal relationships.

Requires controlled setting, limits range of evidence, focuses primarily on results.

Scientific Efficiency

Gives Continuously managers supply evidence detailed needed to fund, evidence direct, & control about complex programs. programs.

Human service variables are rarely amenable to the narrow, quantitative definitions needed.

Individual Differences

Produces valid & reliable Compare test evidence in scores of many individuals & performance groups to areas. Very selected norms. familiar to public.

Data usually only on testee performance, overemphasizes test-taking skills, can be poor sample of what is taught or expected.

Organizer

Politically controlled

Public relations

Experimental research

Management information systems

Testing programs

Threats

Purpose

Page 38 of 50

Common sense appeal, widely used, uses behavioral objectives & testing technologies.

Leads to terminal evidence often too narrow to provide basis for judging the value of a program.

Content analysis

Allows for unobtrusive Describe & analysis of Content of a draw conclusion large volumes Communication about a of communication. unstructured, symbolic materials.

Sample may be unrepresentative yet overwhelming in volume. Analysis design often overly simplistic for question.

Accountability

Provide constituents with an accurate accounting of results.

Popular with constituents.

Aimed at improving quality of products and services.

Creates unrest between practitioners & consumers.

Politics often forces premature studies.

Decisions

Provide a knowledge & value base for making & defending decisions.

Encourages use of evaluation to plan & implement needed programs.

Helps justify decisions about plans & actions.

Necessary collaboration between evaluator & decision-maker provides opportunity to bias results.

Policy studies

Broad Issues

Identify and assess potential costs & benefits of competing policies.

Provide general direction for broadly focused actions.

Often corrupted or subverted by politically motivated actions of participants.

Consumeroriented

Generalized Needs &

Judge the relative merits

Independent appraisal to

Might not help practitioners do a

Objectivesbased

Decisionoriented

Objectives

Performance Expectations

Relates outcomes to objectives.

Page 39 of 50

Values, Effects

Adversary Evaluation

Clientcentered

protect practitioners & consumers from shoddy products & services. High public credibility.

better job.

Requires credible & competent evaluators.

Determine if institutions, programs, & personnel should be approved to perform specified functions.

Helps public make informed decisions about quality of organizations & qualifications of personnel.

Standards & guidelines typically emphasize intrinsic criteria to the exclusion of outcome measures.

Critical Guideposts

Critically describe, appraise, & illuminate an object.

Exploits highly developed expertise on subject of interest. Can inspire others to more insightful efforts.

Dependent on small number of experts, making evaluation susceptible to subjectivity, bias, and corruption.

"Hot" Issues

Present the pro & cons of an issue.

Ensures balances presentations of represented perspectives.

Can discourage cooperation, heighten animosities.

Specific Concerns & Issues

Foster understanding of activities & how they are valued in a given setting & from a variety of perspectives.

Practitioners are helped to conduct their own evaluation.

Low external credibility, susceptible to bias in favor of participants.

Accreditation / Standards & Certification Guidelines

Connoisseur

of alternative goods & services.

Note. Adapted and condensed primarily from House (1978) and Stufflebeam & Webster (1980).

Page 40 of 50

Page 41 of 50

Page 42 of 50

References ______ 1. http://en.wikipedia.org/wiki/Data-informed_decision-making 2. Data-Driven Decision Making: Promises and Limits - The CIO Report - WSJ http://blogs.wsj.com/cio/2013/09/27/data-driven-decision-making-promis...

3. http://en.wikipedia.org/wiki/Needs_assessment 4. www.ncstac.org 5. http://www.rotary.org/ridocuments/en_pdf/605c_en.pdf 6. http://en.wikipedia.org/wiki/Project_management 7. http://en.wikipedia.org/wiki/Requirements_management 8. http://en.wikipedia.org/wiki/Evaluation

Page 43 of 50

Page 44 of 50

Attachment A Data-Driven Decision Making A Powerful Tool for School Improvement

Page 45 of 50

A WHITE PAPER

Data-Driven Decision Making: A Powerful Tool for School Improvement

Compliments of:

Overview Schools have been collecting data for decades, but it hasn’t been until recently that most school district leaders have discovered the power of data for promoting school improvement. Much of the recent focus on data has been triggered by the No Child Left Behind (NCLB) Act, legislation that is intended to increase student achievement across all socioeconomic boundaries and improve results at "lowperforming" schools.

However, recent advances in technology and the increased demand for assessing student learning has led many school administrators to discover that the usefulness of data goes far beyond NCLB reporting requirements. Today, forward-thinking districts across the country are employing data-driven decision making techniques not only to analyze test scores and student achievement, but also to: • Narrow achievement gaps between student subgroups • Improve teacher quality • Improve curriculum • Share best practices among schools and districts • Communicate education issues more effectively with key stakeholders • Promote parental involvement in the education process • Increase dialogue within the educational community In its most basic form, data-driven decision making is about: • Collecting appropriate data • Analyzing that data in a meaningful fashion • Getting the data into the hands of the people who need it • Using the data to increase school efficiencies and improve student achievement • Communicating data-driven decisions to key stakeholders But for many districts, implementation of data-driven decision making practices can be daunting. Not only are there a variety of technical challenges to overcome, but resource, financial and data quality issues as well. Getting started can be the toughest challenge of all.

This paper will provide a basic framework for analyzing today’s data-driven decision making options and outline information about the basic elements and steps involved in implementing a data-driven decision making system to facilitate more informed decision making, boost overall school performance and improve student achievement.

A White Paper

|1

The move toward data-driven decision making A brief history of data in schools Data collection in schools is not a new concept. For years, districts have collected a vast array of student and institutional information, including such items as test scores, enrollment data, budget and finance information and human resources data. In fact, district and state administrators have been dealing with continuously expanding data reporting requirements for the past two decades.

In 2002, those responsibilities increased drastically with the passage of the No Child Left Behind (NCLB) Act. Whether or not you agree with the legislation’s scope and intent, NCLB has heightened awareness and attention on student data to a new level across the country.

As a result of NCLB, school administrators are now responsible for monitoring and enabling student and teacher performance improvement, broken down by important subgroups. This kind of reporting typically requires a sophisticated system for data collection and analysis.

Moving into NCLB, all states and most districts have some sort of a data management system in place.

Unfortunately, the average system is often composed of a number of spreadsheets, databases and paper reports that are loosely connected through various interfaces, making it difficult to retrieve and analyze the data in a comprehensive, integrated fashion.

Today, many districts are using NCLB as a catalyst to move to data-driven decision making. The most enlightened districts are using it to improve their technology infrastructure and formalize data collection and analysis procedures, allowing them to make informed decisions based on data rather than assumptions.

For example, many districts are faced with tight budgets and limited resources, having to make tough decisions about cutting programs. With a data-driven decision making system in place, administrators can quickly and easily analyze the correlation between student participation in these programs and other indicators such as student attendance, discipline incidents and student achievement, giving them a clear picture about the effectiveness of each program. When forced to make cuts, ineffective programs can be eliminated based on real-time facts and figures, rather than emotions or assumptions.

Data-driven decision making has opened a new world of opportunities for schools and districts to provide professional educators, students, and parents access to large amounts of information. Today, schools can enable key decision makers with data and information to facilitate more informed decision-making, boost overall school performance and improve student achievement.

Districts could suffer without proper data States and districts need an effective technology infrastructure not only to meet NCLB’s data management and analysis expectations, but also to identify and fix operational inefficiencies and drive improvements in student performance.

If administrators fail to provide the necessary evidence of improvement required through NCLB, their districts could face several repercussions: • A portion of their Title I dollars to support supplemental services and programs could be reallocated • Students could exercise transfer options and enroll in other districts, resulting in decreased funding for those lost students • Schools could face restructuring

2 | Data-Driven Decision Making

But the repercussions go far beyond NCLB performance requirements. Without a formal data analysis system, districts often fail to uncover and address critical issues that occur at the school level. This puts them at risk for missing important opportunities to improve student achievement and attain greater operational efficiencies.

Running reports from a multitude of data sources without an integrated analysis tool also can be costly for school districts. Costs to upkeep all of the different data sources can be high, and it usually requires extra staff and resources to support it all.

Data can be a powerful tool for districts Knowledge is power, and thereâ&#x20AC;&#x2122;s nothing more powerful than data to help district and school leaders develop a solid blueprint with measurable results for continuous improvement. Through the proper use of data, districts can: Narrow achievement gaps. Data provides quantifiable evidence, taking the emotion and guesswork out of what can be tough calls for superintendents and school boards. With an effective data-driven decision making system, states and districts can more easily analyze performance data by important student subgroups, challenge assumptions and address problems at the school and classroom level. On a classroom level, many principals are already using data to determine student composition. If they find that one or two classes are over-populated with low-achieving students, they can allocate extra support resources for those classrooms, or re-distribute students to other classrooms to balance the mix.

Improve teacher quality. Districts can employ data-driven decision making systems to highlight

specific and targeted professional development needs of district staff and make better staff development investments. For example, an analysis of student achievement data can help superintendents understand which instructional strategies are creating the best results and see where additional training might be needed.

Improve curriculum development. A data-driven decision making system allows administrators and teachers to adopt a proactive approach to curriculum design and development. Perceptions data, for instance, can tell superintendents about parent, student and staff satisfaction with the learning environment, which also could reveal areas in need of improvement. Demographic data can be used to provide valuable information about meeting the learning needs of students now and in the future.

Find the root causes of problems. Data helps districts and administrators see things they might not

otherwise see. When data is examined from all angles, it may highlight a program that, although popular, is not helping students learn. Data can help drill down to the root causes of a problem, allowing districts to solve the whole problem and not just the symptom. This gives educators great insight into interventions such as summer school and after-school programs, allowing them to continue to promote effective programs and to modify or discontinue programs that are not working.

Share best practices. Data can provide useful information within and across classes and schools in

formats that educators at all levels can quickly use to determine best practices. These examples of performance excellence can then be shared with other schools and educators, providing the opportunity for staff to learn from each other.

Communicate more effectively with key stakeholders. Instead of responding defensively to critics or the media, data can arm administrators with facts and figures that tell a more complete story and help key audiences understand the root causes of the challenges faced by their schools.

A White Paper

|3

Motivate students and increase parental involvement. By analyzing data, teachers can identify a studentâ&#x20AC;&#x2122;s weakness in a particular subject. Rather than reviewing the entire curriculum, which can be overwhelming for many students, special focus can be given to just the strands the student needs to concentrate on to improve test scores. Teachers can encourage students by showing them how successful they were in other strands, while helping them to focus on the task at hand. This approach also can be very motivating for parents, helping to increase their involvement in everything from nightly homework to long-term educational planning.

Common elements of a data-driven decision making system Districts can choose to implement everything from "homegrown" data-driven decision making solutions to purchasing complete, turnkey data management and analysis technologies and services from a wide variety of vendors. In Making Sense of the Data, a report by Eduventures, Inc. that examines the K-12 data management and analysis market, the author notes that "most systems built internally lack sophisticated technology infrastructures and data collection processes that allow the system to grow with the needs of the state and/or district."

The report goes on to say that "states and districts are finding that partnering with vendors to develop an appropriate data management and analysis (DMA) system can be more efficient than developing a system internally. â&#x20AC;ŚWorking with vendors enables states and districts to streamline their DMA investments, mitigating the costs associated with the unpredictability of the development and maintenance of these systems."

The following illustrates a full-scale, comprehensive data-driven decision making model. Districts may choose to implement all or part of the following model, based on their specific needs and budget considerations.

SIS

HR

Consulting / Needs Assessment

Extraction, Trasformation, and Loading Tool (ETL) Data Warehouse

Data Analysis Tool

Finance Report Writer

Decision Support Tool

Report Writer

Assessment Professional Development / Training

Diverse Data Sources

Information can come from functional district databases, such as Student Information Systems (SIS), Human Resources, Finance and Transportation, as well as Test Data, which could include everything from state tests and benchmark assessments to specialized tests developed on the school or classroom level.

4 | Data-Driven Decision Making

Many districts also have information stored in a vast array of specialized databases, collecting information on items such as special education programs, disciplinary referrals, professional development and teacher certification, technology support help line calls, community survey results and more. These databases shouldn’t be overlooked in the data collection process, as they help to bring a multidimensional perspective to the data.

Extraction, Transformation and Loading Tool (ETL)

Important Features of a Student Data Analysis System User Friendliness • Software is intuitive and easy to use • Software requires little training • Presentation is familiar to user • Access speed is fast and efficient

This is the interface between the databases and the data warehouse. As data passes through the ETL, it is combed for missing, incorrect or inconsistent data.

These discrepancies are flagged, allowing users to remedy problems and maintain the quality of their data.

User Features

Data Warehouse

Information Access

A data warehouse is an organized storage area for data elements that are pulled from the various databases. It is the integration of all data into one central repository. A well-designed and well-built data warehouse can serve as the foundational layer for a strong data-driven decision making system.

Data Analysis Tool

The data analysis tool is the "engine" that drives a datadriven decision making system. It is a user-friendly, non-cryptic reporting and analytical tool that conducts mining, forecasting and analysis of the information that resides in the various data sources and/or data warehouse. It typically offers an integrated reporting tool that allows users to run real-time, pre-formatted and customized reports, putting data into the hands of those who need it most to expedite analysis and improvement efforts.

It also speeds those efforts by reducing the time and effort it takes to manually pull together data from diverse information systems. In just minutes, staff can conduct a detailed analysis of a subject, investigate alternative causes or correlations and analyze the results from a number of different perspectives.

• Comprehensive query tools available for every level of user • Flexible drill-down capability from any form of data aggregation • Data can be accessed from anywhere

• Multiple ways to access the information • Varied methods of representing information (e.g. tables, graphs) • Wide range of data available for analysis • Interface provides immediate access to relevant information • Pre-formatted reports are clear, varied, relevant and comprehensive • Longitudinal presentation of data available at every user level

Creating and Sustaining Quality Data • Provides capacity to enable clean data • Company accepts responsibility to facilitate data process with schools • System allows for expansion past initial implementation • System provides proper security for data transmission • Integration of different areas of information is seamless to the user • Software accepts many common data formats

Additional Features • Online student work samples available • Software exports into common programs • Users can access electronic discussion groups

Decision Support Tool

A decision support tool takes analysis one step further by recommending and prescribing corrective measures to help administrators and educators address problems highlighted by the data analysis tool. These tools foster a culture of continuous improvement by providing recommendations, real-time alerts, and automatic actions for administrators, teachers and staff.

• Easy access to learning standards information • Software offers capacity to link individual teacher data to student data From: “Software Enabling School Improvement Through Analysis of Student Data”. Report No. 67, published by the Center for Research on the Education of Students Placed At Risk, a national research and development center supported by a grant from the Institute of Education Sciences, U.S. Department of Education.

January 2004.

Copyright 2004, The Johns Hopkins University. All rights reserved.

A White Paper

|5

Support Services

Many vendors who offer data-driven decision making solutions also offer consulting support services such as needs assessment, professional development and training. With a needs assessment, the vendor works directly with the district to identify technology, infrastructure, data, educational and organizational goals up-front, providing a virtual roadmap for implementation.

Once the implementation process has begun, it’s important to make sure that system users learn effective, appropriate strategies for applying the data to support and facilitate improved student outcomes and achieve greater operational efficiencies. Some vendors offer professional development and training services as part of their data-driven decision making packages.

At first searching for vendors who offer data-driven decision making solutions can seem daunting. A report published by the Center for Research on the Education of Students Placed At Risk outlines on the previous page the features to look for in a student data analysis system.

Although it’s not likely that any single provider offers all of these features, administrators should identify those features that are most important to their districts and choose a vendor based on identified needs.

Getting started One of the biggest challenges for any district implementing data-driven decision making is knowing where to begin. Often, there is so much data to choose from that the process can be overwhelming.

Key to creating momentum behind any data-driven decision making effort is a proactive leader who understands the vision, champions the cause, helps others in the district realize the impact of data analysis and understands that the entire process takes time.

Who Does What?

Data-driven decision making, especially in the early stages, demands that district leaders point the way.

Superintendents and school boards both must play important but distinct roles.

The superintendent generally: • Translates the board’s vision for the school district into measurable goals based on data • Works with district faculty, staff, parents and other community stakeholders to craft plans for meeting goals by certain dates • Collects data to show clear, steady progress • Celebrates successes, evaluates shortcomings and revises plans for improvement based on data, along with the board

The school board generally: • Establishes a vision for the school district based on data showing what has been achieved so far and what progress is necessary • Spells out, for the superintendent and other employees and stakeholders, how district performance will be evaluated • Reviews relevant data to evaluate district progress toward identified goals • Revises goals and plans for improvement based on data

Broad participation in improvement efforts serves to: • Promote a high level of support for those efforts • Generate sound solutions by expanding the discussion • Motivate participants and their associates • Increase the likelihood that the effort will lead to constructive action

• Prepare participants for their role in implementing improvements • Increase ownership of and commitment to specific strategies • Empower important stakeholder groups • Foster lasting, rather than temporary, change

Source: “At Your Fingertips: Using Everyday Data to Improve Schools”, 1998. In: ”Using Data to Improve Schools: What’s Working”, a report produced by the American Association of School Administrators, 2002.

6 | Data-Driven Decision Making

The best way for the leader to get started is to lay the groundwork for a district-wide shift to a culture of information, education and communication. One way to do that is to find "data champions" throughout the organization. These "believers" should come from all levels within the system and show enthusiasm over the possibilities of data-driven decision making. Their enthusiasm will quickly spread to their peers, building momentum and increasing the likelihood of buy-in at the district level. It may take time, but it is crucial that administrators and other district leaders are onboard to ensure full participation in the process.

Successful integration of data-driven decision making also requires a team approach – particularly between the board of education and the superintendent. The district’s expectations must be clearly articulated, measurable and attainable, and individual roles must be clearly defined. The table on the previous page outlines suggested roles for both the superintendent and the school board:

Overcoming implementation barriers For leaders who are committed to supporting datadriven decision making in their districts, there are a variety of technical challenges to overcome, many of them involving availability and reliability of data.

Some examples include: • A shortage of staff and resources • Difficulties cleaning up data from multiple sources so that they are compatible • Various entry and accuracy errors that, once analyzed, can lead to incorrect conclusions • The need to digitize large quantities of information Quality of data also can be an issue. Data analysis is only as strong as the quality of data from which it is derived. If the data is suspect, concerns can be raised about the quality of decisions that administrators are making based on that data. Losing trust at this stage of the process could make it difficult to rebuild trust moving forward.

Bringing Student Achievement into Focus St. Paul is an urban district serving 43,000 students in Minnesota. With 34 percent of its students identified as Limited English Proficient and a high mobility index of 22 percent, analyzing student achievement proved to be difficult. To better meet the needs of its students, the district piloted a data-driven decision making system at one of its schools, Arlington Senior High School. The trial was set up as a targeted assistance model, serving the “highest” needs students who receive the most assistance.

To do that, staff needed to examine test data on a regular basis and use it to identify the most academically challenged students in order to improve their test scores and provide them with an opportunity to graduate. Easier said than done.

Prior to implementing the data-driven decision making system, teachers would start the year with little or no knowledge of their students’ past performance. Data was only available for students who had taken tests in their building the previous spring, and it was not easy to use and required extra time for the teacher to retrieve. As a result of these obstacles, teachers used only overall test scores and pass/fail status.

Data-driven decision making changed all of that. Upon implementation of data analysis techniques, teachers were able to look at scale score improvements on standardized tests to evaluate their curriculum, instruction and services. They could look at strands within a test to determine trends in performance and adjust curriculum and instruction accordingly. They also looked at strands to identify and assist students through the school’s comprehensive tutoring program.

Rather than having to key in data manually, the student information was updated nightly, giving teachers real-time information about the students sitting in front of them.

They could review classes by overall test score/performance and drill down to the strand/sub skill level to see how students performed.

As a result of looking at data by class and grade level, Arlington began curriculum mapping and a building-wide literacy initiative. Seeing the data in "black and white" and taking the time to understand and interpret the data helped Arlington solve critical building-wide curriculum and instructional issues.

The project was so successful that efforts are underway to expand the data-driven decision making system’s use across the district.

To overcome these barriers, many districts have opted to pilot their data-driven decision making systems on a smaller scale with a specific school, program or group of students. By showing success on this level, they are able to build trust, create "data believers" and increase the likelihood of buy-in on a district level.

A White Paper

|7

Many districts also are leveraging the power of cooperatives to ease the burden of implementing datadriven decision making systems. By working as a team to set universal goals, explore shared data opportunities and identify common problem areas, district staff and educators not only enjoy the synergies created by bringing multiple perspectives to the table, they also are able to build momentum and share time and resources on matters such as staffing, training and data entry.

Five simple implementation steps Once there’s buy-in and district-wide support for a move toward data-driven decision making, formal implementation can begin. Although the task may look daunting at first, breaking it down into smaller, more manageable steps will make the process go much easier. While each district’s needs and resources may differ, the following steps outline some of the basic activities needed for successful implementation of a data-driven decision making system

1. Conduct an information inventory/audit A good place to begin is to ask the question: What do we want to learn? As districts go about answering that question, they should consider a broad vision of data collection. Think beyond just high-stakes test scores. When it comes to student learning, no one test can possibly provide a full picture of what students understand and can do in relation to national or local standards and curricula.

Once districts have determined what it is they want to learn, a good next step is to conduct an information audit to identify data that is already being collected and determine if the various data sources are compatible. If not, districts should identify the obstacles to making those connections work.

Next, districts need to determine what additional data is needed. The idea here is to take a multidimensional approach to data collection. Think creatively and go beyond the confines of traditional transactional databases, gathering both qualitative and quantitative data. Qualitative information, which usually describes what people say, do, think or feel, puts a human face on quantitative data. Examples of targeted qualitative data sources include: • Surveys and questionnaires (of teachers, students, parents, employers, community members, etc.)

• Interviews or focus groups • Teacher logs/diaries • Classroom observations of actual instructional practices and student responses • Alternative assessments (e.g. work samples, portfolios, senior projects and performance tasks) • Locally developed pretests and posttests The final item to determine is how frequently important data needs to be collected – daily, weekly, monthly, yearly or over multiple years.

2. Standardize data management In its simplest form, data-driven decision making is all about correlating data elements and exploring those factors that contribute both positively and negatively to student and teacher performance. Correlating data elements can be nearly impossible if the process for collecting them hasn’t been standardized.

To begin, consider developing data standards that apply to the whole organization. Some basic standards could include eliminating paper systems, entering all data directly into compatible computer systems and across networks, and organizing data sets with universal keys.

8 | Data-Driven Decision Making

It’s also important to assign ownership for specific data elements. A clear chain of accountability will result in greater data quality and integrity.

This also is an ideal time in the process to assess the district’s information systems to determine if they can handle the data collection and analysis that needs to be done. The first option should be to maximize existing information systems. If that’s not possible, it may be time to begin searching for compatible data management vendors.

3. Analyze the data Education reform is a hot topic that will continue to be scrutinized by the media, politicians and community stakeholders for years to come. With such intense focus on school improvement, it is vital that any data collected be interpreted accurately and fairly and communicated clearly. Employing the following steps will improve that process: Provide ongoing training for system users. Data-driven decision making requires new knowledge and skills.

System users need to know fundamental spreadsheet and database techniques such as filtering, sorting and creating tables and graphs. They also need to be comfortable with fundamental data analysis concepts such as causation, correlation and disaggregation. Make sure that the vendor you choose offers training as part of its implementation package.

Take a long-term approach to data collection. Reacting

to a single test score is perhaps the most common mistake made when interpreting data. Longitudinal measurement – conducted consistently from year to year — is necessary to properly measure progress, growth and change.

Drill down into data to find the real picture. The drilldown process is an effective method of disaggregating data. It begins with a general question, then "drills down" the question into smaller and smaller parts.

While drilling down into data can be rewarding, it can also be misleading. For instance, the further you drill down into a set of data, the smaller the sample size available. As a result, the less accurate your conclusions. Be careful not to drill down so far that the data group is too small to draw acceptable conclusions.

Key Analysis Techniques Disaggregation: Breaking data down to find out what a number looks like for different sub-groups hidden within an average or basic percentage. Users typically do this with a drill-down process, which begins with a general question, followed by increasingly specific questions that focus on smaller subsets of data.

Longitudinal data: Data measured consistently over a period of weeks, months or years to track progress, growth and/or change over time. True longitudinal studies eliminate any students who were not present and tested in each of the years of the study.

Cross-tabulation: Cross-tabulation is comparing data among multiple sets or subgroups. For example, you could start by looking at a simple statistic by one characteristic, such as race. Next, you could cross-tabulate it by an important educational opportunity such as passing rates or reading scores.

Interpret and share results as they become available. It’s not necessary to wait until all data is collected

before analysis begins. Early in the process, districts can begin to observe important trends that will lead to more informed decisions once all the information is available. The practical side of data analysis involves helping teachers and administrators learn how to interpret data and respond with the best resources and strategies for implementation.

Create a culture of information in the district. It’s important to get the data into the hands of the people

who need it most – teachers, administrators and other stakeholders – and to encourage these groups to engage in dialogues that will help them come to a deeper and shared understanding of the data. According to Nancy Love, author of the comprehensive guide Using Data/Getting Results, "These dialogues help participants gain skills in data analysis. They learn to separate data from inference, to bring out multiple perspectives, to test out interpretations of data with additional data and relevant research, and to explore not just obvious explanations, but the root causes of problems."

A White Paper

|9

4. Strive for continuous improvement Increasingly, school districts are using data-driven decision making to ensure continuous improvement.

Once districts have identified relationships or gaps in data, they can take the most important step – making changes and defining new strategies. Whether the decision for change is big or small, the key is to make the most informed decision possible given the data that is available.

There is no end to data analysis, and once district staff understand how to effectively use data, it will become easier and easier for them to identify new opportunities for collecting and analyzing data.

Districts must continue to look at answers to old questions, include new information as it becomes available and make new, more informed decisions. Once these decisions have been made, the process of identifying relationships and implementing remedies begins again.

Forward-looking districts also move beyond the initial work of assessing their own schools’ performance by reaching out beyond their borders and benchmarking their schools’ performance against that of other topperforming schools across the country. Benchmarking is more than comparing test scores. Done well, benchmarking helps districts learn what it takes to be effective, how well they are using key strategies, where improvements are possible or necessary, and how they can learn from the best practices of other successful districts.

5. Communicate results An effective way to build public support and increase community confidence is to show key stakeholders how districts and schools are being held accountable for results. Sharing data in easy-to-read charts and short, jargon-free reports not only lets community members know that schools are making informed decisions based on data, but also can create a deeper community understanding of the issues facing public education.

User-Friendly Report Cards Here are some helpful guidelines to ensure that a school’s report card is read and used: • Keep school report cards short, such as a sixpanel brochure —Have a more detailed version available for people who want more data • Add short narrative descriptions—School data are not self-explanatory to non-experts • Advise parents and others how they can use the data—It’s not obvious to most • Spend as much time working on distribution as on production—Research shows that even most teachers say they’ve never seen district report cards Source: “Reporting Results: What the Public Wants to Know About Schools”, 1999. In: Using Data to Improve Schools: What’s Working, a report produced by the American Association of School Administrators, 2002.

10 | Data-Driven Decision Making

Districts shouldn’t rely on the media alone to communicate the message. Limited time and space, combined with a lack of understanding of the complete information, often lead them to report only the basic facts, which are usually misinterpreted because the details behind the data are not provided.

Many successful districts are reporting information directly to their communities. One method for doing so is a school report card or annual performance report. This report is used to guide discussions on performance and education priorities at various school forums, and at parent, staff and school board meetings.

Training teachers and principals to help facilitate these conversations ensures that everyone is focused on the data, what the data reveals about performance and how to improve instruction in the classroom.

Conclusion Data-driven decision making goes well beyond simply complying with NCLB performance requirements.

It can serve as a powerful process for districts to facilitate more informed decision making, boost overall school performance and improve student achievement.

Key to successful implementation of data-driven decision making is an outspoken leader who understands the vision, champions the cause and helps others in the district realize the impact of data analysis. Finding and using "data champions" throughout the district is an important strategy, creating enthusiasm at all levels and building a district-wide culture of information, education and communication.

Achieving district-wide support takes time. One successful strategy for getting started is piloting datadriven decision making initiatives on a smaller scale, building trust and buy-in from the ground up. Other districts have leveraged the power of cooperatives to ease the burden and tackle the process as a team, sharing resources and benefiting from the synergies that bringing multiple perspectives to the table can bring.

Once education leaders have committed to the process and there is district-wide support for the program, getting started can be the biggest challenge. Breaking the process down into smaller, more manageable steps can ease the burden on individual staff members or departments and significantly improve your districtâ&#x20AC;&#x2122;s chances for successful implementation.

School districts donâ&#x20AC;&#x2122;t have to go through the process alone. Several technology vendors on the market offer everything from simple, easy-to-implement data analysis modules to complete, turnkey data management and analysis technologies and services. When your district is interviewing technology vendors, be sure to identify the features you need up front and choose a vendor based on those needs.

When used appropriately, data-driven decision making can be a powerful process for schools and districts.

It can help to: narrow achievement gaps, improve teacher quality, improve curriculum development, promote better communication with key stakeholders, motivate students and enhance parental involvement in the education process. But even more important than that, it can help districts maximize the use of limited funds to achieve the best impact possible on student achievement.

A White Paper

| 11

Resources "The Administrator’s Guide to Data-Driven Decision Making." Todd McIntire. Technology & Learning, June 2002.

"Cooking with Data to Reduce Achievement Gaps." Craig Jerald. ENC Focus, electronic version, Volume 10, Number 1.

"Data Analysis in Administrators’ Hands. An Oxymoron?" Theodore B. Creighton.

The School Administrator, April 2001.

"Data in Your Hands." Raymond Yeagley. The School Administrator, April 2001.

"Data: Mining with a Mission." Judy Salpeter. Technology & Learning, March 2004.

"How Data Can Help: Putting Information to Work to Raise Student Achievement." Jane Armstrong and Katy Anthes. American School Board Journal, November 2001.

"Improving Teaching and Learning with Data-Based Decisions: Asking the Right Questions and Acting on the Answers." Nancy Protheroe. ERS Spectrum, Summer 2001.

"An Interview with Nancy Love: Building a Professional Learning Community." Ken Mayer. ENC Focus, electronic version, Volume 10, Number 1.

"Making Sense of the Data. Overview of the K-12 Data Management and Analysis Market."

A report produced by Eduventures, Inc., November 2003.

"Software Enabling School Improvement through Analysis of Student Data." Report No. 67, published by the Center for Research on the Education of Students Placed At Risk, a national research and development center supported by a grant from the Institute of Education Sciences, U.S. Department of Education; January 2004. For a full copy of the report: www.csos.jhu.edu/systemics/datause.htm.

"Turning Skeptics into Supporters." Elaine M. Coffin and Laura M. Seese. ENC Focus, electronic version, Volume 10, Number 1.

"Uses and Abuses of Data." Nancy Love. ENC Focus, electronic version, Volume 10, Number 1.

"Using Data to Improve Schools: What’s Working." A report produced by the American Association of School Administrators, 2002.

About the Author: John Messelt has 32 years of experience in the Education field as a Teacher in Special Education, a Director of Special Education, Superintendent of Schools, and currently the Executive Director of the Central Minnesota Educational Research and Development Council. He has an undergraduate degree in Industrial Education, as well as a graduate degree in Special Education from St. Cloud State University and an Educational Specialist degree from the University of St. Thomas. He has been an active member of the Minnesota Association of School Administrators and has served as President of the Minnesota Administrators of Special Education. For the past ten years, he has been an associate of The Cambridge Group, providing strategic planning services throughout the United States and Canada. Through his consulting work, he provided many organizations with training in planning, team-building, and conflict management.

12 | Data-Driven Decision Making

3601 Minnesota Drive Minneapolis, Minnesota 55435 1-800-533-5430 U.S. and Canada 1-507-725-5411 Worldwide 1-507-725-2301 Fax info@sagebrushcorp.com www.sagebrushcorp.com

Š Copyright 2004 Sagebrush Corporation. All rights reserved.

STXX04213 0554-00010002

Page 46 of 50

Attachment B 10 Things You Always Wanted To Know About Data-Driven Decision Making

Page 47 of 50

10 Things You Always Wanted To Know About Data-Driven Decision Ma...

1 of 4

Teachers

Parents

Kids

Ad ministrators

Librarians

http://www.scholastic.com/browse/article.jsp?id=423&print=2

Reading Club Book Fairs

Sign in -or- Reg ister

10 Things You Always Wanted To Know About Data-Driven Decision Making Everyone's talking about D3M. Use this guide to help preven t al l that data from drivin g you n uts.

By Pamela Wheaton Shorr | September 2003

Way bac k in the 1980s, a new way of doing busines s evolved as corporations began collecting, c ombining, and crunching data from s ources throughout the enterpris e. Their goal was to improve the bottom line by disc erning hidden patterns and thereby improving the dec ision making proces s. Two decades later, the No Child Left Behind (NCLB) law is pus hing s chool distric ts to do the same—this time, of cours e, the goal is increased student achievement.

While data alone won't provide sc hool leaders with the finess e, experience, and intuition needed to get kids on the right learning path, a growing number of administrators are convinced that the data-driven decis ion-making (D3M) proc ess can fundamentally change education— from our unders tanding of what really works with kids to administrative proc es ses and profes sional development.

To ass is t you in getting your head around the realities and ramific ations of data-driven decis ion making (and, yes, to make the s ubject a bit livelier), we've compiled this lis t of 10 truths. Some of thes e things will entice you; others will s urpris e you; a c ouple are sure to momentarily paralyze you. But, at least, now you know.

1. If you 'r e not using d ata to make decision s, you 'r e flying blin d.

Imagine you're piloting a plane at night, in the middle of a storm. W ithout data from your navigation instruments, you'd be a goner. "Running a s chool without a data warehouse is like being a pilot without an ins trument rating," says Brian Osborne, s upervis or of evaluation, as ses sment, and research at the Plainfield Public Sc hools in New Jers ey.

"This is why there's so muc h policy c hurn in sc hools. Administrators simply don't have enough information to make good dec is ions." The Plainfield sc hools began us ing a data warehous e and analys is solution from eSc holar three years ago. Although Osborne is a rec ent arrival in the distric t, he says he quickly bec ame a big fan of the tec hnology's power to help sc hool leaders find the right flight path toward student achievement.

2. Th is is all abou t a pr ocess, n ot a specific tech no logy.

Traditionally, D 3M solutions have centered around a data warehous e— a central repository that collects data from many different sourc es —in combination with high-end decis ion-support tools that run queries and reports on the data. But in fact, a wide variety of tec hnologies and s olutions c an be used to support a D3M proc es s. Harry Hayes, superintendent of the Bloomfield, N ew Mexico, school dis trict, says the D 3M process should always begin with these questions: "H ow's busines s? H ow do you know it? And what can you do to improve it?" H ayes is a big proponent of the management-review process , and uses N orthwes t Evaluation As sociation (N WEA) tes t data to help him analyze teaching strategies and make better-informed dec isions about profes sional development. Gregory S. Decker, principal of Lead Mine Elementary School in Raleigh, North C arolina, analyzes s chool data using Microsoft Acc es s database s oftware. He also uses Pearson's Suc cess Maker management sys tem for s tudent ass essments and on-demand reports. D ecker and his staff have spent the past four years mapping, benc hmarking, and making predic tions about where each child— and eac h teac her —s hould be at every stage, and he s ays test sc ores have s oared as a result.

10/2/2014 10:07 AM

10 Things You Always Wanted To Know About Data-Driven Decision Ma...

2 of 4

http://www.scholastic.com/browse/article.jsp?id=423&print=2

3. Get r ead y to feel th reatened.

"The downs ide to data-driven dec ision making is that it makes people muc h more ac countable; there are no corners in which to hide," says Robert Ewy, direc tor of planning at C ommunity Cons olidated Sc hool D is trict 15 in the suburbs of Chic ago. Ewy runs MicroStrategy's dec ision-support tools on an IBM data warehouse. "Before we began us ing a data warehouse, we thought we were making dec isions based on data," Ewy says , "but now the quality has dramatically improved." Of course, the idea behind D3M is improvement, not punis hment— and it's not just about the quality of teac hing, but about the quality of leaders hip. "Data-driven decision making s ucc eeds if the educ ational administration c an model the habit of mind involved in data inquiry," Osborne says. "It's a culture shift, a new learning paradigm." H e says that rather than having the answers all the time, educators need to c ome up with the right questions. Still, as king the right ques tions and interpreting the data are not the same as c oming up with s olutions, Hayes warns . "Data are s imply indicators, tools to be used," he s ays . "They are a means , not the ends in thems elves." Proponents of D3M in sc hools say this is where the quality is sue really comes in: Once there is evidenc e of problems, do administrators and teac hers have the chops to address the problems effec tively?

4. You will be spending more mon ey, not less.

Irene Spero, vice pres ident and project director of the D3M initiative at the Cons ortium for School Networking (C oSN), s ays adminis trators need to prepare for a cos tly long haul when implementing data-driven decision making. "The first year is all about setting goals in the community and distric t. Year two is about roll-out and implementation, and it's not until years three or four that you can really s ee the effects ," s he explains.T hat investment in time can translate to a hefty price tag. Regardless of whether you've bought an expens ive solution or not, the proces s its elf demands a serious commitment to training, data cleanup, and maintenance that is often overlooked. "Every three to s ix months there mus t be a review of the data validation proc esses and cleansing of the data to ensure quality," s ay s Katie E. Lovett, chief information offic er of Fulton County Sc hools, in Atlanta, Georgia. "End users must be trained on the tool, but more important is training on how to us e data for analys is and understanding how the data is structured in the data warehouse. And users also mus t be trained in data-warehous e security."

5. Data-driven decision making does not save time.

Don't start planning a vac ation with all the time you think you're going to save hav ing D 3M in your distric t, warns Timothy L. Sc haap, data analyst for Township H igh School Dis trict 214 in Arlington Heights, Illinois. "Data-driven decision making refocuses your time," Sc haap s ay s, "but it is definitely not a times aver." Schaap believes D 3M is akin to doing ac tion res earch—a process of s imultaneously pursuing c hange and understanding, continuous ly refining methods and interpretation in light of what's been learned. Schaap notes that both adminis trators and teachers have to take owners hip in the ac tionres earc h process —and that means more work, not les s.

6. Your data's clean lin ess is next to Godlin ess.

You've heard the express ion, "garbage in, garbage out." T he trans actional sys tems that supply data to your warehouse— including s tudent information s ystems, instructional management sys tems , tes ting applications , and financial software—are us ed and updated daily by staff members throughout your dis trict, making data-entry errors a regular fact of life. You may als o disc over stagnant "pools" of data that have been forgotten and aren't regularly updated. If you don't c lean up this "dirty" data, your entire D3M initiative could be worthles s. The proc ess of c leaning trans actional data and exporting it to a data warehous e can be c omplicated and time-cons uming, says Shawn Bay, founder and C EO of eScholar. The bigges t risk in implementing a D3M s olution is that people may think it's a lot easier than it really is, he s ays . "W orking with data is very deceptive," Bay s ays, "and educational data is the most complex I've ever worked with."

The les son: Make sure your selected vendor has expertis e in data cleans ing, and write it into the contrac t.

10/2/2014 10:07 AM

10 Things You Always Wanted To Know About Data-Driven Decision Ma...

3 of 4

http://www.scholastic.com/browse/article.jsp?id=423&print=2

7. Don 't shoot first an d ask questions later.

You s hould figure out exac tly what ques tions you want the data to answer before you tackle the is sue of whether to buy a turnkey D3M solution or build your own us ing off-the-shelf components. "It's abs olutely critic al to know that before you dec ide," warns Illinois's Ewy. Some dis tricts go it alone, others s witc h vendors along the way, and s till others c reate "blended" s olutions, supplementing products and services as needed. Dic k Barkey, exec utive direc tor of information technology for the Adams 12 Five Star Sc hools in Thornton, Colorado, s ays his district decided to build its own data warehouse s olution becaus e no c ommercial applic ations met the dis trict's needs. Called Scholars Mart, the homegrown solution was c oded entirely in-house using standard software development tools suc h as Microsoft's SQL Serv er and Visual Bas ic . "The Sc holars Mart sys tem is tightly linked to our SASI s tudent information sys tem from Pearson," Barkey explains .

Teac hers in his dis trict can ac cess information about c urrent or prior students as well as state and dis trict test res ults, and schedule changes are updated daily.

8. A go od D3M solu tion is one you can affor d to ch an ge.

Don't lay out a lot of c ash up front, Schaap says. The reason: It's hard to walk away from a lemon when you've inves ted a lot—and you never know when you may want to chuc k the whole thing and start ov er. D is trict needs and product features change, Schaap explains , s o it's important to cons ider initial costs as well as interoperability. Odds are, you won't have to bag a s ystem that is adaptable, however. The same D 3M tools can often be used for very different goals : David Heis tad, exec utive direc tor of tes ting, evaluation, and student information in the Minneapolis Public Schools , uses the NW EA item banks to create tests aligned to state and local standards. In contras t, Bloomfield, New Mexic o, superintendent Harry Hayes uses the NW EA s olution for profess ional development.

9. NLC B is ju st the beginn in g o f your jour ney.

Although NCLB is driving the momentum for data-driven dec ision making, Spero says, school dis tricts s hould view D3M solutions in the c ontext of sweeping and system-wide school improvement efforts— not as simply a tool to meet federal mandates . "This is an opportunity to go way beyond what's required," Spero s ays. Heistad agrees : "One mus t go beyond the current federal model of c omparing this year's third graders to las t year's third grades and comparing average tes t s cores for one site vers us another." He s ay s he believes that sc hools should be rewarded and supported bas ed on the degree to whic h every s tudent is making progress towards high s tandards— in addition to the number of s tudents who have "jumped over" the bar.

10. Wor d of war ning: D3M is h igh ly ad dictive.

Raymond Yeagley, superintendent of the Roches ter (NH ) Public Schools , recalls a school board meeting in whic h a board member as ked how s tudents were doing in math.

"With the web-bas ed Quality Sc hool Portfolio, two minutes later I had the ans wer," he says. That kind of power is not just addic tive— it's c ontagious . Getting that information to the people who need it—and in a form that they c an use—is critic al. The Minneapolis schools offer paper reports for any one who cannot acc es s the reports on-line; thes e are distributed to sc hool teams, parents , and the community. Barkey advises that regardless of how the information is trans mitted, it's important to make sure that it is presented in a simple list, table, or drill-down form with good built-in explanations and definitions. "The more you use a data warehous e," c oncludes Ewy, "the more you want to use it."

Reprodu cib les: D3M Buyer's Guide (PDF)

About the Author

10/2/2014 10:07 AM

10 Things You Always Wanted To Know About Data-Driven Decision Ma...

4 of 4

http://www.scholastic.com/browse/article.jsp?id=423&print=2

Pamela Wheaton Shorr is editor of The Heller Reports ' Educ ational Sales and Marketing Insider, and is a frequent c ontributor to Scholas tic Administr@ tor.

About Scholastic

Our Website

Who We Are Corporate Responsibility

Teachers Parents

Media Room Investor Relations International

The Stacks (Ages 8-12) Family Playground (Ages 3-7) Librarians

Scholastic en Español Careers

Administrators Common Core Standards

Need Help?

Join Us Online

Customer Service Contact Us

PRIVACY POLICY · Terms of Use · T M ® & © 2014 Scholastic Inc. All Rights Reserved.

10/2/2014 10:07 AM

Notes ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 48 of 50

Notes ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Page 49 of 50

Page 50 of 50