OpenStreetsTO - A Summative Evaluation Proposal

OpenStreetsTO A Summative Evaluation Proposal Conducted for TABIA

PLG 830 - Policy Analysis and Program Evaluation Instructure Dave Guyadeen Nikolas Koschany, Thanuja Mannie, Ramiya Rajalingam April 4, 2016

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Background First established during the Summer of 2014 as a pilot project for the City of Toronto, Open Streets Toronto was developed in part as a build up to the Pan Am 2015 Games. A program where the streets are opened to the public and closed to vehicular traffic, people replace cars and streets become “paved parks”, promoting physical activity and allowing opportunities to explore the city in healthy and enjoyable ways (OpenStreetsTO, 2013).

Background Context Originating from Bogota, Columbia in 1974, the OpenStreetsTO Program is defined as a “community building and social engagement exercise that provides free recreational opportunities and public space for people to meet, interact and make social connections” (Williams, 2014). The intent of establishing this program in Toronto is to allow the public to enjoy selected open streets as if they were a “large linear park” with various uses including that of walking, biking, skating, dancing and other health inspired programming that aim to animate the existing street life. Planned as an ongoing initiative by the City of Toronto, the program is aimed to encourage urban health and community outreach by connecting members of the community through pedestrian interaction (Williams, 2014); the program has also been touted for its economic benefits.

As a new not-for-profit organization, OpenStreetsTO was organized to run the annual program every summer (Williams, 2014). In July 2013, 8-80 Cities had secured $180,000 in grants from the Ministry of Tourism and Culture (Ngao, 2014) as well as additional funding from corporate sponsorship and private funding to help manage the Open Streets Program in four cities during 2014, including that of Toronto, Kingston, Windsor, and Thunder Bay (Williams, 2014). The route in Toronto includes a 15 kilometers of open streets with “soft” roadway closures along Bloor Street, from Christie Street to Parliament Street and along Yonge Street from Bloor Street to Queen Street (Williams, 2014). These soft closures allow for participants to stop at major intersections, allowing for regular north-south traffic to commence (Ngao, 2014).

OpenStreetsTO Program Objectives The OpenStreetsTO program aims to: ●

Define the legacy of a transformative program in Toronto (similar to that of Nuit Blanche).

●

Deliver a program with the tangible qualities that make a city great

●

Establish an impactful east-to-west and north-to-south connector in Toronto on its iconic streets

●

Become a program, attract partners, that collectively become leaders in health promotion

●

Become part of the magnetic system that brings Toronto together through an innovative repurposing of existing public assets 1

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016 ●

Create a foundation for high profile, recurring events of local business to increase vitality in uniquely meaningful ways (OpenStreetsTO, 2013)

Intended Users and Uses of Program The OpenStreetsTO initiative is intended for and invites the general public to meet, interact, and make social connections with each other while on some of the most popular streets in Toronto (The Concept, 2013). It does so by increasing pedestrian and cyclist mobility throughout the route by allowing visitors to choose their mode of travel, whether it be cycling, walking, skating or even using the space as a playground; and is focused on exercise rather than entertainment. Since there are no food stalls or vendors during OpenStreetsTO, all purchases that happen along the route benefits local businesses (FAQ-OpenStreetsTO, 2013).

Purpose and Goals of Evaluation The purpose of this evaluation to be completed over the course of OpenStreetsTO 2016, is to determine whether TABIA should continue to contribute money to OpenStreetsTO, by assessing whether local businesses along the route benefit from the program. The goals of this program, are to determine whether OpenStreetsTO increases revenue of local business, whether OpenStreetsTO increases the number of purchases from local businesses, and determine what types of businesses receive the greatest volume of people.

This evaluation is not to determine whether OpenStreetsTO should occur or not, but rather, to determine whether TABIA’s money should be spent on subsidizing and promoting the event or whether it could be better utilized on other expenditures.

Stakeholders in Evaluation Stakeholders in this evaluation include Toronto councilors, as well as various BIAs and non-profit organizations. Ward 27 and Ward 28 Councilors Kristyn Wong–Tam and Pam McConnell are active proponent of the program, given it primarily runs through their wards; residents of those wards, as well as Ward 20 and Ward 19, which the route also runs through are also largely affected. 8-80 Cities, a non-profit organization working under the direction of Gil Penalosa (the man who greatly expanded the original Ciclovía in Bogota Colombia), is also an active stakeholder, given that they are the parent company to OpenStreetsTO. Various Business Improvement Areas, including the Downtown Yonge BIA, the Bloor-Annex BIA, the Bloor-Yorkville BIA, and the KoreaTown BIA, play a further important role, both by assisting with the permit application process and encouraging businesses to participate, and by promoting the event.

2

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Lastly, evaluation stakeholders include various nonprofits and volunteer groups, including the Ryerson Association of Planning Students, who host activities at so-called “activity hubs� along the route, such as yoga, blackout poetry policy, chalk drawing, and others. The decision TABIA makes whether to continue partially funding the program will have wide-ranging effects on many of these stakeholders, from budgeting to community activism, to event promotion.

Logic Model

Key Evaluation Questions The key question being asked, is: Does OpenStreetsTO improve commercial activity for businesses on its route?

The following sub questions will guide the evaluation process to answer the main question: 1. Does OpenStreetsTO increase the revenue of local businesses?

2. Does OpenStreetsTO increase the number of purchases from local businesses relative to people during the event?

3. What type of businesses see the most people during OpenStreetTO? Evaluation Design The evaluation will be summative, and take place both during the project implementation (ongoing), and after (ex-post) the program has ended. The intent of the evaluation is to assess whether the

3

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

program has reached its intended outcome, rather than measure its process. As such, the evaluation will use a strictly quantitative data set, to determine absolutely, the answers to the above three subquestion. Data will be verified by triangulating it so it comes from multiple BIAs, and can be disaggregated if need be to ascertain trends.

Data Matrix In this section, the sub questions addressed above have been placed into a matrix to show how, from where, and over what timeframe we intend to evaluate them. The components of this matrix are addressed in sections below.

Indicators and Data Sources The following section is broken down by evaluation question to best explain the proposed indicators and data sources for each.

Question 1 - Does OpenStreetsTO increase the revenue of local businesses during the event?

Aggregated revenue data from the BIAs will be used as an indicator to answer the evaluation question. A spike in sales generated on days when OpenStreetsTO is operating will be attribute to a positive effect of the program. Three sets of data will be retrieved: revenue on OpenStreetsTO Sundays, revenue on normal Sundays, and revenue on aggregate days. Generally collection of the first two sets of data is sufficient for tracking whether the initiative has had an impact on the businesses. However, the latter data sets will provide for a more refined understanding of whether there are any real increases despite the normal commercial activity patterns. A comparison will be drawn and where there is a difference in revenues earned will inform the analyst whether the initiative has had a positive effect or not.

4

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

The revenue data will be collected in aggregate through existing BIA revenue charts. Though this data could also be collected through disaggregated business surveys, the primary concern with this method is that individual business owners may decline to participate.

Question 2- Does OpenStreetsTO Increase the number of purchases from local businesses during the event?

The number of transactions that occurred in businesses along the OpenStreetsTO route, along with the number of individual business visitors, will be collected and used as an indicator to determine whether the program results in a change in purchasing patterns.

Data will be gathered by collecting transaction data (number of purchases, but not the product or price) for OpenStreetsTO Sundays and normal Sundays. Comparing the transaction numbers of Sundays when OpenStreetsTO does and does not operate will determine whether the program increases purchases from businesses. In addition to this, the number of visitors an individual business attracts can be quantified using “trackers� which measure the entries and exits from the building.

The number of transactions can be assessed in conjunction with business visitors using a conversion analysis. Conversion is a measure of how many transactions were made compared to the foot traffic the business saw, and is important because it informs owners whether they are maximizing their sales potential. A simple increase in transactions and number of people seen alone can be interpreted as positive effect, but when compared, it could determine that the business did not effectively utilize the exposure from OpenStreetsTO to their complete advantage.

Question 3 - What types of businesses are seeing the most people at OpenStreetsTO? The number of transactions that occur at different business types, will be used as an indicator to determine if OpenStreetsTO attracts more people to spend at certain types of businesses.

Similar to the data collection for the second question, the number of transactions from each business en-route will be recorded. Surveys will then be used to identify the type of business associated with each transaction record, to determine which business types received the most purchases. This information will inform whether a type of business is more favourable and whether these businesses see an increase in exposure in comparison to other businesses due to OpenStreetsTO. Project Management Plan The following Gantt Chart is meant to provide an implementation plan for this evaluation, to occur from July to October, 2016. This chart will provide a clear timeframe for the project, which we believe is doable within a four-month period, from beginning to end.

5

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

6

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Communication Method Even the best data collection and analysis can be hindered if the data from an evaluation is not displayed through its most effective output. To this end, this section of the evaluation proposal details how the results for each evaluation questions’ data shall be presented, accompanied by an illustrative example.

Question 1 - Does OpenStreetsTO increase the revenue of local businesses during the event?

The results showing whether there is increased revenue for local businesses from the OpenStreetsTO program will be shown as a bar graph. This will display data for revenues received on days when OpenStreetsTO occurs, Sundays when it does not occur, and aggregated data showing average revenues for the months of September and October (the two months when OpenStreetsTO occurs).

Question 2- Does OpenStreetsTO Increase the number of purchases from local businesses during the event?

The results showing whether there are an increased number of purchases at businesses in relation to pedestrian traffic will be displayed on a map of the OpenStreetsTO route with a colour scale. The average conversion rate of businesses along the OpenStreetsTO route for a normal week, will be used as an average in the creation of the map, with any change above this average being shown in green, and anything below this average being shown in red.

7

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Question 3 - What types of businesses are seeing the most people at OpenStreetsTO? The results showing the types of businesses profiting the most (and least) from OpenStreetsTO (if at all), will be displayed in a bar graph. Further to this, a map will be provided showing the areas with the most and least number of poor performing businesses.

8

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Evaluation Reporting The entirety of the evaluation shall be put into a report and presented at the Toronto Association of Business Improvement Areas (TABIA) on October 18th. The report shall be broken down into the following sections: ●

Executive Summary

●

Program Context1

●

Evaluation focus

●

Data gathering procedures

●

Results

●

Recommendations

●

Dissemination and Sharing Plan

The dissemination and sharing plan will depend on the outcome of the evaluation. If it is determined that OpenStreetsTO is improving commercial activity for businesses along its route, the report will be released physically and online, and promoted through social media channels such as Facebook and Twitter, on various BIA pages, and through OpenStreetsTO, announcing the news in support of current use of funds to help the program.

1

Inclusive of a logic model for the entire program.

9

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

If, however, it is determined that OpenStreetsTO is decreasing commercial activity at businesses along the route, the report will be released in a much more nuanced way, so as to not incite negative press against the BIAs in question. Due to the political support for OpenStreetsTO, and its promotion of fitness, health, and social activities, it is unlikely that an evaluation showing poor economic outcomes will prevent the event from occurring. However, the purpose of this evaluation is not to determine whether OpenStreetsTO should occur or not, but rather, to determine whether TABIA’s money should be spent on subsidizing and promoting the event.

In such a circumstance where the evaluation results are negative, the report will be presented privately at TABIA, rather than publicly; TABIA would pull funding from the event, to minimize its losses, and the funding would be put towards programs in the area with a better economic return.

Evaluation Constraints and Limitations The evaluation may be constrained by several factors. An evaluation is only as good as the data it collects, and to begin, the largest risk to the evaluation would be if the revenue data asked for from the BIAs is not provided. Should this not come through, the first question in this evaluation would need to provide entirely different data collection methods in order to be answered, potentially through individual business surveys rather than aggregated financial returns. This could lead to an incomplete data-set, thereby skewing the quality of data. There is also the potential for skewed or misleading data to be provided to the evaluators in order to create a pre-determined result.

Misinterpretation of data may also create spurious relationship or false-associations between datasets. The fact that some businesses may not have trackers, or transaction records in their stores also poses another data collection problem.

Costs and funding pose another potential constraint. As this evaluation would be funded collectively

by TABIA and the Business Improvement Areas affected, and not by the government, depending on market conditions the evaluation may not be funded. The current backlash against unpaid internships may also drive up evaluation costs, given that volunteers are slated to perform surveys as part of our data collection methods.

Lastly, while this evaluation is summative, it cannot be used to generalize the success of the OpenStreetsTO program as a whole, given it only measures the economic impact of such an event on local businesses. Therefore, a risk in this evaluation is blatant misinterpretation of one policy goal for another.

10

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

Conclusion OpenStreetsTO remains seen by many as a program that improves public health, social wellness and interaction, and community activism. The effect that OpenStreetsTO has on business however, is less clear. Through this evaluation, we will determine if TABIA should continue to invest in the program.

And should we determine that TABIA’s investment into OpenStreetsTO is not providing adequate returns, we will admit it frankly, and advise investment somewhere else.

11

PLG 830 - Policy Analysis and Program Evaluation Nikolas Koschany, Thanuja Mannie, & Ramiya Rajalingam April 4, 2016

References FAQ- OpenStreetsTO. (2013). Retrieved April 03, 2016, from http://www.openstreetsto.org/#!about1/c1o2h Fact Sheet- OpenStreetsTO. (2014). Retrieved April 3, 2016, from http://baycloverhill.com/bcca/2015/Events/Open Streets TO Fact Sheet.pdf Healthiest Practice Open Streets. (n.d.). Retrieved April 03, 2016, from http://healthiestpracticeopenstreets.org/ Hurd, D. (2015, August 07). Ciclovias Are Opening Up Cities to People Around the Globe. Retrieved April 03, 2016, from https://momentummag.com/six-ciclovias-opening-cities-people-aroundglobe/ Ngao, E. (2014, July 07). Opening the Streets of Toronto. Retrieved April 03, 2016, from http://rusustainability.ca/2014/07/open-streets-to/ OpenStreetsTO. (n.d.). Retrieved April 03, 2016, from http://www.openstreetsto.org/ TABIA (2015). Who We Are. Retreived April 03, 2016, from: http://www.toronto-bia.com/index.php Williams, M. H. (2014, March 31). OPEN STREETS TORONTO (Rep.). Retrieved April 03, 2016, from http://www.toronto.ca/legdocs/mmis/2014/ed/bgrd/backgroundfile-68197.pdf

12