D2.3 Community Based Evaluation by COMRADES EU Project

www.comrades-project.eu

D2.3 Community Based Evaluation Project acronym: Project full title:

COMRADES Collective Platform for Community Resilience and Social Innovation during Crises

Grant agreement no.:

687847

Responsible:

Tina Comes

Contributors:

Kenny Meesters, Christophe Nemnom

Document Reference:

D2.3

Dissemination Level:

<PU>

Version:

Final

Date:

23/06/17

Disclaimer: This document reflects only the author's view and the Commission is not responsible for any use that may be made of the information it contains.

This project has received funding from the European Unionâ&#x20AC;&#x2122;s Horizon 2020 research and innovation programme under grant agreement No 687847

D2.3 Socio-Technical Community Requirements

History Version

Date

Modification reason

Modified by

0.1

30-05-2017

Initial draft

Tina Comes

0.2

10-06-2017

Revision outline, setup sessions

Kenny Meesters

0.3

15-06-2017

Session outline, theory

Kenny Meesters

0.4

17-06-2017

First results

Kenny Meesters

0.5

19-06-2017

Review draft

Kenny Meesters

0.6

20-06-2017

Updates workshop results

0.7

21-06-2017

Input from developers

0.8

22-06-2017

Final input

Kenny Meesters Christophe Nemnom GrĂŠgoire Burel Kalina Bontcheva Shadrock Roberts Kenny Meesters

1.0

23-06-2017

Final version

Kenny Meesters Harith Halani

2 | 51

D2.3 Socio-Technical Community Requirements

Table of contents 1

Introduction

1.1

COMRADES Validation

1.2

Communities & Coordination

1.3

Structure of this document

1.4

Relation to other deliverables and work packages

Coordination theory as evaluation framework

2.1

Introduction to Coordination Theory

2.2

Coordination & Community resilience

2.3

Coordination & Information exchange

2.4

COMRADES

2.5

Evaluation framework

Setup of the sessions

3.1

Part 1: COMRADES Introduction (~25min)

3.2

Part 2: Requirements (~35min)

3.3

Part 3: Platform (~45min)

Evaluation Data Collection & Analysis

4.1

The COMRADES ambition and objective

4.2

Approach & Requirements Analysis

4.3

The COMRADES platform & development roadmap

4.4

Data collection

Evaluation participants

Workshops

6.1

Online workshops

6.2

In-person workshop

Internal evaluation

Results from Technical Evaluations

8.1

YODIE-service

8.2

CREES

Results from Workshops

9.1

Online workshop with digital humanitarian network

9.2

In-person workshop IDP, China

10 Discussions & Conclusion 10.1

Theoretical Findings

42 42

3 | 51

D2.3 Socio-Technical Community Requirements

10.2

Results

10.3

Implications for COMRADES

10.4

Next Steps & outlook for evaluation

46 48

Appendix A.1 References

A.2 Guiding questions for focus group

A.3. Invite to Evaluation Workshop

List of figures Figure 1 Evolution of information gaps & quality ....................................................................... 8 Figure 2 COMRADES Communities ............................................................................................ 11 Figure 3: Components of Coordination in a Disaster Context ................................................... 14 Figure 4: Common Dependencies among Activities and Processes for Managing them .......... 14 Figure 5 Resilience as a process, in red (adapted from The Conservation of Change) .............. 17 Figure 6 Decision making & Information ................................................................................... 18 Figure 7 COMRADES prototype platform used in the evaluation ............................................. 19 Figure 8 User communities and COMRADES WP2 engagement ............................................... 25 Figure 9 Digital humanitarian network structure ...................................................................... 27 Figure 10 Disaster Management Cycle, illustrating different stages & media focus ................. 35 Figure 11 Workshop at Chinese Institute for Disaster Prevention ............................................ 38

4 | 51

D2.3 Socio-Technical Community Requirements

Executive summary COMRADES (Collective Platform for Community Resilience and Social Innovation during Crises, www.comrades-project.eu) aims to empower communities with intelligent sociotechnical solutions to help them reconnect, respond to, and recover from crisis situations. This deliverable presents the first evaluation of the COMRADES platform. It builds on the theoretical framework of resilience developed in D2.1, and the socio-technical community requirements described in D2.2. As such it uses both sets of requirements and combines them with theories on coordination in disaster management to evaluate the first built of the COMRADES platform. The content of this deliverable is based on: 1. Theoretical research on coordination in disaster management 2. An internal evaluation and self-reporting on the ongoing developments within the COMRADES project. 3. A series of online workshops and training sessions to evaluate the project in general and the platform in particular with key stakeholders of COMRADES. This deliverable consists of three parts. In the first part of this deliverable we present the evaluation approach. Coordination theory is introduced as the theoretical background, focusing on building community resilience through improved coordination. This section also introduces the setup and design of the COMRADES evaluation sections along with the approach for the data collection and analysis. The following section introduces the actual planned evaluation sessions, including the practical organization and the attending participants. In the final section, we present the results and finding of the both the internal evaluation and the workshop sessions. We conclude by synthesizing the key findings and look forward to the next steps in the development of the COMRADES platform and project. The first evaluation of the platform and project presented in this deliverable is centered around three main topics. The verification of the COMRADES objective and ambition (1), mainly focused around the COMRADES definition of resilience. We focus on not only verifying the motivation behind this definition, but also aim to assess the added value and potential impact (as described in D2.1). The requirements that are resulting from this definition and those that are further explored in the Nepal workshops (results in D2.2). We additionally explore important considerations for successful development and implementation of the project. Finally, we examine the current prototype platform and asses how the current functions and features contribute to the COMRADES objectives and address the requirements. We also examine what features and functions could be added to further improve the match between the projectâ&#x20AC;&#x2122;s ambition and the ongoing and planned development. These points have been discussed and explored with different groups in both off- and online focus group sessions. These sessions were conducted with members of the Digital Humanitarian Network, representing the COMRADES Volunteer & Technical community, and the professional responders affiliated to the Chinese Institute of Disaster prevention. These groups do not only complement our existing community interactions, which previously focused on reports and affected communities, but also increase the geographical and cultural coverage of the COMRADES project, adding to a more global perspective. The data collected from the sessions is completed with evaluation data from internal assessments. This selfreported information will help further verify the results and determine the path forward. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

5 | 51

D2.3 Socio-Technical Community Requirements The results of the evaluation have been very positive. Although the platform is still in the early stages, and during the evaluations the platform prototype mainly served as an illustration of the COMRADES objective, the workshop participants have responded positively to the demonstrations and hands-on sessions. The platform also helped to demonstrate the project ambition and goals. Overall, the participants appreciated the workshops and were engaged throughout the session, actively providing input and feedback into the discussions. Participants have expressed their interest to stay involved in the COMRADES project and to stay up to date on further project developments. The evaluation focused on the three topics mentioned above. First the verification of the COMRADES objective and ambition, along with the COMRADES definition of resilience. The participants underlined the importance of improved coordination between different stakeholders. Workshop participants also recognized that information exchange and situational awareness is an important prerequisite for an improved interaction between different stakeholders. Also, the importance of considering the role of the government (either local, regional, or national) is mentioned as an important consideration for the successful adoption of the COMRADES ideas. The participants emphasized however, that the COMRADES ambition builds on ongoing trends and developments in the humanitarian and crisis domain. These advantages include building on the local knowledge already existing in the community, and creating cooperation between different stakeholders. Finally, the increased demand for situational awareness and information for different stakeholders in a crisis response was mentioned as an important trend that COMRADES builds on. Regarding the requirements, the workshops have shown that there are important elements that need to be considered in the development of the platform. These are in part technical requirements, ensuring that the platform and services can handle different contexts, ranging from dense urban environments, to more remote local communities. The differences will affect the technical requirements for the platform, but moreover the adoption of the platform in the local community context. For example, the alignment with the local organizational structure. It should also be noted that the information needs of communities and their members shift over time, from personal (in the early stages after a critical event) to more community focused perspectives (in the later stages). These findings align closely with the socio-technical requirements identified earlier in the COMRADES project (outlined D2.2). Finally, the participants much appreciated the prototype platform. The interactive sessions gave the participants a guided, hands-on opportunity to use the platform and the integrated services. Using both real-world and fictional examples it was demonstrated how the platform could be configured and the services used to support information extraction. Participants did mention the importance of support and training when such a platform would be deployed, as not all communities will have the necessary knowledge and capabilities themselves. Finally, the integration of the platform in non-crisis situations (daily life) was mentioned as a key element for adoption. The participants however did see the potential of the platform and highlighted the added value of the automated services that support communities in managing, analyzing, and sharing the incoming information to a wide range of stakeholders. Overall, the evaluation has shown that the current developments are in line with both the COMRADES objectives and ambition, and the expectations of the workshop participants. The COMRADES project itself, specifically the objective and ambition, are building on trends and developments in the humanitarian and crisis domain. Moreover, both the COMRADES project and platform already provide at this stage tangible results that further develop and advance the approach to community driven resilience building. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

6 | 51

D2.3 Socio-Technical Community Requirements

Introduction

COMRADES is a three-year project with the aim to create an open-source, community resilience platform financed by the European Union Horizon 2020 Research and Innovation Framework Programme H2020-ICT-2015. This deliverable presents the first evaluation of the first version of the COMRADES platform based on the resilience framework developed in D2.1 and the community requirements elicited in the field research in Nepal (presented in D2.2). This first version of COMRADES is tested with control groups, using the Nepal case and resulting questions as a backdrop. Specifically, this first evaluation of the intermediate results of the COMRADES project focused on three major components. First, the verification of the COMRADES objective, ambition, and approach as outlined in Deliverable 2.1, validating the focus of the project. Second, the evaluation examines and validates the requirements for delivering the COMRADES objective and ambition as outlined in Deliverable 2.2. Finally, the evaluation looks at the current platform prototype, put forward by the joint work of WP3,4 and 5. The platform is used to further validate the requirements, examine the functions and features provided, and provide input to the development roadmap.

1.1

COMRADES Validation

In this deliverable, we combine technical and social evaluation of the platform, in terms of specific functionalities and community requirements. The overall objective of the deliverable is to verify the results of the project thus far, specifically validating the COMRADES objective and socio-technical requirements, and design the roadmap moving forward. It serves to both verify the COMRADES requirements and ensure alignment with the technical developments. COMRADES objective and ambition Specifically, we are looking to validate the COMRADES ambition and objective as outlined in Deliverable 2.1. Central to this ambition is the COMRADES definition for community based resilience building: “Continuously enhancing community driven resilience through ICT entails enabling a broad range of actors to acquire a relevant, consistent, and coherent understanding of a stressing situation, empower decision makers and trigger community engagement on response and recovery efforts, including long term mitigation and preparation. “ On one hand, as Turoff, Chumer, Van de Walle and Yao (2004) stated in their seminal paper on Dynamic Emergency Response Management Systems (DERMIS), the unpredictable nature of a crisis implies that the exact actions and responsibilities of geographically dispersed individuals and teams cannot be predetermined. Therefore, an information system that aims to improve community resilience and support the local response should be able to support assigning decision power to the most operational level. In addition, and to increase transparency and reciprocity (Van de Walle & Comes, 2015), also reverse flow should be supported to improve accountability and communicate status information upward and sideways throughout any formal or informal responding organization.

7 | 51

D2.3 Socio-Technical Community Requirements One of the main findings in scholarly literature on community resilience is that a crisis may come in various forms, including either as an acute crisis or more akin to a chronic stressor, and resilience seen as a process. Moreover, we have noted that establishing a coherent understanding and a sense of control in a crisis will aid immediate responses and possibly also contribute to healing and recovery afterwards. Information and decision-making form part of our emphasis on coherence. Grasping the complexities of information and decision-making seems essential when thinking about community resilience as a process. On the other hand, considering the (potential) role of information to contribute to resilience in these stages, we recognize that an information platform could and should contribute to two different elements, depicted in the graph below (Figure 1). However, the role of information, or the significance of information changes over time as the crisis response shifts across different stages. First, an information platform enables a reduction of uncertainty, inconsistency, and information gaps [red]. Secondly, a platform supports an improvement of the quality of information (timely, relevant, actionable, ...) [green]. These elements present different requirements and settings for the platform dependent on the phase. Below we outline the different phases and identify different requirements associated with each of these. As can be noted, throughout these phases the focus rests on providing a coherent understanding and empowering decision makers, while also triggering community engagement on response and recovery efforts.

Figure 1 Evolution of information gaps & quality

Requirements This objectives and ambitions shave been further explored in interviews with stakeholders, including the preliminary interviews conducted with various Ushahidi deployers, and the fieldwork conducted in Nepal in January 2017. This research has resulted in the COMRADES socio-technical requirements.

8 | 51

D2.3 Socio-Technical Community Requirements The COMRADES platform should support local communities moving from the current ad-hoc information management practices, to a digital services culture, not only of data but also a repository of public awareness, knowledge, skills, and information. The main requirements from the various community-engagement activities we carried are the following: Front-end ● The integration with existing platforms that are already in use by the community. For example, allow posts to the COMRADES platform to be submitted via Facebook. Or allow visual outputs of the COMRADES platform, such as maps and timelines, to be embedded in Facebook pages ● Existing platforms can also support the security and community management, for example, examine whether Facebook can be used as authentication for the COMRADES platform. ● There are multiple platforms that could serve as a frontend. COMRADES could identify other social media platforms that, like Twitter, provide a public API and would be appropriate for integration with the COMRADES platform Feedback loops ● A key element is reciprocity. For example, by ensuring and showing community members that share information how they can benefit from these actions. ● To enable easier feedback loops, ‘barriers’ for posting and updating information should be removed. For example. develop method for allowing users to receive alerts or make posts without being signed up to a deployment Enterprise/Federated architecture ● An enterprise structure might be needed since many communities might have their own platforms, with support from governing bodies or international organizations reaching multiple communities. One approach is to develop a federated approach to the COMRADES platform, by allowing multiple deployments to feed into a larger common deployment. Trust and Validations ● Trust in a platform and the presented information is key to adoption. COMRADES needs to design and develop methods for assessing information validity in the COMRADES platform Situational Awareness ● Communities need to act on certain triggers that affect their (current state of) resilience. COMRADES needs to develop a method for identifying the emergence of events (e.g., food request) from the content supplied to the COMRADES platform In addition to the design principles of DERMIS, the field work has also uncovered requirements that are not included in the DERMIS principles, stemming for example from the fact that the affected community would be the direct user, or the more cyclic nature of a system tailored towards sustained community-based resilience building approach. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

9 | 51

D2.3 Socio-Technical Community Requirements COMRADES prototype A final element in this evaluation is the prototype platform, delivering some of the functionality and features. This platform is the result of the developments in Work Package 3, 4 and 5. These developments have resulted in new services, and features added to the new Ushahidi platform. These developed features and functions include the integration of several online services and added functionality to the Ushahidi platform. Specifically, the following features been developed and delivered: ● The integration of the YODIE (Yet another Open Data Information Extraction system) service. This service performs named entity recognition and disambiguation against DBpedia across varied/noisy/short input text.1 The service identifies key entities in the text of posts and links them to the DBpedia knowledgebase. ● The integration of the CREES service, providing automatic categorization based on the provided text. The service examines the text of the posts and identifies the relevant categories, supporting users for example when dealing with a larger number of incoming messages. ● Updates on the Ushahidi platform, integrating the above-mentioned services and aligning the development processes. Furthermore, additional features based on the input and requirements gathered from the field work in Nepal, such as Facebook integration. While many other features and services are currently being developed as part of the COMRADES project, this evaluation is mainly focused on these features which are currently part of the prototype platform. While the ongoing developments are not part of this evaluation, they can be evaluated regarding contributing to the requirements and overall COMRADES objective listed above.

1.2

Communities & Coordination

In addition to the internal validation of the COMRADES results thus far, we examine COMRADES in the context of supporting the interaction between different communities or user-groups. As illustrated in Deliverable 2.1, access to information is critical to support the community driven resilience building process. The different communities defined in COMRADES each contribute in their own way to this improved information exchange, either by providing or consuming information, or by facilitating the information exchange. COMRADES’ goal is to further enhance this interchange between the communities. Specifically, COMRADES is centered around the interaction between 4 of these communities. (1) The affected community; the people affected by a critical event who may need aid and support, (2) the responders; organizations or individuals who provide aid and assistance to affected communities, (3) Reporters; individuals, communities, or organizations who are supplying information, for example eye-witness reports, and (4) The volunteer & technical communities; those facilitating the exchange of information between the different groups, for example by providing and supporting information and communication technology. As illustrated in the figure below, there is an overlap between the different groups, as people, organizations and communities may fulfill multiple roles.

https://gate.ac.uk/applications/yodie.html

10 | 51

D2.3 Socio-Technical Community Requirements

Figure 2 COMRADES Communities

To analyze how the platform supports interaction and information exchange across all COMRADES communities we additionally base this evaluation on theoretical background in coordination theory. This deliverable therefore has three main objectives: ● ● ●

1.3

Improve the theoretical / scientific understanding of communication platforms to improve resilience through (better) coordination Provide feedback to COMRADES developers and directions for further improvements Continue to engage the COMRADES user communities

Structure of this document

This deliverable consists of 3 parts, describing the stages of our evaluation approach. Part I: User Communities and Context The first section outlines the context and setup of the evaluation. In this section, the selection process for the participants of the evaluation is described, ensuring they provide sufficient, relevant and quality feedback to the project. We also present the context of the evaluation, and the relation to the evaluation objectives. In addition, the sessions and participants also describe the other sources used for the evaluation and feedback. Part II: Theoretical Background and Evaluation method The next section provides the input and framework for conducting the evaluations. These are on the one hand the input for the evaluation, the current output, and results of the COMRADES project, as presented in the WP2 deliverables and the prototype framework. It also establishes the framework for collecting the feedback of the participants in evaluation sessions. This framework provides a linking pin between the specific feedback collected from this evaluation based on the presented results in the session and how these can be translated to overall findings and linked to the COMRADES evaluation goal described in Deliverable 2.1. This framework is most notably based on coordination theory. Part III: Results The final part of this deliverable provides the results of the evaluation. These results encompass both the feedback and input received from the evaluation sessions, as well as the results from internal evaluations on specific components of the COMRADES platform. The combined results are compared to the overall (validated) objectives and requirements outlined in the previous deliverables. Finally, we outline some key considerations for moving © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

11 | 51

D2.3 Socio-Technical Community Requirements forward with the COMRADES project, specifically for the development roadmap and the alignment between the ongoing developments and the socio-technical requirements.

1.4

Relation to other deliverables and work packages

The work presented in this deliverable has a strong relation with other deliverables and work packages in the COMRADES project. Work package 2 As illustrated above, the evaluation builds on the objectives and requirements laid out in deliverable 2.1 and 2.2 respectively. The evaluation serves to verify and validate the ideas and results of these deliverables. It also provides additional input to the (socio-technical) requirements of the COMRADES project Technical developments The evaluation also builds on the combined work of the ‘technical’ work-packages (WP3, 4 and 5) and uses the current prototype platform as part of the evaluation which includes the ongoing developments. The evaluation focusses on the validation of the current features and functions provided in the prototype. Vice-versa, the evaluation aims to provide input into the development roadmap. Integration A key aspect of the evaluation is to examine the alignment between the objectives and requirements outlined in Work Package 2, and the ongoing work in technical development of the COMRADES platform.

12 | 51

D2.3 Socio-Technical Community Requirements

Part I: Evaluation design This first section describes the design and setup of the first evaluation sessions. This setup of the evaluation is based on the evaluation setup as it is introduced in Deliverable 2.1: section 7.3 Scenario 1 (M18). As the evaluation focuses in part on the validation of the COMRADES objective, as well as on the specific implementation of derived features and functionalities in the prototype platform, the evaluation is based on the Coordination Theory, specifically on the exchange of information as enabler to align activities between stakeholders and create synergy. In addition, due to the experimental nature of the platform, and the further refinement of the COMRADES requirements, the evaluation has been conducted in the form of interactive workshops or focus groups. Guidance was provided to the participants with regards to the rationale behind the COMRADES project, and to validating the current design and requirements and soliciting input for further expansions.

2 Coordination theory as evaluation framework Coordination theory is applied to multi-actor systems. Similarly to the COMRADES project, these systems bring together actors with different mandates, motivations, organizational structures, resources, and knowledge. Coordination theory examines the interplay between these stakeholders and identifies key elements for successful collaboration. As COMRADES also aims to bring together different actors (the COMRADES communities) Coordination theory is used at the framework for the evaluation: examine how COMRADES support bringing the different communities together.

2.1

Introduction to Coordination Theory

Coordination theory is an interdisciplinary field which occurs in many domains such as, economy, computer science and organizations. Most of the literature reviews study coordination under interdisciplinary approaches in order to characterize different kinds of dependencies and to identify the coordination processes which help to manage those dependencies (Malone and Crowston, 1994). A well-defined term helps with both filling the gap between the perception of the concept and reality, and exposing possibilities of starting points for studying the concept itself. For our purpose, the definition which fists best has been given by Malone and Crowston (1994) and commonly used in field: ‘Coordination is the process of managing dependencies among activities’. According to the literature on coordination theory, there are three main components of coordination: activities, actors and dependencies (Malone, 1990). However, most of the literature reviews do not consider the dynamic vision of the problems encountered in-fields when a disaster occurs. In order to bridge the gap between theory and practice, this consideration leads to reconsider the components of coordination in the following way. There are three high-level components: systemic, organizational, and operational components. For each high-level component, there are also three sub-components as presented in Figure 3 below. This conceptual map mainly highlights both the shared and the dynamic dimension of coordination.

13 | 51

D2.3 Socio-Technical Community Requirements

Figure 3: Components of Coordination in a Disaster Context

Indeed, one of the most important features of crisis situations is an ever-changing environment. For instance, the responders should give an adapted response for a specific context while bearing in mind the key principles inculcated during trainings and/or emergency exercises. It is for this reason that coordination components and a fortiori coordination processes are reframed to bridge the gap between static challenges highlighted in the literature and dynamic situations encountered in-field before, during and after disasters. The common dependencies between activities and alternative coordination processes for managing them are presented in the following table (Malone and Crowston, 1994).

14 | 51

D2.3 Socio-Technical Community Requirements Furthermore, an important matter is to know and understand how is coordination theory applicable to crisis situations. A broad answer is to lever up on information systems and infrastructures which will help to empower communities by giving them to some extent a global vision of the local context. Finally, the fundamental importance of coordination theory is to develop a dynamic shared perception of a specific situation based on standardized principles and processes which help stakeholders to reduce uncertainties due to dependencies among activities.

2.2

Coordination & Community resilience

The main objective of this part is to link coordination theory to community disaster resilience. According to the definition of coordination theory, it is possible to identify dependencies between different stakeholders (responders, volunteers, and technical communities, affected communities and reporters) before, during and after crisis situations. Moreover, in the past, coordination theory has focused strongly on responders and professional agencies. However, the community is also a strong stakeholder as illustrated in D2.1 and need to be part of the emergency response, decision making and coordination processes. Central at the COMRADES definition of community resilience is the shared situational understanding between all stakeholders, on which coordination structures can be built. Relating this definition to Coordination theory, the components of coordination link the COMRADES communities and coordination theory. In order to be effective, communities, have to be involved to some extent in the coordination components. This interaction improves both the community engagement and the mutual awareness which are common evaluation criteria of community disaster resilience.

2.3

Coordination & Information exchange

An important part of coordination is the exchange of knowledge and information. Information exchange is one element in coordination theory which is detailed in this evaluation. Indeed, information systems and infrastructures enable different stakeholders to better insights, such as risk and options for mitigation and disaster preparedness and make better decisions. Information is one of the major component of coordination and one of the most important lever of community disaster resilience. Indicators such as translation of knowledge and skills to disaster preparedness, mitigation, social network development, communication infrastructures and technologies, community engagement, inter- and intra-community communications are fundamental in the capital-based approach of community disaster resilience improvement (Cox and Hamlen, 2014). In addition, by the application of coordination in crisis situations and the community involvement in the management of dependencies listed in the table above, the community empowerment and engagement are fostered while education (skills and knowledge) is encouraged. On the other hand, as a lot of relevant information exists between the stakeholders, fostering coordination to information systems and infrastructures ensures that everyone in the system benefits from each other.

2.4

COMRADES

COMRADES aims to support information exchange, which supports an improved coordination and contributes to improved community disaster resilience. To enable meaningful and effective information exchange, the platform needs to facilitate the exchange of information between different stakeholders. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

15 | 51

D2.3 Socio-Technical Community Requirements To complete this objective, it is necessary to collect, analyze, share and archive information that is relevant to a specific stakeholder in a different way. At the same time, these stakeholders also supply different types of information, for example information supplied by the national government might be broad and apply to a larger area, while information from certain community members are only applicable to a small area. COMRADES needs to be able to transform information between these different layers and convert this information in a manner / situation that is applicable to situation / scope of the users receiving that information. For example, the platform needs to be able to ‘aggregate information’ and ‘break it down’ depending on the user and their information needs. In other words, the COMRADES platform needs to serve relevant information. For instance, in each context, the shared components of coordination facilitate the community involvement in information exchange. Hence, data collection is accelerated, information analysis is more precise, information sharing is more secure and archiving is clearer and/or better defined.

2.5

Evaluation framework

Coordination theory introduces different elements that are critical components of the COMRADES project. The interaction, coordination and alignment between different stakeholders is a key element of the project. More specifically, one of the key elements in realizing this alignment is the facilitation of information exchange between different stakeholders. This information exchange supports the development of shared situational understanding among the stakeholders, including the local community. This enables the stakeholders to make informed decisions, considering the actions undertaken by others. Furthermore, coordination theory is supplemented by the evaluation criteria from Deliverable 2.1, which have been based on the DERMIS principles. This combination forms the basis for the evaluation framework used in the evaluation sessions. By employing these evaluation frameworks, the COMRADES project is able to evaluate its work so far on multiple levels: (1) the overall project ambition, objective and approach, (2) the technical and socio-economic requirements for the successful adoption of the platform and ideas, and (3) the platform and its specific functions and features it should provide to enable a more effective coordination between the different COMRADES communities. The above introduced coordination theory, linked to the COMRADES definition of community resilience provide several key elements that are relevant to this mid-term evaluation of the COMRADES project and its results thus far. We examine specifically in the evaluation sessions if COMRADES: ● ●

●

Facilitates the coordination between different stakeholders, specifically including the community as key stakeholder. Supports information exchange between different stakeholders to improve a shared situational understanding considering the different information needs, again, specifically for the community. Considers the relevant socio-technical requirements for the development, deployment, and adoption of platform to support this information exchange and coordination.

16 | 51

D2.3 Socio-Technical Community Requirements

Setup of the sessions

At this stage of the project we aim to evaluate and discuss the first results; both the technical developments and the ongoing work on the community processes and the embedment of the system therein. While several components have been developed, tested, and implemented in the platform, they are not mature enough to be used without proper guidance and supervision. However, feedback on the current state of the art will help to confirm the direction of the development and potentially provide insights to further develop the platform. In addition, the evaluation is seeking to validate the COMRADES objective and socio-technical requirements. Therefore, this evaluation will be in the form of a more guided workshop. In this workshop, we will present several elements of the COMRADES project as listed in the evaluation objectives, grouped in three sections. In each of these sections weâ&#x20AC;&#x2122;ll introduce the rationale followed in the project and (1) validate this rationale, the assumptions, and the expected outcomes, and (2) gather additional input to further improve the quality or impact of the project. The evaluation sessions for COMRADES have been based on the facilitating of information exchange between the above-mentioned community. To ensure participants to the evaluation can provide the right feedback, the evaluation has been set up to not only introduce the current prototype platform but also to focus on the rationale behind the COMRADES project. An example program and invitation is included in Appendix 2. The evaluation sessions have been designed as 2-hour sessions consisting of three parts in accordance with the evaluation scope as described in the introduction:

3.1

Part 1: COMRADES Introduction (~25min)

Following a general introduction of the session (in which the objective and agenda of the session is introduced and consent for participating in the evaluation is asked) the first part of the evaluation session focused on the COMRADES project itself. This part of the evaluation session is largely based on the work outlined in Deliverable 2.1, and is aimed at validating the COMRADES objective and definition. This entails illustrating and validating the rationale behind the COMRADES definition of resilience, describing resilience as not only a specific state of a system with certain assets to deal with shocks (absorptive capacity), but also as the capacity to continuously enable the system to move to a more resilient state over time, as illustrated in the figure below.

17 | 51

D2.3 Socio-Technical Community Requirements Based on the introduction of the COMRADES perspective on resilience, we examine the changing landscape of a crisis response. Most notably the changing role of the community, regarding other stakeholders. This changing role encompasses the empowerment of local communities as key decision makers. Finally, the session part introduces information as a key enabler for the community as empowered decision makers. More specifically, how information exchange between the different COMRADES user communities ensures an overall improvement and specifically a more resilient community. Based on these topics (resilience, empowered communities, and information exchange), the following guiding questions were put forward in this first part of the evaluation: ● ● ● ●

What role do different stakeholders, and especially communities currently play in a disaster or crisis response? How does community engagement contribute to a more resilient community? What would be the benefits this engagement provides? To enable resilient communities, building on engaged communities, what role should communities play in a disaster response? How does the interaction between the stakeholders change the current coordination between the stakeholders? How does information exchange support these changes?

3.2

Part 2: Requirements (~35min)

After the introduction of the COMRADES project, the objectives, and the rationale behind the focus on information as a key enabler for community driven resilience, the evaluation session focuses more specifically on the requirements for enabling the information exchange between the COMRADES communities. This part of the session is mainly based on the results from the COMRADES workshops conducted in Nepal and the results thereof presented in Deliverable 2.2. This part of the workshop presented the currently identified requirements as identified and how they are derived and related to the overall COMRADES objectives. In a similar fashion to the workshops conducted in Nepal, the decision-making process is used to illustrate this concept, as illustrated in Figure 6.

Figure 6 Decision making & Information

In the previous section, we have identified the community as key decision makers, and how information empowers these actors as key stakeholders. Facilitating these communities in their (continuous) decision making process, by providing information is a key aspect of the COMRADES platform. Information enables communities to identify which decisions need to be made, the different alternatives to choose from and how each of these options can (or could be) evaluated, the implementation (and communication) of decisions made, and finally how the results of the decision and implementation can be evaluated to further improve the decision-making process. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

18 | 51

D2.3 Socio-Technical Community Requirements The above illustrated process is not linear, but rather cyclic and is happening continuously for different stakeholders, including the community overall and individual citizens (specifically users) of the system. In addition, each of these users makes different kinds of decisions and requires different information. In addition, as observed during the Nepal workshops, there are important socio-technical requirements to consider, such as embedding any information system in a sustainable a way in the community and lowering the adoption threshold. This section of the evaluation used this introduction into the role of information for improved community empowerment and supporting the resilience building process to validate the identified requirements thus far, and to identify additional technical, organizational, operational, and social considerations for the project moving forward. Specifically, the following guiding questions, based on the coordination and information exchange between the different communities and stakeholder groups, were used: ● ● ● ●

What are the key elements which help the main stakeholders to be coordinated during a disaster? What relationships exist between them? What elements should be considered in a system to empower communities? Besides technical considerations, what other requirements should we keep in mind? Coordination is the process of managing dependencies among activities. What is a good ‘coordinated approach’ in the COMRADES context?

3.3

Part 3: Platform (~45min)

In the last part of the evaluation approach we focus on the prototype platform with the current integrated services. The platform demonstration serves multiple purposes in this round of evaluation. First and foremost, the platform serves to demonstrate a possible instance of the COMRADES objectives, and requirements that are introduced in the previous sections of the evaluation. In this regard, it will help to show the evaluation participants how the objectives and requirements could be implemented in a real-life platform. This demonstration helps not only the participants to better comprehend the COMRADES ideas but also to demonstrate how these can be applied to specific use-cases. Furthermore, it will also serve as an additional validation of the COMRADES project ideas as in the previous sections the participants have been introduced to the rationale behind COMRADES, the demonstration of the platform will confirm if these ideas are interpreted in a similar fashion by the participants as by the COMRADES project itself.

19 | 51

D2.3 Socio-Technical Community Requirements The COMRADES platform was demonstrated in the session, using specific and tailored usecases to demonstrate the workflows of the COMRADE platform. Specifically, the use-case of citizens providing various reports about an incident was used for example with a flood, earthquake or other major incident affecting a large population. The choice of use case was dependent on the participating users in the evaluation sessions, to ensure it is closely aligned with their background, experience, and interest. Next, a general introduction of the Ushahidi platform is given, showing the basic functions of the (version 3) Ushahidi platform from both reporters’ perspective (those providing information) and the moderators/admins perspective (those processing the posts). Finally, the evaluation demonstrates COMRADES specific functionality added to the COMRADES platform, the CREES auto categorization and the YODIE annotation services. Using pre-tested examples, these services are demonstrated, as well as with live examples provided from the audience. During this demonstration, it is made clear to the participations that these services and the platform are early prototypes and are mainly demonstrated to illustrate potential elements of the COMRADES concept and to solicit input from the evaluation participants. Following the demonstration of these functions and features, and describing the relation to the COMRADES objective and requirement introduced in the previous sections, the evaluation asks the following guiding questions: ●

Do the current platform features contribute to improved community resilience? What features and functions are missing? ● Are the previous elements implemented in the platform? If so, how is it represented? Otherwise, how could they be represented and/or implemented? ● What other elements and/or functions could be implemented in the COMRADES platform in order to perform an effective coordinated approach in disasters? At the closing of the evaluation sessions, contact details are exchanged for further follow ups. Users were encouraged to give further thoughts on the COMRADES project and platform, and to share their future insights with the project staff. The session is closed with a recap of the initial feedback received and the final word is given to the participants to ask any questions or to provide any last-minute feedback.

20 | 51

D2.3 Socio-Technical Community Requirements

Evaluation Data Collection & Analysis

The COMRADES evaluation focuses on three main components, as illustrated in the breakdown of the evaluation session above: the validation and verification of the COMRADES goals and objective, the refinement of the identified socio-technical requirements, and the specific functions and features currently implemented in the platform, or are expected to be added to the platform. The approach to collecting this information is through interactive workshops, or focus groups. The feedback collected from these sessions are analyzed in accordance with the evaluation framework presented in Deliverable 2.1 and refined with the relevant elements of the earlier introduced coordination theory. We breakdown the analysis of the data/results received from the evaluation workshops according to the following categories.

4.1

The COMRADES ambition and objective

Regarding the COMRADES ambition and objective, based on deliverable 2.1, we are looking to validate three major components. Relevance of the project We examine the need and perceived added value of the COMRADES project. This is in part done by validating the rationale behind the COMRADES project as described above in the evaluation session setup. In addition to the COMRADES specifics, we also examine the project while considering other ongoing projects, initiatives, and developments. For this we examine how the COMRADES project is different from these other ongoing activities, and the unique selling point and/or added value. Furthermore, we examine how COMRADES supports and contributes to these activities and the humanitarian agenda in general. Resilience definition Next, we focus on the COMRADES specific definition of resilience, which is at the core of the project. Central to this definition is the community as empowered decision makers through improved information management capacities, delivered by the COMRADES platform. Specific attention is given to resilience as the process that supports in continuous resilience building. Approach To enable empowered decision makers, the access to information is a critical aspect. COMRADES aims to provide relevant, timely and accurate information for the different information needs that key decision makers within the community may have. It also aims to support the connection between the community and the (external) responders to community needs. This connection entails the community providing relevant information to these responders for example through providing reports and local knowledge. On the other hand, the expertise and information that the responders have can be leveraged to support the community. For example, through the information exchange an improved, more timely and accurate situational awareness arises, enabling more accurate decisions.

4.2

Approach & Requirements Analysis

The next section of the evaluation session focusses on the requirements as outlined in deliverable 2.2. Having previously established (verified) the COMRADES definition of resilience, highlighting the importance of information exchange between different stakeholders, we examine what requirements should be considered and conditions met to ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

21 | 51

D2.3 Socio-Technical Community Requirements successfully establish this information exchange. Based on D.2.2 we specifically look at: Empowered decision making COMRADES aims to provide communities with relevant information that enables them to (continuously) build and improve their community resilience through an improved situational awareness. However, to improve community resilience, actions need to be undertaken; a better understanding in itself does not improve the resilience. Therefore, this understanding needs to inform actions. For this it is important to consider local governance structures, community organization, and specific local contexts. In this evaluation, we explore how COMRADES transforms a common situational understanding into actionable information. Adoption & integration COMRADES is centered around the communities, empowering them as key decision makers through improved information access. To this end, communities need to adopt both the ideas (rationale) and the platform. A key success factor of COMRADES therefore is the adoption of the COMRADES results and the integration of it specific communities. In the evaluation session, we explore considerations for successful adoption in more detail. In particular, we examine how a sustained adoption of the platform can be accomplished through improved design of the platform and its functions / features as well as socio-technical considerations. Feedback loops & Reciprocity A key element, identified in the workshops in Nepal and expanded upon in D2.2 are the feedback loops. The platform needs to provide a benefit to the users to ensure its continued use. This added value is not only a key element for the adoption, but the value creation from the provided information on the platform is a core function. Using and contributing to the platform should not only benefit the other communities or stakeholders, but also provide a benefit to the user supplying the information in the first place. This reciprocity could either be direct (own improved situational understanding) or indirect (through improved community resilience) but either way the role of the platform should be evident. This creates also a positive feedback loop ensuring the platform is used more frequently and remains of value in the long run. These loops are examined in detail in the evaluations, specifically, how they can be established for individual users and the community.

4.3

The COMRADES platform & development roadmap

A key element in the evaluation is the demonstration and interactive session with the (prototype) COMRADES platform. This interactive session with the platform serves to illustrate the COMRADES objective, the requirements and how these are implemented in an actual platform. Moreover, it serves to verify the ongoing developments and examine how the provided services, functions and features contribute to the overall COMRADES objective and what elements could be further added or developed to further realize the projectâ&#x20AC;&#x2122;s ambition. In this demo session, we specifically asses: Ushahidi Ushahidi provides the base for the COMRADES platform. The Ushahidi platform is a platform for crowdsourcing information, triaging and cataloging the received information, and providing overviews in different formats. Ushahidi has recently released version 3 of their platform, providing even more features and options for deployers of the platform to configure the platform to their needs. In this evaluation, we demonstrate the platform and its ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

22 | 51

D2.3 Socio-Technical Community Requirements new features and examine how the new Ushahidi platform and its functions and features support the COMRADES objective and match with the socio-technical requirements. YODIE / CREES service On this new platform, new services are provided to support users in further examining and cataloging the information. YODIE provides annotated information to the report submitted by users. This for example allows users to click on a building name to find out more about the building through an online database. CREES supports the automatic categorization of incoming reports, reducing the burden on moderators and administrators to review them. While these are only 2 early versions of the services that could be part of the COMRADES platform, they demonstrate how COMRADES aims to support communities. In our evaluation, we examine how these services support users in their information management and improve their situational understanding while at the same reducing the learning curve. Overall functions & user experience In addition, we look at the overall feedback from the users of the platform. In the evaluation, we not only review the current status of the platform, but also ask the evaluation participants to provide additional ideas of functions that should be implemented to achieve the COMRADES objective, to support improved situational understanding and to facilitate information exchange between different stakeholders and actors. We also look at the user interface and experience, examining what improvements could be made to improve the adaptability and integration of the platform into communities and organizations. These could be either technical improvements in the platform itself, or in its deployment (training, support, etc.).

4.4

Data collection

The participants in the evaluation workshops have been provided with a copy of the COMRADES consent form along with the invite for the evaluation session. This consent form is similar to the consent forms that have been used for the interviews in the earlier stages of WP2. However, rather than collecting signed copies from each participant, we have asked verbal consent from the participants at the start of the evaluation workshop. The reason for this is two-fold. First, due to the dynamic agenda of some of the participants (as most of them are professionally involved in crisis response), the composition of the evaluation group could vary at the last minute. We have provided the flexibility for others to join at the last minute, without a written consent form. Additionally, for one of the evaluation sessions the native language was English (rather Chinese). To ensure that the participants well understood the nature and conditions of participation in the evaluation workshop, we have introduced the evaluation workshop, the role of the participants, the data collection, and other aspects of the consent form verbally with the support of translators. This also allowed us to clarify and address any points of concern. Finally, the desire was expressed by some of the participants not to have a written record of their participation, citing they would feel more at ease to express their ideas and input. In all cases of the evaluation, verbal consent has been asked of each of the participants and recorded. The data from the workshops have been collected in different ways. First, the sessions have been recorded. Either on camera (in-person workshops) or using recording software of the video-conferencing application used. In accordance with the ethical policy of the COMRADES and as agreed with the participants, these recordings only serve to support the note-taking ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

23 | 51

D2.3 Socio-Technical Community Requirements processes allowing the facilitator to focus on guiding the discussion. Secondly, notes have been taken during the workshops by the facilitator(s) as well as by the translators/researchers present. These notes form the main source of the evaluation data. Finally, participants could provide additional insights, information, and documentation during or after the evaluation.

5 Evaluation participants The COMRADES evaluation of the intermediate platform is focused on gathering initial and early feedback on the development of the platform, as described in the section “Scenario 1 (M18)” of deliverable 2.1. Considering the early stages of the COMRADES platform, specific requirements for participants of the evaluation sessions apply. Next, participants have been invited and the workshops were planned. Participants took part either in person or online, depending on the location and availability of the participants. Selection criteria Given the early stage of the platform, with a few services implemented, the participants for the evaluation workshops need to be carefully selected. As mentioned earlier, the platform in its current form serves to illustrate the COMRADES objectives and ideas and how these can be translated into a practical platform. This current implementation however does not include all the desired features, and some of the key functionalities may not be present at all. Moreover, as the platform is still under development and internal testing has just started, users may still encounter bugs in the system. It is therefore important that participants are able to ‘look beyond’ these technical shortcomings and understand the potentials and intentions behind the current and planned features. Although this is addressed in the workshop set up, by taking the participants through the COMRADES project following the approach outlined above, it still requires participants to see the current implementation as one possible translation of these objectives into a platform, as well as to reflect on abstract level on the platform. Additionally, as listed in Deliverable 2.1, we are looking to (in large part) focus this evaluation on one of the COMRADES communities: the responders and Volunteer & Technical community. There are multiple reasons for this. First, the previous community engagement activities and workshops of COMRADES focused mainly on other COMRADES communities as illustrated in Figure 6. The initial interviews, conducted as part of task 2.1, have examined the perspective of community leaders and activists. People who have deployed the platform in their community. In these interviews, we have also touched upon the technical support (V&TC) that is supporting these deployers of the Ushahidi platform. Next, in task 2.2, we have mainly engaged with the affected community and the related reporters / information providers as well as affiliated responders. Although these engagements have also touched upon the other COMRADES communities, the main focus has been on the reporters/deployers and the affected community.

24 | 51

D2.3 Socio-Technical Community Requirements

Figure 8 User communities and COMRADES WP2 engagement

As this evaluation serves to both validate the work thus far, and solicit additional input, the evaluation focusses on the groups that have not been at the center of the COMRADES engagement activities. Involving these other groups helps to validate the input received from the other input gathering activities. At the same time, it also provides valuable new insights from the perspective of other COMRADES communities than those analyzed thus far. Both the responders and the volunteer and technical communities also have extensive experience with both the coordination between different actors (mainly the professional responders) as well as facilitating the information exchange between different groups (the volunteer & technical communities). In accordance with the evaluation setup and the analytical framework (build on Coordination Theory) these groups are well suited to support the evaluation and provide valuable feedback, specifically on the mentioned topics at this stage in this project. Finally, these groups have also worked with different systems and innovations over the past years, and are able to relate the presented material to other ongoing developments. Their expertise also allows them to look beyond the current ‘readiness’ of the platform and discuss the potential of the platform. Invitation Following the selection criteria mentioned above, and the resulting focus on the professionals and volunteer & technical communities, the network of the COMRADES partners has been used to identify potential participants. From the personal networks of the COMRADES partners, a list of potential participants has been identified. We have mainly approached two networks: ●

●

Members of the Digital Humanitarian Network (DHN), an international network of volunteers that support humanitarian organizations with online, remote, and digital services such as mapping, satellite image analysis or social media tracking. As several project members have close contacts with members of these networks, or are even a member, we have invited individuals from different organization within the DHN to join the evaluation session. The evaluation time-frame coincides with a visit of one the partners to China as part of a summer school and exchange program. During this visit of several weeks, the partner visited several key organizations and government agencies in China

25 | 51

D2.3 Socio-Technical Community Requirements responsible for disaster risk reduction and response, including the Chinese Institute of Disaster Prevention and the Beijing Earthquake administration. This enabled us to not only involve professional responders in the evaluation session but also expand the COMRADES project scope to include input/perspective from China. An invite (see A2) has been sent to the contacts within the organizations. Following the replies, several have been invited to attend workshops, organized both online (in the form of an online seminar) and offline (in person). From the 40 invitations send, around 35 people replied positively. However, due to some scheduling conflicts, not all were able to attend. Nevertheless, some people attended sessions that they were not directly invited to, but were brought on by other session invitees. Planning Based on the response received to the invitation, the evaluation sessions have been planned to accommodate the wishes of the potential participants as much as possible. This has resulted in two evaluation session. One of the sessions is conducted online, using videoconferencing to enable participants to join from different geographical locations. In this instance, specific attention has been given to the timing of the workshop, although most of the participants in the online evaluations session are located in the European or British time zone. The other session has been planned in conjunction with organization (IDP, see below)

Part II: Evaluation sessions 6

Workshops

For the evaluation, two types of workshops have been conducted. One set of workshops took place online using online seminar software. Attending these workshops were mainly members of the Digital humanitarian network. Furthermore, in-person workshops have been organized, specifically at the Chinese Institute of Disaster Prevention in Beijing China. These workshops focused on the Volunteer & Technical communities (online with the digital humanitarian network) and responding professional (in-person at the Institute of disaster prevention in China). These workshops are further supplemented by internal evaluations and self-reporting.

6.1

Online workshops

Digital humanitarian network and the Digital Collaborative Spaces are digital volunteers that support the work of (international) humanitarian organizations through digital channels and tools. The purpose of the Digital Humanitarian Network (DHN)2 is to leverage digital volunteers in support of 21st century humanitarian response. More specifically, the aim of this network of networks is to form a consortium of Volunteer & Technical Communities (V&TCs) and to provide an interface between formal, professional humanitarian organizations and informal yet skilled-and-agile volunteer & technical networks. As frequent user of technology, the members of the digital humanitarian network are familiar with the importance and potential of information technology to support humanitarian operations. 2

http://digitalhumanitarians.com/about

26 | 51

D2.3 Socio-Technical Community Requirements

Figure 9 Digital humanitarian network structure

In recent years these volunteer groups have begun to also explore the application of their tools, technologies and expertise to support affected communities directly by providing them with crucial, timely and relevant information, acting as an information clearing house between the international humanitarian response and the affected communities. In addition, they are supporting initiatives to help communities become more data-aware, for example by helping them map their community on OpenStreetMap, or build data-sets that might be relevant to responders when a disaster strikes. Workshop setup & attendees The participants of the evaluation sessions, just like the volunteer & technical communities themselves, have a diverse background. Among the participants were people with a highly technical background such as software developers, people who work (or worked) for humanitarian organizations such as the International Federation of the Red Cross, and students and academics who have been involved in past deployments and study the role of technology in disasters. All the participants have been active in at least 2 deployments, supporting international organizations, for example after the 2015 Nepal Earthquake, Typhoon Haiyan in the Philippines in 2013 or the Ebola outbreak in West Africa. The workshop was conducted online, following the evaluation setup as described above. As some participants were familiar with the Ushahidi platform this helped to move quicker towards the new COMRADES features and services added to the platform. Workshop specific focus This workshop specifically builds on the experience and expertise of the volunteer and technical communities. This experience and expertise, consists on the one hand of the ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

27 | 51

D2.3 Socio-Technical Community Requirements knowledge of current and developing technology that is employed by humanitarian organizations. On the other hand it also encompasses the knowledge of how these technologies can be leveraged to support humanitarian organizations, more specifically what are critical success factors in the successful deployment and adoption of information and communication technologies in humanitarian emergencies.

6.2

In-person workshop

China Institute for Disaster Prevention The Institute of Disaster Prevention (IDP)3 was established by State Seismological Bureau (later China Seismological Bureau) in 1975. In 1985, it was promoted as the College of Seismology Techniques, and renamed College of Disaster Prevention Technique by the former State Education Committee in 1992. It was promoted as Institute of Disaster Prevention Science and Technology in 2006 and shortened to IDP in 2009. Under the administration of China Earthquake Administrationâ&#x20AC;&#x2022; (CEA, former China Seismological Bureau), IDP is now the only higher educational institute of disaster prevention and reduction in China. At present, IDP consists of 9 departments including 23 majors, like Earthquake Sciences, Disaster Prevention Engineering, Disaster Prevention Equipment, Disaster Information Engineering, IT & Geophysics, Urban Rescue Technology, Engineering Vibration and Exploring, Building Engineering, Construction Cost Prediction, Telecommunication Technology, Enterprise Resource Planning (ERP) and so on, covering the fields of science, engineering, economy, management, and arts. Workshop setup & attendees The participants included a range of staff members and students from the institute who have not only studied the subject of disaster prevention and response, but are also active as responders themselves in various emergency services and/or government agencies. Attendees included representatives from the Beijing Earthquake Administration, the Chinese Search & Rescue team and firefighters. In total 11 participants attended the +/- 3 hour workshop (the session was extended due to the extra time required for translation). The workshop followed the approach introduced in section 3. Workshop specific focus In this workshop, specific attention is given to the fact that all of the attendees have some professional experience in crisis response and/or disaster management. This experience ranges from firefighters to people involved in governance role. The perspective of these professionals from the official capacity has been a specific focus of this workshop. Furthermore, the structure and government unique to China provides an interesting case to the COMRADES project and that has been explored more in depth as well during the evaluation workshop.

http://www.fzxy.edu.cn/col/col346/

28 | 51

D2.3 Socio-Technical Community Requirements

7 Internal evaluation In addition to the evaluations involving external stakeholders, the midterm evaluation has been expanded to include internal evaluations of the COMRADES project. Specifically, evaluations on the development of the technical components (services) of the COMRADES project. These internal evaluations are based on self-reporting by the developers of these services and serve a dual purpose. First the modules help to illustrate how the COMRADES objectives are translated into ‘tangible’ results, as services provided on a platform. In addition to the internal evaluation of the performance of these services, we also examine how these services align and contribute to the overall COMRADES objectives. Specifically, we are looking into the following two topics in this internal evaluation:

Link to COMRADES objective

First the self-reporting help to determine the level of alignment between the ongoing (internal) developments, and the perceptions of or needs identified from people outside the project. Internally, identifying the ‘contribution’ of these services to the overall COMRADES objectives shows the anticipated or intended link between the service and the improved information management capabilities of the community. This internal reasoning, perception or intentions can then be compared to the feedback of the people participation in the evaluation workshops to see if they see the same contribution of these services to the overall COMRADES project and ambition.

Performance

Next, the internal assessment of these services will determine the performance of these internal developments, and how they perform given certain criteria. These criteria do not only assess the performance of the service in term of speed but also asses the accuracy of the service to identify the correct and relevant elements in the provided reports. These performance evaluations show how reliable and depend the provided services are, and are indicators of how well they perform their automated functions, thus relying less on the expertise for data management from the deployers of the platform, making it more universally useable to communities.

29 | 51

D2.3 Socio-Technical Community Requirements

Part III: Results 8

Results from Technical Evaluations

In addition to the evaluation workshops organized to verify the developments of the COMRADES project, and to solicit additional input for further developments, internal evaluations have also taken place in the past period. These internal evaluations focus on the performance of the provided services and platform, and are performed by different partners involved in the technical developments. The details of these evaluations, focusing on the technical parts of the COMRADES project are listed in the respective deliverables of WP3,4 and 5. However, we have included a summary of the results to (1) provide a comprehensive overview of the current state of the project and (2) compare the internal results with the externally received input. This will help to assess the alignment between the overall project objectives and the ongoing technical developments.

8.1

YODIE-service

YODIE is a tool for finding entities and concepts in text, and resolving them to something unambiguous. It does this by linking terms to individual DBpedia entries, based on Wikipedia, each of which corresponds to a very specific concept, thus giving the precise interpretation required in crisis response. Harnessing social media for crisis response requires precise interpretation of the postsâ&#x20AC;&#x2122; textual content. One important part of processing text is to detect the locations, organizations, people, concepts and so on in the text. In crises, this gives the stakeholders information about needs and resources. However, language is ambiguous, and communication is often heavily context-dependent, both in terms of the spatial and temporal context of an emerging event, and in terms of the linguistic context (i.e. the other words in the sentence). Ambiguity can be around the specific usage of a word -- i.e. when we see the word Paris, is this Paris France, Paris TX, or one of the other Parises? -- or the general meaning of a phrase: in "The tsunami hit hard in Banda Aceh", what kind of words are "Banda" and "Aceh"? Social media texts tend to be terse, especially texts produced under time pressure. This reduces the amount of data available for analysis, which places a real burden on systems that rely on surrounding words in order to produce results. Traditionally, disambiguating the meanings of words and connecting words to concepts (or "entity linking") have used the other words surrounding the target concept to work out what it means. Following the above example, if we're trying to resolve which Paris is intended, it is easier when there is a whole paragraph of context, which contain helpful terms like Versailles. However, this broader context is generally absent in social media and in terse texts. This makes it very hard to determine which concepts are intended. YODIE specifically aims to address this tough entity linking problem, in order to precisely interpret data in crisis-hit areas. Additionally, this kind of event typically involves places and organizations that have not featured in the mainstream news much, and so are harder to resolve, because there is less prior data on them. In COMRADES, text should be processed in real-time (or close to it), ruling out retrospective analyses where the data might make more sense after the event has unfolded. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

30 | 51

D2.3 Socio-Technical Community Requirements To sum up, the three challenges addressed by YODIE: 1. Entity linking in short and noisy text 2. Finding unusual, uncommon entities 3. Detecting emerging entities while events are still unfolding Performance measurement Scoring is based on a benchmark dataset, created for the purpose of measuring entity linking performance in this non-standard text type, social media. This was annotated over tweets. The resulting corpus was 794 documents comprising roughly 11K words. This contained 681 entity mentions. The data was divided into test and training datasets of equal number of documents, with the evaluation partition having 424 of the entities and the training set having the other 257. This division was kept evaluation set-heavy in order to provide a more useful evaluation resolution. The metrics used for evaluation are precision, recall and F1. Precision measures how noisy the output it, with 100% score meaning that every output link was correct and none were spurious. Recall 100% means that all the entities in the documents were found, and none were missed. F1 is the harmonic mean of these two. To evaluate YODIE as a framework, we compare performance with other available state-of-the-art entity extraction approaches. Results are reported over the widely used "Test B" part of the Aida/CoNLL corpus. Table 1: Performance result and comparison

System

Precision

Recall

YODIE (base) YODIE (expanded) Aida 2014 Lupedia Spotlight TagMe TextRazor Zemanta

0.44 0.50 0.59 0.50 0.09 0.10 0.19 0.48

0.55 0.62 0.38 0.24 0.51 0.67 0.44 0.56

0.49 0.55 0.46 0.32 0.15 0.17 0.26 0.52

Note that YODIE performs best overall, but also gets the highest usable Recall score (“usable” because although TagMe tops Recall, its precision of 0.10 means that accurate results cannot be extracted from the noise this tool generates). Recall is important in the case of emerging entities; it is the hardest factor to achieve in social media entity extraction, and being able to pick out new location names - like “Rigopiano” in the case of the 2017 earthquake - is very important in crises. This is indeed the focus of our Emerging Entities shared task (WP3). Following are evaluations over German and French. We did not have bases for comparison here due to the paucity of open multi-lingual entity linking systems which can operate in the required conditions. Table 2: Performance in other languages

System

Precision

Recall

YODIE French YODIE German

0.75 0.56

0.87 0.52

0.80 0.54

31 | 51

D2.3 Socio-Technical Community Requirements

8.2

CREES

During crises, a very large number, sometimes in the millions, of messages are often posted on various social media platforms by using the hashtags dedicated to the crises at hand. However, a good percentage of those messages are irrelevant or uninformative. In order to support communities in extracting the relevant information to support their situational awareness building in a crisis, they need to be able to quickly identify the relevant messages. CRESS (Crisis Event Extraction Service) is a web service designed for extracting multiple eventrelated information from short social media documents (i.e., tweets). CRESS provides three different services: 1) Identifies if a document is related to a crisis (i.e., related, non-related); 2) Identifies the type of event discussed in a document (i.e., shooting, explosion, building collapse, fires, floods, meteorite fall, haze, bombing, typhoon, crash, earthquake and derailment.), and; 3) Identifies the type of crisis-related information present in a document (i.e., affected individuals, infrastructures and utilities, donations and volunteer, caution and advice, sympathy and emotional support, useful information, other.). Given a textual document, the service returns a label that can be used for helping the curation of social media messages and the identification of key events and messages more efficiently. For example for the following message: “Typhoon Haiyan: thousands dead as devastation hampers aid efforts http://t.co/y86LeTeQ3J”, the CRESS service can returns the following labels: 1) related; 2) typhoon, and; 3) affected individuals. Although several approaches exist for detecting general and global events from social media (floods, wildfires, bombings, etc.), the automatic identification of fine-grained emergencyrelated information (e.g., affected individuals, infrastructure, etc.) is still in its infancy. Current approaches for event identification from social media data make use of supervised and unsupervised Machine Learning (ML) methods, such as classifiers, clustering and language models. More recently, deep learning has emerged as a promising ML technique able to capture high level abstractions in the data, providing significant improvement for various tasks over more traditional ML methods, such as text classification, machine translation or sentiment analysis. However, deep learning hasn’t been applied to the problem of finegrained information detection in crisis situations. Current approaches are also limited by the lack of semantics when dealing with short documents. One of the inherent issues of small documents is the amount of contextual information they carry. In the context of fine-grained event identification, semantics can be used for adding relevant contextual information that may help document classification (e.g., locations, organizations, etc.). Rather than using traditional classification approaches that rely on bag of words features and classification algorithms, we propose to use deep learning by adding a layer of semantics to traditional Convolutional Neural Network (CNN) models to capture the contextual information that are generally scarce in short, ill-formed social media messages. The new model is called Dual-CNN and uses two parallel CNN layers to embedded words and semantics into a classification model.

32 | 51

D2.3 Socio-Technical Community Requirements Different version of CRESS were evaluated using data provided by the CrisisLex T26 dataset (http://crisislex.org/data-collections.html) that contains ~26K annotated tweets posted during 26 crisis events in 2012 and 2013. The models were trained and evaluated on the CrisisLex T26 annotations and evaluated using 5-folds cross validation. The accuracy of the CNN and Dual-CNN models was compared with Naive-Bayes, CART and SVM using Precision (P), Recall and the F1-measure (F1). In general, the results showed that the CNN approach outperforms classical methods with improvement of up to +3.7% for the identification of crisis-related documents. CNN was able to identify crisis-related documents with F1 = 83.8%, event-types with F1 = 98.3% and emergency-related information with F1 = 61%. Dual-CNN seems to not improve much in accuracy compared to the standard CNN model with very similar F1 values (83.3% / 98.5% / 61.3%). Since the accuracy improvement of the Dual-CNN is low compared to standard CNN, we deployed the CNN models to CRESS as they do not require any semantic extraction step and are therefore more robust.

33 | 51

D2.3 Socio-Technical Community Requirements

Results from Workshops

The workshops that have been conducted have provided valuable insights for the COMRADES project. They have helped to validate and further refine the COMRADES objective, while also placing some critical remarks for consideration for the project to move forward. However, it should be recognized that the evaluation workshops involved a diverse set of participants, each with their own expertise, background and area of interest. Although the workshops have been structured in a specific manner to introduce participants to the COMRADES ideas and objectives, and the discussions have been guided by key questions derived from the work thus far in the project, it should be recognized that in these early prototype stage of development, there is ample room for interpretation.

9.1

Online workshop with digital humanitarian network

The online workshop was conducted early June 2017 and involved 5 people from the Volunteer & Technical communities (V&TC). The specific focus of the workshop was on the role of the volunteer and technical communities during a crisis event These communities support professional responders (mainly international humanitarian agencies) through modern information and communication technologies. In recent years these V&TCs are shifting their attention to supporting affected communities directly. Not only in the response of a disaster but also in terms of data-preparedness. These topics align closely with the focus and scope of the COMRADES project. During the session, participants were asked to reflect on the presented platform from their past experiences and the potential benefits it could add to their work. During the workshop, several real-world examples and cases have been mentioned to illustrate. These specific examples are omitted in this report for privacy reason (of both the participants of the workshop, and the organizations involved in the examples). It should be noted that the background and experience of the participants in the call mainly relates to large-scale, international, humanitarian operations, and in particular in response to disasters. Part 1: COMRADES objective The participants of the workshop recognized the various parts of the COMRADES resilience definition, although they have not directly linked the information access to improved community resilience before, in particular as part of the resilience ‘as a process’-lens. After the introduction, the participants confirmed that the approach and focus of the COMRADES project align closely with other ongoing initiatives that aim to close the gaps between the different communities. Ad-hoc initiatives in certain cases are being formed that aim to connect to community closer to the (international) professional responder.

Community as key stakeholder

The coordinated approach, creating alignment between the affected communities and international responders is, according to the participants, an increasingly important element in disaster response. International responders are aiming to move away from ‘a blanket approach’; providing large and diverse set of relief items and capacities. Rather, organizations aim to provide more targeted relief. In order to enable this more precise intervention, improved information about the communities is needed. COMRADES aim to provide this more detailed information. In addition, organizations are employing cash-for-work programs as a replacement for providing direct aid. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

34 | 51

D2.3 Socio-Technical Community Requirements This stimulates the local economy and provides people with both resources and income that in turn will support other affected members, in addition to reducing logistical requirements. These cash-for-work programs require a strong community engagement as they need to (co)design the projects and work that will be funded based on their needs. An improved situational understanding will help communities in proposing high potential projects related to their (continuous) resilience building objectives.

Local capacity

The participants of the workshop also emphasized the importance of a changing information management focus. The current focus of the (remote) information management tasks and activities deployed after a disaster are related to so-called ‘needs assessments’; determining the needs of the local communities. These needs can be immediate rescue and relief operations, for example providing medicine, food, and temporary shelter. In addition, the needs assessments can also focus on more longer-term recovery needs. These are often established by comparing baseline information (normative state) to the current state. However, these assessments rarely include the local capacity, the resources, assets, and capacities that the community could contribute themselves. By including these in the assessments as well, it avoids duplication of efforts and even potentially disrupting local (labor) markets. The more comprehensive focus on the community of COMRADES addresses this opportunity: it enables the community to map not only their risks but also identify their capacities and how they can most effectively be utilized.

Longer term focus

Linking relief to development (LRRD) is a key strategic area for humanitarian and development agencies. Each of these areas have their own organizations. And even if an organization is operating in both development and humanitarian aid, they will likely have different departments working on each. Furthermore the ‘attention’ given to a specific crisis (and related the funding and resources available) subsides relatively quickly (weeks). However, the recovery and reconstruction operations take a longer time (years) as illustrated in the diagram below. A critical component to ensure continuous resilience building is the handover between each of these stages. According to the participants, due staff changes within, and especially between the stages, retaining knowledge and an effective handover are challenging. However, the community is continuously present and could facilitate these handover moments and retain the relevant knowledge.

35 | 51

D2.3 Socio-Technical Community Requirements Part 2: Requirements In the next part of the workshop, the participants discussed the requirements for an effective coordinated approach. Building on the discussions from the first section of the workshop, they illustrated the importance of adapting to local contexts.

‘Translation’

A key concern expressed by the workshop participants is the translation between the different actors. This does not only refer to information exchange in different natural languages, but more over to the different terminology used by different stakeholders. International humanitarian organization use, for example, many specific abbreviations that refer to key aspects of their operations. On the other hand, local communities will likely use different terminology to refer to their needs, even if they refer to the same thing. At the same time, organizations might aggregate information differently, for example grouping WASH (Water, Sanitation, Hygiene) operations together while the local community refers to only element by another name (‘comfort room (CR)’ for example). Being able to map between different languages, those of professionals and of the community is a key element that the Volunteer & Technical community provide at the moment and is something that should be considered when connecting stakeholders ‘directly’.

Adoption by international organizations

Furthermore, adoption and integration is mentioned as a key element in ensuring that a system is put into use. It will depend on the target user what specific functions and features are needed to ensure a successful adoption of the system by either the community or the organization. Workshops participants emphasized the importance to examine what the specific added value is or could be for one of the actors in the system. Next, the system, platform or deployment should be able to be configured to provide this added value. This implies that the COMRADES should anticipate what (potentially) could this added value be, so the system can be prepared for it. Participants did provide a specific suggestion: the increased need for data to improve decision making within the international organizations. Being able to extract and share data-sets that can be used by international organizations to plan operations or request funding would provide an added value. Especially if linked with existing platforms for exchanging information and data such as Reliefweb.int and the Humanitarian Data Exchange.

Preparedness

Volunteer & Technical communities are often deployed (‘activated’) when a critical event occurs by one or more of the international humanitarian organizations. In this short time frame, there is limited to no time to design, test and train with new systems. The workshop participants therefore highlighted the importance of having certain systems, procedures, organizational structures, and support in place that can be quickly deployed by the V&TC when called upon. However, since every deployment has certain unique aspects they need to be able to quickly adapt these elements to the specific situation. They have an established ‘core’ of systems, procedures, etc., however this setup with a certain bandwidth and flexibility in mind. In addition to their own internal procedures, the participants also highlighted the importance of established contacts with organizations pre-deployment: an improved mutual understanding of they can do and the organization may desire, reduces lead-times between activation and the first valuable results. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

36 | 51

D2.3 Socio-Technical Community Requirements Participants also highlighted that the use of platforms and system by the volunteer & technical community (including Ushahidi) is largely depended on advocacy. This advocacy comes in part from successful cases and deployments in the field which are then highlighted by both development or humanitarian organizations but also through media outlets and websites, or personal blogs. In addition, so called ambassadors or ‘evangelists’ further support the disseminate the work of the V&TC. These people are highly involved and key advisors to the V&TC and are part of international organizations, universities, tech companies, etc. Especially for COMRADES, with the community as one of the central adopters, it is recommended to build a local community of ‘evangelists’ or ambassadors, who advocate its use to the different users. Part 3: Prototype platform The demonstration of the platform was of interest to the volunteer & technical community. As a community that works mainly with information and communication technologies to support humanitarian organizations, the platform is a potential new tool that could be added to their ‘toolbox’. In addition, these communities are already familiar with the (previous versions of) Ushahidi platform, and are excited to see new functions added. In addition to welcoming the new addition that makes it easier for them to work with large volumes of data and to narrow down quickly to specific reports with relevant information, the V&TC added some remarks on the platform in general.

Compatibility with existing platforms

There are many ongoing developments that are related to COMRADES. Platforms that can either feed COMRADES with information and data, such as baseline data from the Humanitarian Data Exchange, or serve as a communication (dissemination) channel, such as Reliefweb.int or the channels of the national government. More specifically, the V&TC group emphasized the need for flexibility to use a mix of tools available to them that they can combine a way that fits the need of their specific deployment. Some referred to the so called ‘Service Oriented Architecture’ or ‘Micro-services’ concept, of separate functions that could be programmatically used by other applications.

Last & First mile communication

The workshop participants also confirmed many of the findings of Deliverable 2.2 (Nepal Workshops). In particular, the importance of the so-called first and last mile communication. Many of the developments in information and data management technology seem to assume that all the data is digitally available in the platform (or some other accessible data sources). However most of the information that is available is not relevant. Relevant information, especially at the community level, can also be conveyed through direct person-to-person communication, via text messaging or other means. This so-called first mile communication, getting the information into the platform, is critical to ensure that the relevant data is in the platform. This could be accomplished through motivating people to submit information on the platform (incentives) but also have people work as a proxy for people with no access to digital communication channels. Finally, the sharing and dissemination of the information can also happen through non-digital channels. The V&TC, in line with the findings from the Nepal workshop, emphasized the importance of FM radio to communicate critical information. Working with existing communication channels for both collecting and sharing the information is key to both filling the platform with sufficient data as well as sharing to people who it might be relevant for. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

37 | 51

D2.3 Socio-Technical Community Requirements

9.2

In-person workshop IDP, China

The workshop conducted at the Chinese Institute for Disaster Prevention China on June 14th, 2017 provided valuable insights for the COMRADES project. The 11 participants provided several key considerations for each part of the evaluation session. It should be noted that the responses of the participants were in part English and part Chinese and translated by fellow researchers of Delft University of Technology and the Chinese Earthquake Administration. It should also be noted that the governance and culture differences between China and other countries have an important influence on the results. Nevertheless, considerations specific to the Chinese context could hold true to more or lesser extend for other countries, and should be considered in the project.

Figure 11 Workshop at Chinese Institute for Disaster Prevention

Part 1: COMRADES objective The participants of the workshop underlined the importance of community resilience. Most notably they emphasized the importance of communities to be ready for disaster and provide life-saving aid to another. Using multiple examples of past disaster throughout China, the participants illustrated the importance of disaster preparedness and how it contributed to a reduced number of casualties. However, the resilience in the regard this preparedness is still part of the â&#x20AC;&#x2DC;stateâ&#x20AC;&#x2122; perception and not the continuous process of resilience improvement. In other words, there is a strong focus on the resilience as it relates to assets (build environment, the response capacity etc.). This is further exemplified by the strict enforcement of the buildings codes by the Chinese government. Information needs Participants of the workshop underlined the importance of information for the resilience of the community. The participants particularly emphasized the importance of situational awareness during a crisis. This situational awareness breaks down in two components. First there is personal situational awareness, which is most crucial during the early stages of an unfolding crisis. This situational awareness encompasses the need to for information that relates to a personal situation and those of friend relatives. Participants mentioned that at this initial stage people are searching for information about their relatives and friends, such as their location and well-being. Next the focus shifts determine what/when help is arriving, getting an understanding of what aid is expected and how long the affected people need to be self-reliant. Then, combining this information with their personal situation, people are aiming to understand which place provides safety / shelter. This will help people to plan their evacuation routes and find safety. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

38 | 51

D2.3 Socio-Technical Community Requirements Most importantly they mentioned that -in general- this situational awareness needs to be established first before people are able to ‘move on’ to consider the more broader community perspective. They would first need to know the well-being of people they care about, their own situation in relation to the crisis and the aid being send. Providing support Following the establishment of the own situational understanding, participants indicated the focus shifts to the community perspective. Relating to the community based resilience, the participants described various examples of communities supporting each to other. Ranging from rescue operations, providing relief items (food, shelter) to donating support. Notably, the participants mentioned the important role of NGOs such as the Red Cross to facilitate this community driven response and relief provisioning. The participants overall recognized the importance of community driven resilience. Their main focus is on the current risk mitigation measures and focused on the resilience of community assets (specifically buildings). However, in the response stage they recognized the role of the community as key stakeholder. Especially for providing local knowledge about the terrain, landscape, operational circumstances. A specific example of an earthquake occurring in Tibet, in which the local monks worked closely with the responders to support and direct the rescue operations and relief efforts, illustrated the importance of integrating this local knowledge for an effective response and rescue operations. Part 2: Requirements Focusing more on the requirements for community driven resilience building, the conversation focused mainly on the interaction between the different stakeholders, specifically the government and the local communities. Furthermore, the participants highlighted the importance of recognizing the large differences between cities and more rural villages.

Governance

In China, community leaders and facilitators are selected to monitor and signal needs of the community members, most notably the more vulnerable groups in a community such as young kids or elderly people. These community facilitators are often older people who are well known in the community. In their work, they rely a lot on their personal network and connections and own experience to ‘keep an ear to the ground’ on what is happening in their community. These people also work closely with the government, and are a sort of liaison officer from the community to the (local) government. In addition, in China, the government plays a strong role in the planning and resilience building process. China has comprehensive risk reduction regulation and enforcement policy. Therefore, this a strong focus on risk mitigation, and moreover, reliance on this ‘external’ expertise. People and communities rely on the expertise of advisors that create the regulations and implement them. In a sense, there is a high trust in the government when it comes to both the risk reduction and response. Resilience building is therefore seen as (also) as task of the government. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

39 | 51

D2.3 Socio-Technical Community Requirements Different contexts The scale and diversity of China is also a main point that influences the specific socio-technical requirements. Communities in China range from mega-cities with millions of inhabitants to small rural and remote villages. In addition, China has very diverse landscape, local circumstances, cultural influences. These wide range of context requires flexibility, not only from a technical point of view in terms of platform functions and features, or the services being able to handle input originating from different contexts, but also how the system would be embedded in the local communities. Villages will have a stronger social cohesion that the system can built on, in contrast to cities with more resources but less social cohesion. Therefore, leveraging and integrating local knowledge is a key element. Being able to add this local knowledge provides a benefit to responders who then be more able to tailor the response to fit with the local context and operational circumstances. At the same time, this local knowledge would also provide insights for the community themselves, even outside crisis event, to identify where improvements can be made.

Synergy

Finally, a key requirement put forward in the evaluation is the desire to achieve, or -at the very least- avoid duplicated efforts. As responders and government agencies operate quite independently and according to their own structures, there is a risk of duplicating efforts. This could either by duplication of efforts for aid delivery, but for example also duplication in information gathering efforts. This does not only apply to professional responders or stakeholders, but rather all the stakeholders including the community. This also related to what participants called â&#x20AC;&#x2DC;seeing the bigger pictureâ&#x20AC;&#x2122;; or a more comprehensive situational overview. Not only about the efforts going on in their community, or one specific agency, but rather a comprehensive overview of the activities that are being undertaken and needs expressed. This overview would not only prevent the duplication of efforts but also identify key gaps; needs that are not being address by either the responder, the community or other stakeholders. Identifying these gaps enable stakeholders to make a meaningful contribution to the response in a disaster. Part 3: Prototype platform In the final part of the evaluation the platform has been shown the participants. Since the participants have extensive experience in disaster response in China several examples have been used to illustrate how the platform can be used, most notably earthquakes. Furthermore., in the interactive session the platform has been configured for a specific (fictional) disaster (flooding in Beijing) and several posts have been entered (in English) to demonstrate the services currently available. Timing & Service delivery Participants emphasized the importance of timeliness, the ability of the platform to provide the right (relevant) information at the right time. Participants mentioned that this is expected by users of information tools in modern-day China. China has a wide range of digital services available to citizens. The success of these digital services depends on the delivery of timely, accurate and actionable information. Participants indicated that for any platform to be successful these same conditions have to be met. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

40 | 51

D2.3 Socio-Technical Community Requirements Participants also emphasized the importance of integration with existing services and platforms, such as WeChat or AliPay. These platforms are commonly used throughout China and offer access to many daily services, serving as both an application portal, payment system, identity manager and social media platform. To ease adoption COMRADES could also seek to be integrated or available on these platforms.

Planning & Daily use

As illustrated, China invests a lot planning, risk mitigation and preparedness. All these activities take place before a disaster. These also provides an opportunity for COMRADES to be active before a disaster. Exercise are frequently held in China; schools, companies, industries, and other organization hold regular drills to train people on how to respond to a disaster. These are ‘regularly’ recurring activities that provide an opportunity for COMRADES to be used more frequently. Participants emphasized the importance of daily or frequent use of the application. More specifically, they were skeptical that a platform with a narrow focus on crisis response and recovery would be adopted by users. Especially in the chaotic stages following a critical event. Therefore, COMRADES needs to be of added value in more frequently occurring use-cases. Combined with the exercises, COMRADES could be used for collecting (crowdsourcing) information and building a knowledgebase of local circumstances (see above) that could be used during an actual crisis event. Participants added that this would also enable a better integration between the contingency planning (risk reduction approaches) and the exercises. At the moment, these risk reduction mechanisms focus primarily on structural or physical safety, where the exercises focus mainly on the immediate response stages. By combining knowledge gathering and information, these activities could be combined with each other. Practically, the main concerns of the participants were the costs involved with the platform. Not only for the initial purchase of the software or platform but also costs associated with the maintenance. These includes direct costs (servers etc.) but also staff costs to operate and maintain the system (if not done by volunteers). Also, access was mentioned. This relates to the ability to reach the system behind the ‘wall’ (the internet censorship applied by the Chinese government). It also relates to ability to work with the system if connectivity is limited or the possibility of people having the same information who are not able to use the system for example by proxy. In this regard, it is also important to recognize that people should not be excluded or receive a disadvantage from not being able to use the system. Finally, the participants emphasized the importance of support. Despite all digital services in China, there is always a phone number or person they can get touch if they need help. Systems also provide multiple options or alternatives, for example for communication or payment. COMRADES would also have to be able to deal with these different ways of communication and use (see also ‘flexibility’ above). Moreover, participants mentioned that hands-on support, guidance, and facilitation would be needed to setup and maintain a system. A key question is how this support would or could be organized for COMRADES.

41 | 51

D2.3 Socio-Technical Community Requirements

10 Discussions & Conclusion 10.1 Theoretical Findings From the coordination theory literature, requirements were extracted to support and guide the COMRADES platform development. Most of them are turned towards information systems and infrastructures as the various evaluations advocated. According to Coordination theory, the COMRADES platform should support tasks that emerge when engaging with different stakeholders, specifically in a crisis response. First, the promotion of the purpose and missions of a stakeholder. As different stakeholders are engaged each of them bring their own objective. Although all stakeholders share a common overall objective (such as building a resilient community), there may be differences in the way that this should or could be accomplished per stakeholder. Finding a way to support and align these different objectives is a key element that COMRADES aims to support through a shared situational understanding. This is closely related to the objectives selection and activities decomposition (e.g., top-down goal decomposition and bottom-up goal identification) and the prioritization of the objectives. Once the objectives have been established of the different stakeholders and potential gaps and synergies are identified between them, specific operations to reach these objectives need to be executed. These ‘tasks’ have to be supported by the COMRADES platform in different many ways. On one hand, the platform should support joint task handling and resource identification and assessment for each stakeholder in the most dynamic perspective by the synchronization of schedules. On the other hand, based on the shared situational understanding approach which COMRADES aims to support, the platform should allow every stakeholder to be warned and informed by notification alerts. For instance, the notifications could have pointed out the end of a task or new alerts which help to coordination the emergency response. Finally, to support the identification and execution of the tasks, exchange and sharing information is a key prerequisite. In order to achieve those requirements, the COMRADES platform should support both the characterization of the information and the constraints determination in different aspects. The characterization of information helps the stakeholders to analyze and catalogue the information they received. The constraints are determined in four fundamental aspects which are: judicial, ethical, moral, and deontological.

10.2 Results Validation of the COMRADES objective From the various evaluations, the COMRADES objective and resilience building are recognized by the workshop participants as important elements. COMRADES builds on existing topics and trends in disaster risk reduction and crisis response. The changing role of the community as key stakeholder before, during and after disaster strike is recognized by the participants. Information is also considered as a key component of the community-driven resilience process according the received input. Nevertheless, the participants recognize some key challenges in implementing the COMRADES resilience definition. The strong focus on the community as key stakeholder should be balanced with the responsibility of the government to enhance the resilience of the community. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

42 | 51

D2.3 Socio-Technical Community Requirements In fact, in some (or most) cases the government (should) represent the community and are â&#x20AC;&#x2DC;taskedâ&#x20AC;&#x2122; on behalf of the community to protect them from critical events and provide assistance. This should be considered when re-examining the responsibilities and tasks of the different stakeholders. In general, alignment with existing structures is a key element for improved coordination. However, this may differ greatly in different contexts. Although COMRADES focus on community resilience through an improved understanding of a situation (situational awareness), the workshop have illustrated that there are different forms or levels of this situational awareness. Most notably, the personal situational awareness which is most prominently present in the stages immediately following a disaster and the community situational awareness, which follows the initial personal stage. COMRADES should recognize the differences between the two, and may have to find a way to support or facilitate both. Either within the platform or by integrating other platforms and applications. This will ensure an easier adoption by individual community members. Requirements Throughout the evaluation we have examined the requirements that have been previously identified in Deliverable 2.2, resulting from the Nepal workshops. Verifying these requirements with other groups, representing both different local contexts, expertise, and experience, will help to generalize these requirements. Overall the existing socio-technical requirements have been confirmed throughout the evaluations. However, several additions have been made by the workshop participants. One of the key elements that has been confirmed by the evaluation workshops is the importance of information to support situational awareness. Through the improved exchange of information, and specifically placing the role of information management in the hands of the community, this improved understanding supports a more resilient community. Nevertheless, information management (including the collection, analysis and sharing of data) is something that does not only apply to the platform but rather extends to the community. The input from various sources, including non-digital ones, and the sharing of the outcomes with the community through various channels is a key consideration moving forward. In a broader sense, the evaluation has confirmed that the adoption and integration are key elements in a successful deployment of the platform. Not only during the initial activation of the platform, but rather the continued used by enabling an added value of the platform in daily routines. This is supported by COMRADES through the generic services and extensive configuration options. Ensuring a positive feedback loop, or added value to the users is furthermore key to ensure a sustained use. Finally, aligning and integrating the platform with existing services and platform will also support the further adoption. This is currently (Facebook integration) already part of the development roadmap. Prototype platform The prototype platform formed a key element in the evaluation. Although the platform is in the early stages and only provides limited functionality, it not only helped to demonstrate how the COMRADES objective would be implemented, but also showed the direction and focus of the developments in the project. In the evaluation sessions, the participants, in addition to an interactive demo/presentation, also had hands on experience with the platform. Through this demo/test session, useful feedback on the platform was received. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

43 | 51

D2.3 Socio-Technical Community Requirements Overall the workshop participants did see the potential of the COMRADES platform to address the COMRADES objectives and achieve community-based resilience building through improved information sharing and building a shared situational awareness. The demo sessions allowed the participants to define their own use-case and implement it on the demoplatform. Additionally, existing reports (such as news bulletins) or messages created by the workshop participants were added to the platform to demonstrate how the YODIE and CREES service would process this data. The results were well received by the workshop participants as these services were able to identify important keywords in the messages and tag the report with relevant categories. Although users emphasized the importance of support for end-users in setting up and configuring a platform to fit with their specific use-case, they were able to configure the platform themselves with limited guidance. More importantly, the services provided are valuable additions to platform to provide an added value to both users and deployers of the platform. Workshops participants did however mention that these services are one possible implementation of the COMRADES objective and many other services could be devised and added, illustrating the importance of an open platform and service-oriented architecture. Alignment Internal evaluation and Workshops The results from the internal evaluation illustrate that COMRADES is developing and integrating services with unique capabilities that support the COMRADES objective. The internal evaluations demonstrated the potential of the services to support communities in their information management. The high performance and accuracy rates reduce the burden on communities to assess the incoming reports. These findings support the feedback received from the external evaluation sessions. During these sessions, the importance of supporting communities that do not have previous information management expertise and resources was highlighted. The services are able to successfully address the Information Management needs automatically and thus support communities. While these are still early versions, it demonstrates that the direction of the developments so far is in line with both the opinions and ideas of the evaluation participants as well as the overall COMRADES objectives.

10.3

Implications for COMRADES

The evaluation findings not only support the COMRADES project results thus far, but also provide some valuable insights and considerations for the project going forward. These considerations provide some important insights for the technical developments, specific functions, and features, as well as important considerations for the socio-technical requirements and adoption of the platform. Adoption and integration in daily use A recurring element throughout the evaluation as well in the previous WP2 deliverables is the adoption and integration of the platform by users. During the evaluations people emphasized that this adoption and integration is accomplished by ensuring the platform has a continued added value. In other words, the platform should be designed, configured, and used in such a way that it would be used in â&#x20AC;&#x2DC;daily lifeâ&#x20AC;&#x2122;. This could be accomplished in different ways, for example by integrating with other platforms, or designing the platform towards use-cases that occur more frequently. Also, the option to integrate with existing community resilience efforts such as assessments and exercises has been mentioned during the evaluation. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

44 | 51

D2.3 Socio-Technical Community Requirements Role of government and community champions Although COMRADES has a strong focus on the communities as key decision making, during the evaluations the role of the government (either local, regional or national) and international organizations has also been mentioned. In part as a possible dissemination channel for the COMRADES platform as they have a wide network or already close link with communities. But also, a key prerequisite for the adoption of the platform, as these organizations and agencies can provide important and verified information that will support the community in improving its shared situational understanding. However, this aspect and the role of official instances in the platform need to be balanced with COMRADESâ&#x20AC;&#x2122; ambition for community ownership over the platform. One possible direction is to involve community champions, leaders, or committees. These groups often represent some form of local governance (either self-established or through government mandate). As such, the work of these groups can be supported with our platform, and they can advocate the use to their community. At the same time, they could liaison with their government officials using the information on the platform, either digitally or in person. Finally, Volunteer & Technical communities are another possible group that could adopt the platform and advocate its use. These V&TC groups are closely aligned with the COMRADES ideas and objectives. Moreover, they have the technical knowledge and interest, and are familiar with using technology in crisis response context. The COMRADES project can support them in expanding their services beyond humanitarian organizations and to local communities. They also provide a good network to further disseminate the project results. Knowledge exchange The objective of supporting knowledge and information exchange between the different COMRADES communities is something that has been recognized by all workshop participants. The participants as well as the underlying theory refer to this as a unique aspect of the platform that will push the agenda of empowered communities further. Although COMRADES is currently already working regarding the exchange of information, the workshops have shown that this should also be expanded towards more general knowledge. Communities often have local knowledge that is relevant to responders and organizations that aim to support the community. This knowledge enables them to deliver aid more effectively. At the same time, organizations have knowledge that is relevant to the local community. The platform could facilitate communities in building their own knowledge base (for example extracted from the provided information). Having this knowledge available allows communities to more easily connect to other stakeholders, and at the same time provide them with a sense of ownership of their collected information and the extract knowledge. Facilitation and integration Throughout the workshops, the need for support has been mentioned by the participants. The platform and the services provide a great potential for communities to be more empowered through a better situational understanding and information exchange. However, the many options and configurations that are possible with the platform require expert knowledge. At the same time, there is a multitude of different use-cases that we have encountered throughout the COMRADES project, along with cases that workshop participants have provided. In order to effectively leverage the platform toward improved community resilience, support and expertise is needed to aid employers in setting up their own instance. ÂŠ Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

45 | 51

D2.3 Socio-Technical Community Requirements This does not only apply to the initial deployment of the platform, but also for the maintenance and continuous improvements, especially considering that communities and thus the requirements of a platform change over time. It is questionable if supplying guides and formal (explicit) support and information about the platform is sufficient, most likely some user support group has to be formed where best practices can be exchanged, advice can be offered and support can be provided. This may align with the Volunteer & Technical communities (see above). First and last mile At the moment COMRADES provides services for the analysis of digitally provided texts and reports. Considering the increase of digital communication this is a logical focus. However, we should remember that not all relevant information might be communicated digitally, or on the COMRADES platform. Nevertheless, the platform and its services rely on the information being available digitally, although this will not happen automatically. Getting people to put information (or reports) on the platform in the first place is a key step. Similarly, having information on the platform does not automatically mean it reaches those who might benefit from it. During the development of the project and platform it is important to remember this ‘first’ and ‘last’ mile communication. The first mile, referring to the ingest of data and information in the platform. Depending on the local context, available technology and common communication, not all information would make it to the platform automatically. Reporters need to be aware of the platform and might need to have an incentive to provide the information. At the same time, not everyone with information to provide will have access to platform and some might need a ‘proxy’ to share the information on their behalf. The same applies to the sharing of the information on the platform back to the community. This could also happen through other channels like FM radio.

10.4 Next Steps & outlook for evaluation Finally, we reflect on the evaluation itself and the results obtained. This initial evaluation had a broad scope, ranging from reflection on the overall COMRADES objective to the specific implementation in the current prototype platform. Considering the early stage of the platform, the evaluation focused not only on evaluating the current state of the platform and project, but also solicited additional input for future developments. Moving forward with the project, one of the key challenges is to narrow down the scope of the project, to ensure a feasible development path and a clear ‘sales-pitch’ for the final platform. Therefore, the next evaluation would focus more on a specific scope, and bringing together different stakeholders in one evaluation setting to further resolve scoping decisions and focus on synergies between different COMRADES communities. Alignment & scoping considerations One of the objectives of the evaluation was to compare the ongoing internal developments with the COMRADES objectives and requirements. According to the evaluation results, the ongoing developments are in line with the overall COMRADES objectives. Participants have mentioned that the increased information flow will make it harder to transform information into actionable results that can support the community resilience. Therefore, the current services do support the overall COMRADES objective. © Copyright 2016 Tina Comes, Christophe Nemnom, Kenny Meesters

46 | 51

D2.3 Socio-Technical Community Requirements However, it should be noted that there are many other options and functions that can be added that contribute as well. Throughout the workshops the participants have contributed many ideas on the functions and suggestions. Combined with input received from earlier interviews (part of D2.1) and the workshops conducted in Nepal (results in D2.2) the COMRADES project has received a large amount of input, ideas, suggestions, and considerations. In addition, these inputs cover a wide range of stakeholders; people from different countries and cultural background (across the world), different roles (responders, community leaders, politicians, volunteer & technical communities, different levels of experience and expertise, etc. The results lead to many possible directions and considerations for COMRADES. Therefore, a key challenge is to narrow the scope of the project. This will ensure that the outcome will have a clear use-case and purpose and not be too broad with limited depth, reducing the added value to specific organizations and communities. Evaluation considerations The evaluation workshops conducted in June 2017, are mainly aimed at validating the ongoing COMRADES work. This includes the underlying concept, objective, and approach for the overall project (at outlined in the WP2 deliverables) as well as the ongoing technical development of the platform. In addition, this evaluation has mainly focused on two COMRADES communities that have not been the main focus of earlier engagements in the project: the (professional) responders and the Volunteer & Technical communities. In addition to bringing in their own perspective and providing additional input, they as helped to validate the findings from the engagements with the (affected) community and the reporters. serving as a ‘control group’. The next round of evaluations, planned for M24, should provide a more comprehensive evaluation. This applies to two main components of the evaluation. First, the evaluation should bring together the different COMRADES communities in one evaluation. This will help to examine the coordination between the different stakeholders in real-time. It would also help to identify on which elements synergy can be achieved or where the different groups diverge. The evaluation should also aim to be more hands-on. While understandable given the current phase of the project, the evaluation and engagements have so far mainly focused on eliciting and validating input. However, the exact setup, scope and execution of the evaluation will depend on the ongoing developments and the availability of evaluation participants.

47 | 51

D2.3 Socio-Technical Community Requirements

Appendix Appendix A.1 References Cox, R. S. (2008). Assessing community resilience in the context of disaster and emergency management (Unpublished manuscript). Public Health Agency of canada-Centre for Emergency Preparedness and Response, Ottawa, Ontario, Canada. Cox, R. S. (2008). Disaster resilience. Paper presented at the proceedings from the Annual Roundtable for Canadaâ&#x20AC;&#x2122;s Platform for Risk Reduction 2011, Ottawa, Ontario, Canada. Cox, R. S. and Hamlen, M. 2014. Community Disaster Resilience and the Rural Resilience Index. American Behavioral Scientist. Malone, T. W. and Crowston, K. 1990. What is coordination theory and how can it help design cooperative work systems. In Proceeding of the Third Conference on Computer-Supported Cooperative Work. ACM Press, New York, 357-370. Malone, T. W. and Crowston, K. 1990. The interdisciplinary Study of Coordination. In ACM Computing Surveys (CSUR), Volume 26 Issue 1, March 1994, ACM New York, 87-119.

48 | 51

D2.3 Socio-Technical Community Requirements

A.2 Guiding questions for focus group Main question: To what extent coordination approach between communities and disaster responders influences local community disaster resilience? Part 1: objectives, ambition, and ideas Expected outcome: ○ In general: Illustrate the importance and added benefits of coordinated approach between communities and responders. ○ COMRADES specific: Verify the ambition of COMRADES definition: to empower and engage communities in their resilience building through information. Guiding Questions: ● What role do different stakeholders, and especially communities play in a response? ● Why is it necessary to coordinate between them? ● What role do and should communities play in a disaster response? ● How does community engagement contribute to a more resilient community? Part 2: requirements/ design Expected outcome: ○ In general: What should be considered to empower communities as a decision makers? ○ COMRADES specific: Verify the requirements and assumptions about information needs and the use of information by the community to improve their resilience. Guiding Questions: ● What elements should be considered in a system to empower communities? ● Besides technical considerations, what other requirements should we keep in mind? ● Coordination is the process of managing dependencies among activities (Malone and Crowston, 1994) What is a good ‘coordinated approach’ in a disaster context? How should the responders perform it? ● What are the key elements which help the main stakeholders to be coordinated in disaster? What relationships exist between them? Part 3: platform Expected outcome: ○ In general: Which specific information and communication technologies support a coordinated approach between responders and communities? ○ COMRADES specific: Do we provide the correct functions and features, verify our roadmap. Guiding Questions: ● Do the current platform features contribute to improved community resilience? ● What features are functions and features are missing? ● Are the previous elements implemented in the platform? If so, how is it represented? ● Otherwise, how could they be represented and/or implemented? ● What other elements and/or functions could be implemented in the COMRADES platform in order to perform an effective coordinated approach in disasters?

49 | 51

D2.3 Socio-Technical Community Requirements

A.3. Invite to Evaluation Workshop What is COMRADES? COMRADES is a project funded by the European Commission under the H2020 program. The goal of the project is to use intelligent information and communication technologies to help communities in times of crisis. The COMRADES project is centered around community resilience: the ability of communities to help themselves and each other in times of crisis. Communities who are affected by disasters are faced with questions. Who is in need and who can help? What aid is needed and what is available? When does aid arrive? To support communities with these questions they need information such as the current situation, the risks and the different options they have. If communities have access to this information they can make the right choices and are more effective in helping each other. The platform COMRADES is developing a platform that supports communities in gathering, processing and sharing this information. The platform is built on the existing Ushahidi software used to crowd-source information. COMRADES extents the platform with new intelligent features to help communities understand the situation, present decisions to be made and give them actionable information to make the right choices. This platform is designed for and with communities around the globe. Using a communitycentered design approach COMRADES is not only looking at the technological possibilities, but also examining the information needs of communities. The goal is to develop the platform in such a way that communities can deploy, use and maintain it themselves. Our question to you At the moment the COMRADES platform is still under development. We are reaching out to different communities, activists, developers and experts to get their input and ideas on the project. We ask you to join us for an interactive workshop and discussion in which we will briefly present our ideas, project and platform. But most of all we want to open the floor and listen to your ideas, suggestions and comments on how to help communities empower themselves with information. Below you will find a brief outline of the session More information For more information you can check our website: http://www.comrades-project.eu/ or contact Kenny Meesters: k.j.m.g.meesters@tudelft.nl

50 | 51

D2.3 Socio-Technical Community Requirements Program We have divided the workshop in three different parts, each starting with a small presentation followed by a interactive discussion where you can share your ideas, comments and suggestions. The times mentioned here are a suggestion, but we aim to complete the program within 2 hours. Time (min)

Subject

Details

10 min

Introduction

A short introduction of the session and the people

10 min

COMRADES introduction

Introduction of the project including the objectives, the aims and the approach.

15 min

Feedback Session 1

Your feedback on the objectives and ideas: ● Do we have the right objectives? ● Are they feasible? ● Will our approach provide the right results?

15 min

Requirements

Presentation on the requirements we have identified for the COMRADES platform.

20 min

Feedback Session 2

Your feedback on the requirements: ● Do we have the right requirements? ● Are there things we should keep in mind? ● Do we provide the right functions?

20 min

Platform demo

An interactive demonstration of the COMRADES platform and several of its key features.

20 min

Feedback Session 3

Your feedback on the current platform: ● Is the platform addressing our objective? ● Is the design inline with the requirements? ● Do we need to change our direction?

10 min

Closing

Conclusion of the session and any other points that you want to discuss.

51 | 51