DELIVERABLE Project Acronym:
APOLLON
Grant Agreement number:
250516
Project Title:
Advanced Pilots of Living Labs Operating in Networks
D.5.5 - Evaluation Report on Cross Border Experiment
Revision: Final
Authors: •
Tanguy Coenen (IBB)
•
Claudio Vandi, Charles Tijus, Baptiste Gendrot (UP8)
•
Alan Holding (MCC)
•
Sebastien Levy (ISS)
Project co-funded by the European Commission within the ICT Policy Support Programme Dissemination Level P
Public
C
Confidential, only for members of the consortium and the Commission Services
X
Apollon – D.5.5 Evaluation Report
Revision History Revision
Date
Author
Organisation
Description
0.1
19/12/2011
Tanguy Coenen
IBBT
0.2
22/12.2011
UP8
0.3
10/01/2012
Claudio Vandi
Brussels experiment evaluation based on Antwerp experiment and first Manchester experiment
0.4
15/01/2012
Claudio Vandi
UP8
0.5
21/01/2012
ISS
Issy experiments evaluation
0.6
23/01/2012
Catalina Vasilescu
IBBT
Review
0.7
24/01/2012
Tanguy Coenen
ISS
Review
0.8
24/01/2012
Sebastien Levy Baptiste Gendrot Charles Tijus
UP8
Integration of partners’ inputs and lesson learned
Alan Holding Baptiste Gendrot
MMDA
Integration of partners input and first structure
Manchester experiment evaluation Qualitative evaluation of the pilot as a whole Lessons learned
The information in this document is provided as is and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. Statement of originality: This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both. ICT PSP APOLLON
2
Final Version
Apollon – D.5.5 Evaluation Report
Table of Contents 1 WP5 Evaluation ...................................................................................................................... 5 2 Introduction ............................................................................................................................ 5 3 Description of the Pilot ........................................................................................................ 5 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8
Issy-les-Moulineaux experiment (France) - The Digital Fort Use Case ..................... 5 Issy-les-Moulineaux experiment 2 (France) – Museum of Playing Cards Use Case 6 Antwerp experiment (Belgium) – M HKA Museum Use - Case ...................................... 6 Manchester experiment (UK) – Manchester Art Gallery Use Case .............................. 6 Research Focus .............................................................................................................................. 6 Cross-border collaboration focus ........................................................................................... 7 Co-design of the Pilot ................................................................................................................... 9 Method for Data Collections for the Pilot .......................................................................... 11
4 Description of the experiments ......................................................................................15
4.1 ISSY Fort ........................................................................................................................................ 15 4.1.1 Partners involved and their role ................................................................................................15 4.1.2 Overall purpose of the experiment ..........................................................................................17 4.1.3 Transfer focus ....................................................................................................................................17 4.1.4 Research Set-up and Data Collection Methods ...............................................................17 4.2 Antwerp M HKA .......................................................................................................................... 17 4.2.1 Partners involved and their role ................................................................................................18 4.2.2 Overall purpose of the experiment ..........................................................................................18 4.2.3 Transfer focus ....................................................................................................................................18 4.2.4 Research Set-up and Data Collection Methods ...............................................................18 4.3 Manchester Central Library refurbishment (unachieved) ......................................... 18 4.4 Manchester Art Gallery Manchester Art Galley .............................................................. 19 4.4.1 Shortly describe the experiment ..................................................................................................19 4.4.2 Partners involved and their role...................................................................................................21 4.4.3 Overall purpose of the experiment ..............................................................................................22 4.4.4 Transfer focus .......................................................................................................................................22 4.4.5 Research Set-up and Data Collection Methods .......................................................................22 4.5 French Museum of Playing Cards (Issy-les-Moulineaux) ............................................ 23 4.5.1 Partners involved and their role ................................................................................................23 4.5.2 Research Set-up and Data Collection Methods ...............................................................24
5 Qualitative Evaluation of the Pilot as a whole ...........................................................26
5.1 Technology transfer and integration .................................................................................. 26 5.1.1 Shifts from planned work ................................................................................................................30 5.2 Users involvement ..................................................................................................................... 31 5.3 Cross-border Collaboration Experiences & Evaluation ............................................... 32 5.3.1 Scenario design and collaboration management ...................................................................32 5.3.2 Communication ....................................................................................................................................34 5.3.3 Distributed Application management ........................................................................................34 5.4 Benefits .......................................................................................................................................... 35 5.4.1 Test of technology in different cultural settings ....................................................................37 5.4.2 Increased understanding of different stakeholders perspectives ..................................38 5.4.3 Increasing income or status............................................................................................................38 5.4.4 Learning and sharing good practices ..........................................................................................38 5.4.5 Attracting EU funding ........................................................................................................................38 5.4.6 Enhancing broader cooperation ...................................................................................................38 5.4.7 Iterating a test in different cultural settings ............................................................................38
ICT PSP APOLLON
3
Final Version
Apollon – D.5.5 Evaluation Report 5.5 Challenges ..................................................................................................................................... 39 5.5.1 Harmonization of partners’ objectives.......................................................................................39 5.5.2 Sustainability ........................................................................................................................................40 5.5.3 Knowledge sharing .............................................................................................................................40 5.5.4 Cross-border training ........................................................................................................................40 5.5.5 Reduced budget for travel and development ..........................................................................42 5.5.6 Match-making .......................................................................................................................................42 5.5.7 Language.................................................................................................................................................43 5.6 Methods and Tools..................................................................................................................... 44 5.6.1 Communication and collaboration tools ...................................................................................44 5.6.2 Distributed application development.........................................................................................44 5.6.3 Cross-Border training .......................................................................................................................44 5.6.4 Observation, data collection and focus groups .......................................................................44 5.6.5 Knowledge sharing .............................................................................................................................45 5.7 Partners – End-Users ................................................................................................................ 45 5.8 Partners – SME ............................................................................................................................ 45 5.8.1 Air Graffiti – MuseUs ..........................................................................................................................45 5.8.2 Navidis .....................................................................................................................................................46 5.8.3 Virdual .....................................................................................................................................................47 5.8.4 People Voice Media ............................................................................................................................47 5.9 Partners – Local Authorities .................................................................................................. 48 5.9.1 ISSY MEDIA ............................................................................................................................................48 5.9.2 Manchester Galleries .........................................................................................................................48 5.9.3 ISSY Museum of Playing Cards ......................................................................................................49 5.10 Living Lab Perspective ........................................................................................................... 49 5.10.1 Some benefits for IBBT ..................................................................................................................49 5.10.2 Some benefits for MDDA ...............................................................................................................49 5.10.3 Some benefits for LUTIN (UP8 Living Lab) ...........................................................................49 5.10.4 Benefits for ISSY MEDIA (as a Living Lab) .............................................................................50
6 Quantitative Evaluation of the Experiments ..............................................................51
7 Lesson learned from specific pilot results ..................................................................54
7.1 Contextual Factors ..................................................................................................................... 54 7.1.1 Technology and cultural barrier ...................................................................................................54 7.1.2 Language barrier .................................................................................................................................54 7.1.3 Copyright issues ..................................................................................................................................55 7.2 Eco-Systems.................................................................................................................................. 55 7.2.1 Stakeholders identification .............................................................................................................55 7.2.2 Goals definition ....................................................................................................................................55 7.2.3 Openness of the consortium ...........................................................................................................56 7.3 Interoperability Issues ............................................................................................................. 56 7.4 Lead market opportunities .................................................................................................... 57 7.5 Collaboration Process .............................................................................................................. 58 7.5.1 Matchmaking: Living Lab and SMES roles and profiles in technology integration pilots 58 7.5.2 Knowledge transfer ............................................................................................................................58 7.5.3 IPR .............................................................................................................................................................59
ICT PSP APOLLON
4
Final Version
Apollon – D.5.5 Evaluation Report
1 WP5 Evaluation 2 Introduction The present deliverable provides an evaluation of the APOLLON WP5: eMedia & eParticipation cross-border pilot led in 3 cities across Europe: Antwerp, Issy-lesMoulineaux and Manchester. The role of our pilots is to demonstrate that the use of eMedia reinforces the participation and involvement of the citizens in innovation and creation processes and technologies development. The focus of this deliverable is to evaluate the results and benefits brought by the pilots. This evaluation will focus on qualitative and quantitative elements as well as the issues that need to be address for an efficient cross-border experiment. This evaluation will show the benefits for the SMES and for the citizens that are all different criteria of success. The purpose of this document is to supply guidelines for Livings labs/local administration and SMEs which want to create successful eMedia services for citizens. To do so in this deliverable we will expose the issues we came across before the implementation of the pilot and the issues that came up during the pilot. This deliverable will try to present all factors that need to be taken into account in order to implement successful eMedia for eParticipation services be it cultural, technological or social.
3 Description of the Pilot The WP5 pilot was set up to test how integrated eMedia technologies can encourage eParticipation and what are the advantages, best practices and limitations of cross-border activities within the Living Lab network.
The pilot was designed to cluster three running local Living Lab projects in three countries, in which social media are being used to stimulate and facilitate the participation of citizens: the Issy-les-Moulineaux Medialand in France, the Manchester Digital Living Lab in the United Kingdom and the Antwerp Museum of Modern Art in Belgium. In this experiment one specific technology and related service within each Living Lab was selected (3D technologies from French SMEs Navidis and Virdual, social media and community reporters programs from the UK NGO People’s Voice Media, context aware mobile application from the Belgian start-up AirGraffiti now named MuseUS), with the objective to adapt and integrate them with each other. A fourth experiment replicating the Manchester scenario was carried out at the Museum of Playing Cards in Issy in January 2012.
3.1 Issy-les-Moulineaux experiment (France) - The Digital Fort Use Case The French experiment tested an interactive multimedia trail on the History of the Fort of Issy, based on the reconstruction of the fort in 3D, augmented reality, geoloICT PSP APOLLON
5
Final Version
Apollon – D.5.5 Evaluation Report cation, community reporting, QR codes and cross-media. The interactive trail has become a permanent cultural service offered by the city of Issy-les-Moulineaux, accounting for more than 3,000 users.
3.2 Issy-les-Moulineaux experiment 2 (France) – Museum of Playing Cards Use Case The second French experiment replicated the Antwerp and Manchester scenarios, testing the MuseUs application at the French Museum of Playing Cards within the “Alice in playing cards’ land” exhibition. The purpose was to test people’s reactions to the use of mobile games when visiting an exhibition. We focused on the remediation of a number of issues that were the result of the MuseUs trials in Manchester (e.g. lack of progress indication in the application) and tested the software on tablet computers instead of Smartphones.
3.3 Antwerp experiment (Belgium) – M HKA Museum Use - Case The Belgian pilot tested the impact and added value of a mobile 3D game application on youngsters’ museum perceptions and experience in a museum setting. The study has indicated that youngsters overall experienced visiting a museum, rather than playing a game. This is an encouraging result for applications focusing on insitu audience engagement with cultural heritage. The application is currently in beta testing. Release on the market (AppStore) is planned for 2012.
3.4 Manchester experiment (UK) – Manchester Art Gallery Use Case The English experiment worked with Manchester Art Gallery to evaluate the Decoding Art project, which allows visitors and citizens to obtain information about onstreet public art works by scanning QR codes on their mobile phones. The testing sessions took place in April 2011 and were a success as they allowed Gallery staff to use Living Labs approaches to improving public services and engaging with citizens. The ideas from the sessions informed further development of the project. It also provided a baseline evaluation to use for comparison when IBBT's MuseUs application which was tested in the Gallery in November 2011.
3.5 Research Focus The Research Focus of the eMedia and eParticipation Pilot was twofold: to understand what are the opportunities and challenges that arise when integrating complementary technologies; and to test if eMedia technologies are an effective way for engaging users to participate in eParticipation and cultural activities.
eParticipation was addressed in a wide sense of involving citizens in the city life by providing them innovative digital tools that would facilitate their participation to the city cultural life and history. ICT PSP APOLLON
6
Final Version
Apollon – D.5.5 Evaluation Report As far as Living Lab role and missions are concerned WP5 focused on understanding how Living Labs can:
• Enable local SMEs and citizens with new ideas and products for using eParticipation for citizen engagement to gain access to local resources which can be used to test and evaluate those new ideas and products; • Take clients through a structured process for evaluating new ideas and products;
• Facilitate introductions to useful business contacts locally and across Europe; •
Showcase new ideas and products at a local and European level;
• Provide a Single Point of Contact for local stakeholders wishing to use Living Labs methods to improve services;
3.6 Cross-border collaboration focus The main cross-border activities consisted in the transfer of technologies and knowledge across partners Living Labs and SMEs. The approach we used was to build a series of experiments that involved different ways of combining the same technologies and know how. The goal of the pilot was to run at least one experiment per country and with at least one cross-border activity per experiment. The main cross-border activities were the following: •
•
•
•
IBBT o o o o o o o o
Knowledge of game design The game concept Graphic design Cross-platform development of overall application framework App-based support of QR code scanning Back-end Drupal programming API definition and implementation Living lab testing methodology
Virdual o 3D virtual world design and implementation o Shiva to iOS application development o Game design and historical research
Navidis o 3D virtual world design and implementation o Development of a data transfer protocol to allow third party applications to interact with Urbadeus o Knowledge of Urban game design M HKA Museum
ICT PSP APOLLON
7
Final Version
Apollon – D.5.5 Evaluation Report
•
o Knowledge of what works in interaction with young museum visitors o Knowledge of the academic literature on how museum visitors approach art works o Access to museum for application testing in Living lab environment o Content on works of art, custom made for the target audience (youth between 14 and 24 years) o People Voice Media o Community reporters program o Training in the usage of New Media for community reporters
•
MDDA Living Lab o Access to and liaison with local partners o Localization support, like translation of the survey and nativespeaker interactions with the test users o Local logistics
•
LUTIN (UP8) o Supervision of the pilot scenario design o Harmonization of partner’s goals o Living Lab methodology o Translation and adaptation of evaluation materials
•
•
•
•
ISSY o Local living lab management , access to and liaison with local partners o Access to local SMEs o Local logistics Localization support, (adaptation of the survey, training support, interaction with the test-users) Mobexplore (supporting SME) o Game design o iOS application o Integration of the Navidis data exchange protocol
Manchester Art Gallery o Access to museum for application testing in Living lab environment o Content on works of art Museum of Playing Cards o Knowledge of how to attract visitors o Coordination of groups visits o Content provision for the development of the application o Access to its premises for application testing
ICT PSP APOLLON
8
Final Version
Apollon – D.5.5 Evaluation Report
3.7 Co-design of the Pilot The co-design of the pilot was one of the most labor-intensive phases of the project and involved various iterations that led to the final scenario. The main challenges were the harmonization of partner’s objectives and the integration of SMEs partners’ technologies with no budget directly allocated to software development. While each partner was responsible for leading the organization of its local experiment, French partners led the design of the overall pilot scenario. The main reason for this is that the majority of the consortium being in Paris (4 out of 7 partners) it was easier to meet to coordinate the activity locally before validating it at the consortium level. The leading partner ISSY had the role of coordinating the pilot on an operational level, the research partner LUTIN (UP8) had the role of monitoring the process making sure that one pilot was carried out in all the participating countries and that there was at least one technology transfer activity in each of them. The pilot scenario was finalized during two face-to-face meetings in which all the partners participated: one in Issy (April 8-9th 2010), one in Manchester (June 2122nd 2010). The first experiment was carried out in Issy-les-Moulineaux in March 2011 and served as a pilot for the other experiments. For all the experiments, the main steps were the following:
1. The local living lab orchestrates the construction of the pilot with local SMEs and stakeholders; 2. The cross-border SMEs express their needs and objectives for the experiment;
3. The local living lab organizes co-design sessions with local stakeholders and SMEs trying to match the scenario to the cross-border SMEs needs;
4. The cross-border living lab monitors the process to make sure the scenario matches the SMEs goals but doesn’t interfere with it unless there is a need to moderate a debate or harmonize the objectives of the different stakeholders and SMEs;
5. The cross-border SME visits the local living lab for a co-design session in which the scenario for the experiment is finalized;
6. The local living lab is in charge of coordinating the operational phase of the experiment; 7. The cross border SME joins the local living lab for the kick off of the experiment;
8. The local living lab is in charge of collecting and analyzing data from the pilot;
ICT PSP APOLLON
9
Final Version
Apollon – D.5.5 Evaluation Report The Antwerp experiment example We describe here the process of co-designing the Antwerp experiment as an example of how local experiments were planned and designed and as a good practice for future cross-border experiments:
In a first phase, IBBT had 4 intensive co-design sessions in which IBBT and M HKA attended. Virdual, who had to come from France, attended one session during which they presented the technology they could provide. The co-design sessions were crucial in obtaining an application design that reflected the interests of all the involved parties and was compatible with what was expected, which was to engage the user with the art in the museum. It was clear that only in this way would it be feasible to develop and test the pilot.
After the co-design sessions, IBBT created the application. However, it did not seem possible to develop the prototype as an integrated whole. Therefore, it was decided during a meeting in Paris that the solution was to make sure that the two applications (the overall IBBT app framework and the Virdual 3D interface) would communicate with each other in a manner that would be seamless to the user. However, the install procedure required that two different apps be installed on the smartphone.
After the development phase, the M HKA PP (i.e. the first prototype) was tested in the M HKA museum, using living lab methods. An observation protocol and focus groups yielded data that was analysed and reported on in a report. In addition, the living lab tests provided us with the necessary input to define areas of improvement for future versions of the application. After the testing, a debriefing meeting with M HKA provided useful insight in possible business models for the future versions of the app. This has led to possible adoption of a platform-oriented redesign of M HKA PP, which has now been renamed to MuseUs.
The M HKA experiment was also invited to the Future en Seine festival in Issy in June, to present the application to an audience of cultural stakeholders. This yielded interesting leads for possible future commercial opportunities with new versions of the app. After the summer of 2011, work was continued on the application, which led to the MuseUs app (i.e. the second prototype). A more generic data structure was used that will allow the easy portability of the application to different museums. In addition, the user interface of the application was simplified as requested by users during the first user tests that were conducted during the living lab sessions at M HKA. In November and December 2011, living lab experiments were carried out at the Manchester Art Gallery, which yielded valuable additional feedback to be incorporated in future versions of the application. In January 2012, a living lab experiment took place at the Issy Museum of Playing cards for testing an improved version of the MuseUs application.
ICT PSP APOLLON
10
Final Version
Apollon – D.5.5 Evaluation Report
3.8 Method for Data Collections for the Pilot All along the pilot we used qualitative and quantitative methods to collect data from local experiments. In each case the local living lab was responsible of collecting and analyzing the data. Methods and tools were adapted to the context of the experiment. In open settings like the one of first French pilot (Issy Les Moulineaux) where people were invited to follow a path in the city we decided to adopt noninvasive observation techniques like participatory observation and interviews. This choice is a good option when the living lab wants to evaluate the user experience in a general and unobtrusive way, and provided useful insights about the appreciation of eMedia technologies for eParticipation. However, this approach turned out to be suboptimal to collect user data in cross-border settings in a structured way. Based on this experience, in the Antwerp, Manchester and in the second Issy pilot that took place in closed settings (museums) we decided to adopt more precise observation methods and to follow a common protocol across the three experiments. This protocol is described in the “Observation, Data Collection and Focus group” method description document. During the pilot, the methods we used were the following: – Experts meetings
– Direct observations and one-to-one interviews
– On-line questionnaires
– Web analytics
– Direct observation – Focus groups
In the first Issy pilot the methods being used consisted of application log analysis, direct observation of participants, semi-structured interviews filmed by webreporters along the trail (some of them published here https://issyreportages.wordpress.com/ ) and questionnaires for web reporters.
For the Antwerp, Manchester, and the second Issy experiments using the same protocol the local living lab was in charge of translating the questionnaire provided by IBBT in order to ensure the data collected were coherent across different countries. Method used for: Observation, Data Collection and Focus Groups in a museum context. 2. Problem or requirement in the WPx addressed by the Method Sharing and comparing technologies/methodologies in order to understand to what extent local results can be extended to other contexts and which common technology/methodology can be built for generalization ICT PSP APOLLON
11
Final Version
Apollon – D.5.5 Evaluation Report 3. Description of the Method The method consists of a combination of quantitative and qualitative research methods. In particular, it applies a contextual approach as it builds on data that was gathered through transaction data logging, observations using a semi-structured observation protocol, and by means of focus group interviews via a semi-structured interview guide. 4. Use of the Method
Within the framework of the Apollon project, IBBT-SMIT conducted research into the impact of the mobile game application that was developed within the project on the museum experience of young museum visitors. The first version of the game application is the result of a close cooperation between IBBT-iLab.o (who delivered the MuseUs technology) and the French cross media production company Virdual (who provided the 3D technology), in close consultation with the Antwerp museum of contemporary art “Museum van Hedendaagse Kunst Antwerpen” (M HKA, Belgium; who provided the content), the latter both small and medium enterprises participating in the Apollon project as partners (Virdual) or supporting partner ( MHKA). In close consultation with M HKA staff members, IBBT-SMIT researchers facilitated the implementation and user assessment of the game on site. The prim ary aim ofthis research co nsisted of examining the experiences and practices related to the application’s use. In particular, we intended to assess the impact of a game on visitor’s movements, artviewing habits, interaction patterns, learning behavior, etc. In a wider perspective, this research explores the impact and added value of game applications on cultural heritage perceptions and experiences in a museum setting. This method was adapted and reused for the Manchester Gallery experiment (involving MDDA Living Lab) and for the Issy Museum of Playing cards experiment (involving Issy and Lutin Living Labs). The focus of the first tests in Antwerp was more exploratory in nature. Therefore, qualitative methods were used, which have the property of yielding a larger overview of issues that are unknown to the researchers. In the second (Manchester) and third (Issy) trials, we complemented the qualitative approach with a quantitative element, to get a more detailed view of the scale and the importance of certain issues. 5. Implementation of the Method a. Transaction Log Analysis All user activity within the game application was automatically logged. Transaction log analysis is a well-established method in various information sciences, which comprises the study of “electronically recorded interactions between online information retrieval systems and the persons who search for information found in those systems” (Peters, 1993; cited in Jansen, Taksa and Spink, 2009). It enables both the analysis of aggregate user data and individual patterns. The data in this study was collected during three user sessions, each logging session comprising approximately 1 hour. User activities were time stamped and include: (1) scanned work of art per time use per participant, (2) the number of times a ICT PSP APOLLON
12
Final Version
Apollon – D.5.5 Evaluation Report piece of art was matched to an assignment and (3) the number of times an art work was scanned as favourite work.
b. Observations We opted to gather specific use and game experience data by means of observation, an established and standardised method to collect data in diverse qualitative research designs. By means of a pre- designed structured observation protocol, IBBT-SMIT researchers observed the participant groups while they were briefed and engaged in the game assignment. In particular, the researchers assumed the social role of “observer-as-participant”, an observational role in which the observation activities are explicitly made knowable to all participants. The recorded data served as input during the subsequent focus group interviews, and was carefully considered as triangulation technique. The observational protocol was subdivided into three separate chunks aimed at mapping use and, to some extent, usability concerns, yet with a particular interest in the impact on overall game and museum experience. The following items were observed: At the time of briefing the participants, the following items were observed:
• Do the participants ask any questions? If so, about what aspects do they have questions?
• Is there any interaction between the participants at this point? Which kind of interaction? In the course ofthe gameplay, the following points were mapped:
• Do the participants get to work easily? Was the game explanation clear enough? If they have any questions at the start of the game, what are they about? • Are the participants questioning themselves? Does the game seem clear? • How long does it take before the first QR code is scanned?
• How do the participants interact: do they discuss the game with other participants? Do they talk about other issues?
• Which reactions prevail during the gameplay? Are they positive/negative? Are there participants that defect? • Which game components receive positive evaluations during the gameplay?
• Which game components receive negative evaluations during the gameplay? • Are there any signs of frustration? When?
• Do the participants move alone, in twos or in a group? Is the game played individually? • Are the participants immersed in the game? Are they having fun?
• At what time during the observation do they appear distracted? On the level of the ICT PSP APOLLON
13
Final Version
Apollon – D.5.5 Evaluation Report museum experience, the following matters were observed:
• How do the participants move around the museum? Which route do they take? Do they halt at art pieces?
• Do the participants take in the pieces of art in any other way, in comparison with the ‘regular’ (non-participating) visitors that are present? • Do the participants seem more involved with the art pieces? Do they consciously handle the exposition?
• Are the participants particularly focused on executing the assignment? Do they see anything else inside the museum? • To which extent do they seek help from museum staff members, other visitors or peers? c. Focus group interviews
To gather in-depth qualitative data in order to examine game use and overall game experience, three focus group interviews were organised. In each focus group,the youngsters were interviewed by means of a semi-structured interview protocol about their experience while playing the game, the application’s accessibility and use, the educational game component and its broad-spectrum added value, and at the end, they were invited to give general user feedback. The interview guide was semistructured, rather than structured, allowing new questions to be raised during the interview as a result of the interview course. The three focus group interviews were audio recorded and subsequently transcribed by means of “Express Scribe”, professional audio player software designed to assist the transcription of audio recordings. Based on the transcriptions, the author then coded the data in order to generate categories for coding and analysing data.
A significant remark on the methodology is the social desirability bias as the M HKA museum itself served as location for the interviews and a museum staff member (i.e., an apprentice) was present during the interviews. We attended to this concern during the three focus groups, a.o., by presenting the focus group as a learning opportunity for both the university researchers and museum staff and by emphasising the importance and value of a scrupulous attitude for the further continuation and development of the research.
ICT PSP APOLLON
14
Final Version
Apollon – D.5.5 Evaluation Report
4
Description of the experiments See deliverable 5.4 for finer details on the scenario and the technical aspects of each experiment.
4.1 ISSY Fort
The French experiment had as use case the Digital Fort.
The Digital Fort is an ambitious project that aims to create a new Green and Hi-Tech dis-trict in the place of the old Issy Fort, a landmark site and the scene of bloody battles be-tween the French and the Prussians in the war of 1870.
In line with the city’s desire to preserve the memory of this significant page of Issy’s his-tory, the pilot partners developed an interactive multimedia trail, based on the recon-struction of the fort in 3D, augmented reality, geolocation and crossmedia. The objective was to encourage the citizens to get involved in the life of their city, to discover its history and to increase their awareness on the cultural and historical patrimony of Issy-les-Moulineaux.
The interactive discovery trail invites users to discover the History of the Fort of Issy via QR codes placed on historical information boards across the city. Users equipped with a smartphone can take the experience one step further as they have to solve riddles, report on their journey by taking pictures, geolocating themselves and then uploading the data in real time on the 3D map of the city. 4.1.1
Partners involved and their role
SME partners Virdual (FR)
Virdual is an Issean company which provides solutions for TV channels and TV producers to create crossmedia programs. For the experiment, Virdual provided an audiovisual documentary on the history of the Fort of Issy combined with a 3D interactive reconstruction of the Fort as it was back in the 19th century. The documentary proposes a guided visit of the place enriched with documents such as video sequences, engravings, sounds and pictures. The 3D reconstruction of the place allows the user to move around the 3D model of the Fort and access various media resources for more information. This cross media approach enables two different users to consult the device at the same time, according to their needs and preferences. The cross media device was used to raise citizen’s interest and make them aware of the place of the Fort in Issy's history, and was the starting point of the interactive trail. Navidis (FR)
Navidis is a cartographic and 3D media editor for local authorities, education and the general public, based in Issy. Within the project Navidis provided Issy 3D, a 3D cartography of Issy and Urbadeus, a module allowing citizens to post with a smartphone into ICT PSP APOLLON
15
Final Version
Apollon – D.5.5 Evaluation Report the 3D model of the city additional points of information for sharing city experience and providing relevant feedback. Mobexplore (FR, supporting)
MobExplore is a French company which develops location-based mobile games. Within the experiment, Mobexplore provided the smartphone application allowing users to solve riddles, report on their journey by taking pictures, geolocating themselves and then uploading the content created in real time on the 3D map of the city provided by Navidis. Non-SME partners
People's Voice Media (UK) People's Voice Media provided Community Reporters programmes, an innovative way for people to build confidence in the use of social media tools, learn new skills and tell a story about themselves or their communities.
Community Reporters Programmes were used to engage with Community Reporters and create Community Reports. People’s Voice Media worked with and trained Issy Media’s staff to the techniques of community reporting. The knowledge was then passed on to young teenagers from Issy who were recruited as Community Reporters. They made interviews and uploaded the content on a new blog especially created for them 1. Issy Media (FR)
Issy Media is a public private company in charge of the communication and Information Technologies within the city of Issy-les-Moulineaux. Issy Media carries out a range of technology-focused innovation projects on Smart Cities, Open Data and Living Labs. In the experiment Issy Media acted as the Living Lab manager, liaising with local stakeholders and coordinating the overall setup of the experiment. UP8 (FR)
Located in the suburbs of Paris, Université Paris 8 gathers nearly 500 researchers and 27 000 students. Throughout its 8 laboratories, the researchers evaluate the usability of products, services and digital media and observe user experience and user behavior from different perspectives. In the experiment UP8 was in charge of user observation and evaluation.
Living lab stakeholders 1
www.issy.com/web-reporter
ICT PSP APOLLON
16
Final Version
Apollon – D.5.5 Evaluation Report French Museum of Playing Cards (FR) The French Museum of Playing Cards is the only card museum in France and among the seven museums worldwide dedicated to this theme. In 1999 it was awarded with the European Museum of the Year Award by the European Museum Forum.
The Museum provided a detailed list of eight Points of Interest from Issy which bear some traces of the city history in relation to the Fort. The content was the base input for the smartphone interactive application created by Mobexplore.
The Museum provided also the 'setting' for the pilot launch in March 2011, representing the starting point of the discovery trail throughout the streets of Issy.
The Issean Youth Centre, Le CLAVIM (FR)
The Youth Centre provides educational and leisure activities for the Issean teenagers. Within the experiment, the Youth Centre worked with Issy Media to recruit youngsters for participation in the Community Reporters Programme initiated by People’s Voice Media.
The Youth Centre also provided the premises for the Community Reporters training sessions and appointed a coordinator for the follow up work initiated Issy Media and management of the Community Reporters group. 4.1.2
Overall purpose of the experiment
The purpose of the experiment was to test the aggregation of Media technologies using 3D, cross-media, community reporting and context aware mobile applications within the framework of the Real and more project of the Digital Fort. A secondary goal was to test and assess the impact of ICT use in the access to Culture and cultural resources.
From a Living lab perspective, this experience allowed Issy Media to identify the barriers and issues that occurred when running the experiment in order to improve future cross-border Living Lab experiments. 4.1.3
Transfer focus
Cf. 4.1.1 and 5.1 4.1.4
Research Set-up and Data Collection Methods
Quantitative data was collected from Mobexplore and Community Reporters’ websites.
4.2 Antwerp M HKA Name of the experiment: M HKA 3D app test
The Belgian pilot tested the impact and added value of a mobile 3D game application on youngsters’ museum perceptions and experience in a museum setting. ICT PSP APOLLON
17
Final Version
Apollon – D.5.5 Evaluation Report 4.2.1
•
• •
4.2.2
Partners involved and their role
IBBT o Knowledge of game design o The game concept o Graphic design o Cross-platform development of overall application framework o App-based support of QR code scanning o Back-end Drupal programming o API definition and implementation o Living lab testing methodology Virdual o 3D virtual world design and implementation M HKA (Antwerp museum of contemporary art “Museum van Hedendaagse Kunst Antwerpen”) o Knowledge of what works in interaction with young museum visitors o Knowledge of the academic literature on how museum visitors approach art works o Access to museum for application testing in Living lab environment o Content on works of art, custom made for the target audience (youth between 14 and 24 years)
Overall purpose of the experiment
Test the first version of the prototype, called M HKA PP. Provide input to the development of the second version of the prototype. 4.2.3
Transfer focus
Cf. 4.2.1 4.2.4
Research Set-up and Data Collection Methods
Observation protocol, focus groups, transaction log analysis.
4.3 Manchester Central Library refurbishment (unachieved) Manchester Central Library closed to the public in June 2010 for three years of refurbishment and renovation.
As there was a need to keep citizens informed of the progress of the refurbishment work, it was thought that this project would be a good place to carry out an experiment of Peoples Voice Media's Community Reporters model as a way for local government to engage with citizens in new way.
Partners involved included Manchester Digital Development Agency (MDDA) / Manchester City Council and Peoples Voice Media. Co-partners included Laing O'Rourke (the company contracted do the renovation work on Central Library) and Manchester Library and Information Service. ICT PSP APOLLON
18
Final Version
Apollon – D.5.5 Evaluation Report MDDA facilitated meetings between Laing O'Rourke and Peoples Voice Media to enable Community Reporters to go into the Central Library construction site and report on the progress of the work.
Community Reporters went into the site in August 2011 and published to the Community Reporters website, http://communityreporter.co.uk/tags/central-library
Based on the work done so far during this experiment, the relationship between Peoples Voice Media and Laing O'Rourke is planned to continue after the APOLLON project. This work was also useful to Peoples Voice Media as a comparison of the work done to set up Community Reporters in Issy-les-Moulineaux.
It was also originally intended to carry out an experiment of Navidis's Urabdeus application using the area around Manchester Central Library using a 3D city model of Manchester. This planned experiment did not take place, mainly because processes for accessing 3D city model data was problematic and slow, which caused delays in starting the experiment.
4.4 Manchester Art Gallery Manchester Art Galley Name of the experiment: MuseUs app tests in Manchester Art Gallery 4.4.1
Shortly describe the experiment
MuseUs (http://museus.be/) is an iPhone / iPod Touch app developed by IBBT. MuseUs allows users to curate their own exhibitions of works in cultural spaces (libraries, museums, galleries) by matching ‘statements’ (e.g. phrases such as 'The subject of this painting seems more about real life than the other three.') with 'works' (e.g. a painting in a gallery) by scanning a QR code with their smartphone. As users complete their exhibition, it is displayed on a screen in a space - such as a gallery - for others to see.
IBBT had already carried out a test of the MuseUs app in a museum in Antwerp and wanted to do tests in other APOLLON partner cities to help develop the app, get feedback from users and assess how useful the app would be in other cultural spaces.
Manchester Art Gallery has an important collection of Pre-Raphaelite paintings that were bought on behalf of Manchester’s citizens by Victorians. The Gallery embarked on a year-long project called the Pre-Raphaelite Experiment (http://www.thepreraphaeliteexperiment.org/) to find our what, if anything, these paintings mean to Manchester's residents and visitors. A gallery space in Manchester Art Gallery was given over to four paintings from the collection and activities with general visitors, local groups, families and schools were carried out to respond to and reinterpret the paintings. ICT PSP APOLLON
19
Final Version
Apollon – D.5.5 Evaluation Report Following initial discussions between MDDA, IBBT and Manchester Galleries, the Pre-Raphaelite Experiment gallery space was chosen as a place to test MuseUs as:
- The gallery space was already given over for experimentation of new ways of engaging with visitors, so constraints on what equipment could be installed in the space for the MuseUs tests were not as restrictive as in the rest of the gallery building.
- MDDA had recently installed high-speed WiFi in the gallery, so internet connectivity would not be an issue during the testing of an iPhone app that relies on internet access for its operation. - Manchester Galleries has an existing visitor / contacts list (via email list, postal mailing list, website, Twitter, etc.) that could be used to put calls out for testers.
- Manchester Galleries were already running a number of events in the PreRaphaelite Experiment gallery which the testing sessions could be attached to.
Work was then carried out by MDDA, IBBT and Manchester Galleries to set up and run the experiment, including: - Agreeing dates to run the testing sessions.
- MDDA and Manchester Galleries putting out a call for participants on the gallery website, Twitter, etc.
- MDDA facilitating a face-to-face meeting between Galleries and IBBT staff, so that IBBT could demonstrate MuseUs, sees the gallery space where the experiment would take place, and answer Galleries' questions directly.
- MDDA working with Galleries and IBBT to clarify and obtain the material needed to by IBBT in order to prepare the MuseUs app for the testing session. This included obtaining image files (JPGs) of the paintings in the gallery, English language text about the paintings, and a list of English language short texts ('assignments'). - MDDA working with IBBT to develop and improve the English language localisation of the MuseUs app interfaces.
- IBBT developing the MuseUs app and server software to support the experiment. - IBBT providing QR codes for the paintings in the gallery space, which were then printed out by MDDA into laminated sheets and attached the paintings in the gallery space.
- IBBT training MDDA staff in how to use MuseUs app and the server administration software so that MDDA could run testing sessions independent of IBBT. - IBBT and MDDA providing the equipment to use during the testing sessions (installing and testing the MuseUs app on iPod Touches for use by users, installing and testing the 'exhibition display' software on an iMac to display users exhibitions). - MDDA and IBBT setting up the gallery space ready for use during the MuseUs app testing sessions.
- MDDA working with IBBT to develop an English language version of the evaluation questionnaire that would be used during the testing sessions. ICT PSP APOLLON
20
Final Version
Apollon – D.5.5 Evaluation Report - MDDA running the testing sessions, including briefing users on how the test would work, handing out evaluation forms for users to complete, answering users’ questions, and gathering evaluation sheets. - MDDA compiling the information from the evaluation sheets from the testing sessions to IBBT. Two testing sessions of MuseUs in Manchester Art Gallery took place, one on 18 November 2011 and the other on 21 December 2011.
4.4.2
Partners involved and their role
Describe who has been involved in the experiment and what role the partners have had Manchester Digital Development Agency (MDDA) / Manchester City Council (UK) MDDA is part of Manchester City Council and carries out a range of technologyfocused innovation projects on Smart Cities, Open Data and Living Labs.
MDDA acted as the Living Lab in the experiment, facilitating and organising, the testing sessions, liaising with partners and co-partners and acting as 'local' representative of the SME. 4.4.2.1 Manchester Galleries (UK) Manchester Galleries forms part of Manchester City Council’s Cultural Directorate, operates Manchester Art Gallery and the Gallery of Costume, Platt Hall and cares for the city’s internationally renowned and Designated collections of fine art, decorative art and costume. Manchester Galleries provided the 'scenario' where the experiment could take place, worked with IBBT and MDDA to generate materials required for the workshop, and worked to publicise the testing sessions. 4.4.2.2 IBBT (Belgium) IBBT (Interdisciplinary Institute for Broadband Technology) is an independent research institute founded by the Flemish government to stimulate ICT innovation. The IBBT team offers companies and organizations active support in research and development. It brings together companies, authorities, and non-profit organizations to join forces on research projects. Both technical and non-technical issues are addressed within each of these projects. ICT PSP APOLLON
21
Final Version
Apollon – D.5.5 Evaluation Report IBBT were the SME in the experiment and provided the MuseUs app as the product to be tested. 4.4.3
Overall purpose of the experiment
The purpose of the experiment was to run user testing sessions of IBBT's MuseUs app in a UK cultural institution in order to:
- Provide feedback to IBBT on the MuseUs concept from users and cultural institutions. - Provide feedback on the current version of the MuseUs app in order to support future technical development of the app. - Clarify work required to develop an English language interface for the MuseUs app.
- Expose IBBT to the experience of working with a cultural institution in another country.
- Expose Manchester Galleries to the experience of working with an SME from another country.
- Help MDDA identify the barriers and issues that occurred when running the experiment in order to improve future cross-border Living Lab experiments. 4.4.4
Transfer focus
What has been transferred during the experiment?
Data (images, text, metadata) about the artworks in Manchester Art Gallery was transferred from Manchester Galleries to IBBT, and Galleries provided the exhibition space for the experiments.
IBBT provided the MuseUs app and related server software, technical support during the running of the experiments and some iPod Touches to use during the experiments.
MDDA provided some iPod Touches and display equipment to use during the experiments, and provided high speed WiFi connectivity in the gallery space. 4.4.5
Research Set-up and Data Collection Methods
Describe the methods that have been used to collect data in the experiment and how these methods have been applied Data collected included:
ICT PSP APOLLON
22
Final Version
Apollon – D.5.5 Evaluation Report 1. Evaluation of the experience of using the MuseUs app by users during the testing sessions. An evaluation questionnaire was developed by IBBT, who worked with MDDA to develop an English language version. (A copy of the questionnaire is attached.)
Users who took part in the testing sessions were asked to complete the questionnaire.
The completed questionnaires were scanned as PDFs for archiving, and the information from the questionnaires was compiled by MDDA into a spreadsheet (for the quantitative information) and text documents (for the qualitative information) which were then provided to IBBT. 2. Recording of comments by users at the testing sessions.
MDDA kept notes of comments and actions carried out by users at the testing sessions, where the information could not be recorded as part of the evaluation questionnaires. (For example, one user was seen to touch the screen of an iMac computer, expecting the interface to respond as if the screen were an iPad.)
These notes were forwarded to IBBT and Manchester Galleries for reference, and to support and future work. 3. Recording details of web / social media activity around the calls for participants.
Website analytics (such as Google Analytics http://www.google.com/analytics/) and open source web apps (such as ThinkUp App http://thinkupapp.com/) were used to record details of web and social media activity around calls for participants. In particular, ThinkUp was of use as it enabled the gathering of statistics of number Twitter users were reached.
4.5 French Museum of Playing Cards (Issy-les-Moulineaux) The experiment carried out at the French Museum of Playing Cards in Issy-lesMoulineaux tested the reactions of visitors on the use of the MuseUs mobile application.
The experiment was carried out in the framework of the “Alice in playing cards’ land” exhibition organized by the museum from December 2011 throughout March 2012. Users will be able to test MuseUs on iPods and iPads which are lent by the museum. 4.5.1
Partners involved and their role
Describe who has been involved in the experiment and what role the partners have had
ICT PSP APOLLON
23
Final Version
Apollon – D.5.5 Evaluation Report 4.5.1.1 Issy Media (FR) Issy Media acted as the Living Lab manager, facilitating and organizing the testing session, organizing the training session with the museum staff, liaising with partners and co-partners and acting as 'local' representative of the SME. Issy Media also reviewed the content, language and design of the French version of the application. 4.5.1.2 French Museum of Playing Cards (FR) The Museum provided the 'setting' where the experiment could take place, worked with IBBT and Issy Media to generate the content required for the French version of the application and worked with Issy Media to set up the testing session. 4.5.1.3 IBBT (BE) IBBT were the SME in the experiment and provided the MuseUs application as the product to be tested. 4.5.1.4 UP8 (FR) UP8 was in charge of user observation and evaluation. UP8 also translated the evaluation survey created by IBBT which was used in the Belgian and British experiments and worked with Issy Media to adapt it for the French audience. 4.5.1.5 Overall purpose of the experiment The overall purpose of the experiment was to test the third version of the MuseUs prototype in a cultural institution.
Among the specific purposes of the experiment we can include the need to test the impact of a pan-european application in a French environment, after two successful testing sessions in Belgium and England. On a more local level, the purpose was to expose the Museum of Playing Cards to the experience of working with an SME from another country. 4.5.2
Research Set-up and Data Collection Methods
1. Evaluation of the experience by users during the testing sessions.
For this experiment the same evaluation questionnaire developed by IBBT and used in the other two MuseUs experiments was utilized. UP8 translated it in French and worked with Issy Media to adapt it for the specificities of the French experiment. (A copy of the questionnaire is attached). Users who took part in the testing sessions were asked to complete the questionnaire. ICT PSP APOLLON
24
Final Version
Apollon – D.5.5 Evaluation Report The completed questionnaires were collected and analyzed by UP8. Given that the experiment will last throughout the whole “Alice in playing cards’ land” exhibition (until March 11th 2012), a set of evaluation questionnaires was left at the museum for users to complete after testing the application. 2. User observation during the testing sessions.
UP8 took notes of the comments and actions carried out by users at the testing sessions, where the information could not be recorded as part of the evaluation questionnaires. These notes will be forwarded to IBBT and the Museum of Playing Cards at the end of the exhibition, for reference and to support future work.
ICT PSP APOLLON
25
Final Version
Apollon – D.5.5 Evaluation Report
5 Qualitative Evaluation of the Pilot as a whole WP5 pilot succeeded in its main objectives: -
Pilot the integration of eMedia technologies;
-
Involve users in testing 3D eMedia technologies, context aware technologies and citizen media technologies;
-
Conduct at least one cross-border experiment involving the transfer of one technology in each of the consortium countries;
During the pilot a four experiments were carried out: two in France, one in the UK, one in Belgium. Each experiment combined at least two cross-border technologies. One SME (Virdual) participated to two experiments; another (Airgraffiti) participated to three experiments. An average number of 30 participants was directly involved in each experiment evaluation activity. A greater number participated in later activities (focus groups, remote usage, …).
In the following chapters we describe the results obtained by the pilot under different respects.
5.1 Technology transfer and integration One of the challenges of the Pilot was to pilot the integration of SMEs which profiles were quite varied:
- Two of the four technologies (Virdual and Navidis) were 3D media technologies which product was not easily transferrable abroad without investing a considerable amount of time and money for ad hoc development; - Only one technology (Airgraffiti) had an Application Programming Interface (API) allowing third-party applications to interact with its core technology.
- One of the participants (People Voice Media) didn’t develop a proprietary technology but was focused on methods and tools to use for eParticipation initiatives.
The pilot showed that it was easier for startups and organizations based on knowledge transfer to adapt their technologies to cross-border testing, while it was harder for SMEs that had to develop specific software to ensure cross-compatibility as no budget was attributed to this task. At the same time, working with start-ups with solutions that are still under development also implies some risks such as having to delay an experiment or working with an incomplete solution. However, all partners could participate to technology and knowledge transfer activities.
The following tables give an overview of the cross-border activities (technology and knowledge transfer) that were developed during the WP5 experiments ISSY Fort Pilot
ICT PSP APOLLON
Technology
26
Knowledge
Final Version
Apollon – D.5.5 Evaluation Report Navidis
Navidis developed a script embedded in Urbadeus in order to allow external partners to embed media in the platform following a data exchange protocol.
Knowledge of Urban game design coming from previous experiments.
This protocol was developed to allow Airgraffiti to send geolocalized pictures and texts to Urbadeus. Since Airgraffiti wasn’t ready at the moment of the pilot and was replaced by a local partner “Mobexplore”.
Virdual
People Voice Media
IBBT MobExplore (supporting) ICT PSP APOLLON
This protocol worked perfectly and can now be reused by other partners. Virdual 3D reconstruction of the Issy Fort was used as the beginning of the trail.
Game design, historical material to build the trail
Wordpress, flip cameras, youtube, twitter for community reporters.
Training for Issy Media on how to build a Community Reporter program.
No technology integration between Virdual and Navidis and Airgraffiti was implemented
Media produced by community reporters and citizens were geolocalized in Urbadeus
Mobexplore joined the project to replace Airgraffiti on the first Issy 27
Co-design of the scenario, technical needs definition Knowledge of Urban game design (core business)
Final Version
Apollon – D.5.5 Evaluation Report experiment.
ISSY
Mobexplore developed a feature allowing it to take geolocalized pictures and send them to Urbadeus. Communication on classic and digital media
Identification and coordination of local stakeholders Coordination of the experiment
Setting up the Community reporters program
LUTIN (UP8)
-
Museum of Playing Cards (no partner)
Historical materials
Antwerp Pilot
Technology
Knowledge
Navidis
-
-
Virdual
People Voice Media IBBT
MKKH A Museum ICT PSP APOLLON
Living Lab methodology
3d Environment interacting with Museus to host snapshots created with Museus. Shiva to iOS development -
QR codes mobile application AirgraffitiMuseus
Cross-platform design for connecting MuseUs and Virdual applications Exhibition
28
Scenario definition -
Previous knowledge on how to build 3d Galleries
-
Distributed application development
Groups animation Final Version
Apollon – D.5.5 Evaluation Report
Manchester Central Library (prepared but not experimented)
Technology
Navidis Virdual
People Voice Media IBBT
MDDA
Knowledge
Knowledge on how to build a 3d model of a city Social Media for community reporters
QR codes mobile application AirgraffitiMuseus
Conducting community reportages
Organization of expert meetings
Identification of local partners that could help Navidis developing the 3D Model
Manchester Galleries
Technology
Knowledge
Navidis Virdual
-
-
-
IBBT
QR codes mobile application AirgraffitiMuseus
Living Lab methodology
People Voice Media
MDDA
-
-
Experts meetings
Localization of validation material Data collection and
ICT PSP APOLLON
29
Final Version
Apollon – D.5.5 Evaluation Report evaluation protocol
Manchester Galleries
Exhibition
Facilitation with groups of visitors
Issy Museum of Playing Cards
Technology
Knowledge
Navidis
-
-
Virdual
-
IBBT
QR codes mobile application AirgraffitiMuseus
People Voice Media
Museum of Playing cards ISSY
LUTIN (UP8)
5.1.1
-
Exhibit materials
Web and printed communication materials Mobile devices (iPod and iPad) for visitors
Community reporters methodology
Organization of visitors groups
Identification and coordination of local stakeholders Localization of validation materials
Living Lab testing methodology
Shifts from planned work
Most relevant shifts from what was previously planned concern the Navidis, the Virdual and the Airgraffiti technologies: Navidis participated to a technology integration experiment in France but couldn’t run an experiment in Manchester.
It was also originally intended to carry out an experiment of Navadis's Urabdeus application using the area around Manchester Central Library using a 3D city model of Manchester. This planned experiment did not take place, mainly because processes for accessing 3D city model data was problematic and slow, which caused delays in starting the experiment. The main reasons for this are the following: ICT PSP APOLLON
30
Final Version
Apollon – D.5.5 Evaluation Report -
The cost of developing a 3D model of Manchester Central Library was too high and was not covered by the project budget. Manchester Living Lab found a local partner that could provide a 3D model but the negotiations took too long to be useful in time for the experiment; Due to construction works in the Manchester Library the access to the building was forbidden for a long time and delayed the implementation of the experiment ;
In the case pf Virdual, we could experiment the integration of a 3D art gallery technology in the Antwerp museum as planned, but the SME didn’t want to engage in further experiments involving the integration Real And More 3D technology in the Manchester pilot or in the Issy Museum of Playing cards pilot. The main reason for this was a divergence in the goals of Virdual and IBBT (MuseUs). MuseUs had an academic and research approach and wanted to carry out further cross-border experiments before exploring the business opportunities for the application. Virdual on its side judged that before investing further effort in developing ad hoc solutions for other museums they needed to explore potential business models for the application (namely a revenue sharing model with museums). Since this wasn’t foreseen in the new experiments, Virdual decided not to engage in the new pilots because they couldn’t evaluate the ROI for these activities. Concerning Airgraffiti, the main difference from the work planned in D 5.4 is the fact that Airgraffiti wasn’t used in the first French experiment that took place midMarch. Airgraffiti planned to publish a first Mobile Web version by mid-February but the local partners decided not to take the risk of working with a solution that they couldn’t extensively test beforehand since it was under development during the definition of the French experiment (November 2010-January 2011). For this reason, the decision was taken to momentarily replace Airgraffiti with a local partner: Mobexplore. The two technologies being quite similar, we didn’t need to change the scenario. The iOS version of Airgraffiti was tested in Issy during the second French experiment that took place in January 2012 after participating to the Antwerp and Manchester experiments. All the other technology integration experiments took place as planned.
5.2 Users involvement All the experiments involved the participation of users as testers of the proposed technologies and as an invaluable source of feedbacks for SMEs.
As a whole, the experiments reached a total of 120 users (an average of 30 users per experiment) directly involved in testing the technologies through focus groups, observations, and questionnaires. Moreover, since the launch of the Issy experiment, a total of 239 users have been using the application for exploring Issy after the kick off of the experiment without directly participating to user research activities. ICT PSP APOLLON
31
Final Version
Apollon – D.5.5 Evaluation Report
5.3 Cross-border Collaboration Experiences & Evaluation WP5 partners are positive about the collaboration experience. All partners could benefit from the cross-border collaboration to improve their technology and better define their solutions. Beyond the results of the pilot, the cross-border collaboration allowed Living Labs and SMEs to start building a European ecosystem that is a fertile ground for future projects. The majority of the partners are already working on new European projects with partners they met through Apollon. 5.3.1
Scenario design and collaboration management
As described in 3.4 the co-design phase is what took the most effort in all the experiments. This involved many aspects from the definition of the scenario to identifying the relevant stakeholder to associate to the project. The process is described “Design of the Pilot”. As described in 3.4 this was done mainly through physical meetings in each partner’s country. The scenario design was co-conceived by Living Labs and SMEs. One of the Living Labs partner (UP8) had the role of making sure that the overall coherence of the pilot was ensured and that each experiment included at least the transfer of a technology or knowledge cross-border. This supervising role seems important for future projects. For future experiments it seems important to have one partner (or someone inside the coordinating living lab) with the precise task of 'project manager' for the experiment making sure that all the partners . The LL should not step back after the SME and local participants have been introduced and a scenario identified, but work with all participants to ensure that the experiment is carried out successfully. The Living Lab must ensure that communication keeps flowing between the SME and other participants in the experiment to ensure all participants are aware of their roles and responsibilities during the experiment.
Example of the steps involved in scenario co-design: the case of MDDA Manchester Galleries (achieved) Manchester Galleries has an existing project called Decoding Art (http://www.manchestergalleries.org/decodingart/) which uses QR codes attached to public art works in the city centre to provide citizens and visitors with information about the art works through their smartphones.
Manchester Galleries wanted to continue experimenting with using QR codes and so it was thought that carrying out an experiment using IBBT's MuseUs (formerly known as Air Graffiti) iPhone application would be useful. Partners involved included Manchester Digital Development Agency (MDDA) / ICT PSP APOLLON
32
Final Version
Apollon – D.5.5 Evaluation Report Manchester City Council, and IBBT. Co-partners included
MDDA carried out initial meetings and presentations to Manchester Galleries to introduce them to Living Lab concepts and identify an appropriate scenario in which an experiment involving IBBT's MuseUs app could take place. MDDA facilitated initial face-to-face meetings between Manchester Galleries and IBBT, and acted as the facilitator between Galleries and IBBT as the experiment was being defined. Manchester Galleries decided to use the "Pre-Raphaelite Experiment" exhibition (http://www.thepreraphaeliteexperiment.org/) as a place to test the MuseUs app in Manchester Art Gallery. That exhibition was already being used as a venue in which to test different ways of allowing the public to interact with artworks in the gallery, which made carrying out a test of the MuseUs iPhone app a good choice.
MDDA acted as the liaison between Manchester Galleries and IBBT to define the experiment based on both Galleries and IBBT's needs, set up the exhibition space ready to be used for the experiment, publicised the testing sessions with users, worked with IBBT to prepare the equipment to be used during the testing sessions, provided some equipment for testers to use (iPod Touches, iMac, high speed WiFi connectivity in the Gallery) facilitated the testing sessions and gathered and collated the evaluation materials from the testing sessions. Manchester Galleries donated the exhibition space and materials required for IBBT to set up MuseUs to work in the gallery space. IBBT prepared the MuseUs app ready for use, attended one of the testing sessions, provided guidance on the evaluation materials and ensured that the testing sessions ran smoothly by ensuring that the software was working. Greater Manchester County Records Office (unachieved) Greater Manchester County Records Office (GMCRO) has recently started to use social media tools like Twitter and sites like Flickr to explore how to engage with citizens who would not normally access the material in the archives.
The GMCRO were funded by the (UK) Museums, Libraries and Archives Council to explore how to use social media tools to engage with the public. GMCRO approached MDDA for assistance with this work and it was thought that this would be an opportunity to test Peoples Voice Media's Community Reporters model and IBBT's MuseUs (formerly Air Graffiti) application.
Partners involved were Manchester Digital Development Agency (MDDA) / Manchester City Council, Peoples Voice Media. Co-partners were Greater Manchester County Records Office and Manchester Digital Laboratory (MadLab). The Manchester Flickr group were approached to take part in a day with Archives to take new photographs of old scenes of a street in city centre Manchester, around which an experiment could be based. MDDA carried out initial meetings and presentations to GMCRO to introduce them to Living Lab concepts and identify an appropriate scenario in which an experiment involving IBBT's MuseUs app could take place (the Manchester Flickr group enICT PSP APOLLON
33
Final Version
Apollon – D.5.5 Evaluation Report gagement). MDDA facilitated initial discussions between GMCRO and Peoples Voice Media to carry out the Manchester Flickr group day, and assisted with the logistics of arranging the day.
GMCRO placed images from the Manchester local images collection (http://images.manchester.gov.uk/) on the website Flickr and used Twitter and a blog on WordPress.com to publicize work.
Peoples Voice Media were involved in recording the activity on the day and citizens views on this work: http://manchesterarchiveplus.wordpress.com/2011/02/01/flickr-tib-streetrevisited/. An experiment using IBBT's MuseUs app was planned to take place whereby QR codes in the street could be used to match that place with historical photographs in the archives, but cuts in local government funding in the UK meant a reduction in the available staff in Manchester City Council to support the experiment, so it did not take place.
5.3.2
Communication
The communication among WP5 partners consisted mainly of e-mails and face-toface meetings.
Communication needs to be subtle and as rich as possible. For example communication over email, which is a relatively poor communication medium, risks hindering the harmony in the cross-border team? Indeed, different national cultures entail different ways of communicating. In addition, email is a medium that can easily lead to an interpretation by the receiver which was not intended by the sender. Such communication-related risks can be mitigated by communicating more face-to-face, using tele-conferencing, over voip or over the phone. 5.3.3
Distributed Application management
Development in which different partners need to rely on each other’s input runs the risk of slowing down the overall development time. The implementation of methods like those described in the “Distributed Application Development” is strongly encouraged.
ICT PSP APOLLON
34
Final Version
Apollon – D.5.5 Evaluation Report
5.4 Benefits All the participating Living labs and SMEs had benefits from participating to the pilot.
AirGraffiti (BE)
Navidis (FR)
Access to new markets
Developed new solutions
Test of technology in different cultural settings
Increased understanding of different stakeholders perspectives
Learning and sharing good practices
Airgraffiti evolved in a new product called MuseUs which is dedicated to museums and cultural institutions
Yes, during the pilot Airgraffiti worked with a museum in Antwerp and in Issy Les Moulineaux and discovered a new market. It changed its name to MuseUs, thus becoming an application particularly dedicated to Museums and cultural organizations.
Yes in England and Franc e
Yes, understanding the difference between a Research startup and SME priorities.
Shared good practices about distributed application development (see method description)
Got a better understanding of the local administration and endusers’ needs when developing new services and applications.
Shared its knowledge about the design of urban games.
Yes, potentially the English market, but no concrete experiment has been carried out. However, discussions are underway with a
Yes, opened its communication protocol to Mobexplore in order to be able to receive and embed external media. This protocol is reusable for other applications
ICT PSP APOLLON
Attracting EU funding
Found other partners that are willing to use and test the software on its way to final market deployployment.
Inreased understadning of the prospective business model. Only in Franc e
35
Had a chance to reduce the costs of developing a 3d model of Manchester by reusing an existing model from a local company but the work has been delayed due to IPR issues and
Enhancing broader cooperation
Yes: EU funded Smart Cities project: EPIC (European Platform for Intelligent Cities). This new project extended the collaboration with Issy Media and other local stake-
Yes: discovered a new competitor in the Manchester area and redefined its offer. Participated to many EU conferences
Final Version
Apollon – D.5.5 Evaluation Report British SME.
People Voice Media (UK)
Virdual
Developed a partnership model for its Community Reporters program
(FR)
Mobexplore (supporting partner, FR joined in 2011)
ISSY Living Lab
construction works in the Manchester Central Library.
Yes, a “train the trainers” program for foreign groups. Developed its first crossborder community
Yes in Franc e
Yes, needs of a foreign user group
Yes: exported a 3D environment developed in Shiva to iOS mobile operating System
Yes in Belgium
Yes, cultural institutions like museums
Only in Franc e
Yes, local authority wanting a “permanent” solution
Added a “pictures” feature to send picture to external platforms like Navidis
Crossborder testing
Consolidated its Living Lab activity, made it more visible and joined a new Living Lab on Augmented Reality
ICT PSP APOLLON
Yes, shared its knowledge and formalized a protocol for training foreign user groups (see “community reporters” method)
Working on a new project called ISABEL
Shared its knowledge about the design of urban games.
Shared its knowledge about the Issy Ecosystem and the organization of activities with citizens
36
holders from Issy, created new collaboration opportunities with EPIC project partners and also offered the opportunity to work with a key IT player, IBM.
During Apollon, ISSY started a new European project on smart cities (EPIC)
and was able to promote its work internationally
Extended its Community Reporters network
Mobexplore is now cooperating with the City of IssylesMouline aux
Participated to international activities as a Living Lab
Final Version
Apollon – D.5.5 Evaluation Report
MDDA Living Lab
Crossborder testing
Consolidated its Living Lab activity and promoted the Living Lab approach to local partners
Contributed to the creation of expert groups to evaluate partners technologies
LUTIN (UP8 Living Lab)
Crossborder testing
Consolidated its activity as a Living Lab
Shared Living Lab methodology
5.4.1
Became partner of a new Living Lab on Augmented Learning
with other Apollon partners: MDDA and Enoll
During Apollon, MDDA started a new European project on smart cities (EPIC) with other Apollon partners: Issy and Enoll
Thanks to the experience acquired in crossborder testing through Apollon LUTIN was able to join a new European project dedicated to innovation in Science Education: PriSciNet
Thanks to LUTIN MDDA was introduced to Silicon Sentier: a SMEs and Startup association in Paris Became more engaged in the Enoll community (GNSS Living Lab prize, EsoceNET forum, …)
Test of technology in different cultural settings
Almost all partners could participate to a cross-border experiment that took place outside their home country. AirGraffiti (MuseUs) could experiment its technology in Antwerp (home country), Manchester and Issy. On the other side, Navidis planned to participate to the Manchester City Galleries experiment but could only participate to the French experiment. This example shows that this kind of cross-border testing is more profitable for “light” technologies using widespread technologies (iOS, Google Maps) with a startup approach that can quickly adapt to different geographical and cultural settings. ICT PSP APOLLON
37
Final Version
Apollon – D.5.5 Evaluation Report 5.4.2
Increased understanding of different stakeholders perspectives
The pilot involved the participation of partners with different profiles: SMEs, Living Labs, local administrations, cultural institutions, Startups, NGOs. All partners had to accept some compromise in order to harmonize the expectations and goals of each other. This was a very useful experience for future projects involving a consortium of European partners. 5.4.3
Increasing income or status
No partner had direct increases in business thanks exclusively to Apollon, but everyone could benefit from a positive impact on their status as a European SME or Living Lab. For Living Labs, the main benefit was the opportunity to make their activity visible at both a local and European level. For SMEs, the main benefit was to be able to enter a wider market and promote their technologies as European validated services. 5.4.4
Learning and sharing good practices
Besides the experiments involving technology integration, the project was a good opportunity to share good practices at different levels: Living Lab methodology, application development, data collection, users involvement and community building. For all partners this was the first time they could share their experience as a Living Lab and the first time they questioned their Living Lab role as part of a wider European ecosystem. 5.4.5
Attracting EU funding
All partners except Virdual are today engaged in European projects with partners they met directly or indirectly through Apollon. 5.4.6
Enhancing broader cooperation
WP5 partners see Apollon as a ground for further collaborations in the Living Labs ecosystem. Apollon succeeded in showing that inside the Living Lab network there are partners that are mature and skilled enough to develop international projects with SMEs. 5.4.7
Iterating a test in different cultural settings
Within the framework of the Apollon project, IBBT developed and conducted research on the impact of a mobile application to engage museum visitors. The first version of the application was the result of a close cooperation between IBBT-iLab.o (who delivered the Air Graffiti technology) and the French cross media production company Virdual (who provided the 3D technology), in close consultation with the Antwerp museum of contemporary art “Museum van Hedendaagse Kunst Antwerpen” (M HKA, Belgium) who provided the content. In close consultation with M HKA staff members, IBBT researchers facilitated the implementation and user assessment of the game on site. ICT PSP APOLLON
38
Final Version
Apollon – D.5.5 Evaluation Report A second version of the prototype, which was renamed to MuseUs, was tested in the Manchester Art Gallery, in collaboration with MDDA. This version of the prototype was developed by IBBT only, without the help of Virdual.
The primary aim of this research consisted of examining the experiences and practices related to the application. In particular, we intended to assess the impact of the application on visitor’s movements, art-viewing habits, interaction patterns, learning behaviour, etc... In a wider perspective, this research explored the impact and added value of mobile application on museum engagement.
Besides more academic-oriented research goals, it was investigated how a crossborder pilot could be carried out. An important aspect of cross-border testing was the language issue. The software and all the related aspects (documentation, surveys …) needed to be translated to every country in which an experiment is conducted. This generated a significant amount of work that needed to be done by the local partner. Indeed, local language evolves more rapidly than what people outside the country borders can know. An example of this is the activity that in Belgium and England would be called “the scan a QR code with a mobile device”. In France, there was already a short expression for this: “flasher le code”. This shows that translations need to be done by native speakers of the language or people who are very active in the target country. Beside the language aspect, no main cultural issues were encountered during the M HKA PP/MuseUs trials. This is not surprising, as France and the UK are neighboring countries to Belgium. As these countries are so close, the cultural differences are limited. In addition, the researchers working of the M HKA PP/MuseUs speak both English and French, which further reduces the cultural gap.
5.5 Challenges
5.5.1
Harmonization of partners’ objectives
Given the different profiles of SMEs partners, the harmonization of partner’s objectives was a challenge we needed to solve at the beginning of the project. SMEs like Virdual and Navidis expected to find business opportunities through Apollon; NGOs like People Voice Media expected an international validation of their activity; Startups like Air Graffiti and the supporting partner Mobexplore wanted to get user traction and visibility through Apollon. The way in which we managed to harmonize all partners’ objectives was through the process of co-designing the scenarios for each experiment in a way that the goals of each partner were addressed if not completely matched. The process of harmonizing these goals involved some level of compromise for each partner. Living Labs willing to realize cross-border activities involving multiple partners will need to take into account these different goals. One solution to reduce the complexity of managing this issue could be to start from one SME goals and try to find partners that have compatible expectations. ICT PSP APOLLON
39
Final Version
Apollon – D.5.5 Evaluation Report 5.5.2
Sustainability
Though APOLLON gave the opportunity for Living Labs to demonstrate how Living Lab processes work, a main challenge following APOLLON is how to turn this into a marketable service that can be sold to local SMEs, be well-resourced and begin to be known as a useful service to local SMEs, and not the work of a time-limited, grantassisted project.
On the SMEs side, no partners had direct business opportunity (outside new EU funding) through Apollon, but they could benefit from this experience to structure their offer for international customers and partners and start a process of internationalization that they will continue to develop after the end of the project.
What we are looking forward now is to collaborate with Enoll in the definition and construction of a platform in which Living Labs and SMEs could meet, match their needs and skills and exchange services. Initiatives such as the GNSS Living Lab prize (to which both Navidis and LUTIN took part) are the first good examples of how this could work and should be encouraged. 5.5.3
Knowledge sharing
In Apollon, Living Labs and SMEs shared knowledge in a rather unstructured way answering contextual needs. While this was effective for the purpose of the pilot, the process of sharing knowledge between Living Labs in the same thematic network will need to be structured and managed. Even if this does not need to be a big challenge, (e.g. when a Living Lab agrees to join a thematic network, part of the agreement could be to use a shared wiki to transfer knowledge between members) someone will need to set up and maintain a platform that will facilitate this process as this will speed up the process of finding and reusing existing methods and tools. 5.5.4
Cross-border training
One of the main knowledge sharing activities in WP5 consisted in a two days training on the methods, tools and strategy of building a Community Reporters that members of People Voice Media organized for Issy-les-Moulineaux in Manchester. This was a very rich and productive activity that we would like to be further developed in future cross-border Living Lab activities. Focus on: the agenda for a two days cross-border training for Issy Media by People Voice Media (Manchester) Stage 1: Understanding the model and learning the skills Day One: They were taught the basics of the programme – our approach, the informality, the ICT PSP APOLLON
40
Final Version
Apollon – D.5.5 Evaluation Report peer support and learning, peer review. They were also taught the importance of Editorial Guidelines, getting consent, health and safety for reporters and copyright – these are the elements that make up what we call Community Reporter Best Practice. Next were some skills training around video techniques, interview techniques and how to conduct oneself when out filming in public. That evening, the groups were all set the task of reporting in the community. They were all able to put the skills training into practice and record interviews with members of the public (one member of the group was lacking in confidence in his English so he videoed without interviews). Day Two:
Now the group were taught how to edit using Windows MovieMaker. They had different versions of WMM which flagged up some issues they might face when transferring the training to their own centres. They used their own footage from the night before to edit short films about their experiences. Next came blogging training using Wordpress. Wordpress has the option for numerous languages so the groups were able to use French Wordpress. One member of the group already used blogging software but felt that Wordpress was much more easy to use and had more functionality.
The group then uploaded their videos to the blogs and ended the day with video evaluation of what they learned and how they hoped to take it forward. Comments included: ‘most importantly we learned a methodology’
‘we learned small things that were easy to replicate’
‘the training was very practical so we’re up to doing the same thing in Issy’ Stage 2 : Duplicate the experiment in a different cultural setting
The group returned to Paris and began the process of recruiting in early Spring. We offered remote consultancy via email and Skype. Issy targeted young people as one of the people who completed the training was from a young person’s project in Issy. They felt that the word Community Reporter did not translate well and therefore changed this to ‘Web Reporters’. These Web Reporters went out interviewing members of the public about the multimedia trail that was being deployed across the city. Content can be viewed at their blog here: http://issyreportages.wordpress.com/
And People Voice Media website: http://communityreporter.co.uk/videos/trainedICT PSP APOLLON
41
Final Version
Apollon – D.5.5 Evaluation Report manchester-young-french-community-reporters-report-event-paris
Issy modified the reward system and included the badges (as per UK Reporters) and hats (rather than the T-shirts used in the UK). The use of the QR code on the hats is a improvement that will be implemented in the UK. The French pilot confirmed the value and importance of making Reporters feel like they were part of wider group.
People’s Voice Media had the chance to visit the group in June 2011, to see what content they’d created and to interview them about their experiences. This reinforced the sense of being part of a international group and we were keen to have their content shared on the communityreporter.co.uk web site. One of the Parisian reporters is included in a film about Community Reporters on the about page of communityreporter.co.uk website. http://communityreporter.co.uk/about-us
5.5.5
Reduced budget for travel and development
One of the main challenge for a cross-border pilot aiming at testing the integration of multiple technologies was that the travel budget and the development budget for SMEs was very limited. Travel and accommodation costs to enable SMEs to visit potential partners and scenarios in foreign markers will need to be resourced if cross-border Living Labs are to continue to operate after APOLLON. In the same way, not having a budget for developing specific ad-hoc solutions that would have facilitated the integration of two or more technologies was a big constraint for the pilot. Future projects will need to consider this when defining the budget for crossborder pilots involving technology integration. 5.5.6
Match-making
At the beginning of Apollon started, partners didn’t have a complete knowledge of each other expectations and skills. These were progressively defined during the process of co-designing the scenario for the pilot. As the project evolved it turned out that the consortium couldn’t answer all partner’s expectations according to their core business. For example, a partner like Virdual specialized in the implementation of 3D worlds for cross-media television shows could adapt its technology to the pilot (providing a 3D Gallery for the Antwerp pilot) but couldn’t benefit from a network of Living Labs specialized in the domain of new forms of interactive Television. Future projects will need to make sure to attribute a sufficient time to the match-making phase before beginning the project since this is a fundamental step for making sure that Living Labs and SMEs expectation and goals can be fulfilled. ICT PSP APOLLON
42
Final Version
Apollon – D.5.5 Evaluation Report
5.5.7
Language
The partners and co-partners involved in the all experiment spoke English well, but this may not always be the case in cross-border experiments. Getting access to translation resources can be a challenge for future projects. Translation resources that Living Labs can access (either via support from ENoLL, by reserving budgets or pooling local resources) would need to be made available to support future crossborder experiments.
ICT PSP APOLLON
43
Final Version
Apollon – D.5.5 Evaluation Report
5.6 Methods and Tools
5.6.1
Communication and collaboration tools
Low cost and / or free web and internet based collaboration tools have helped a lot in the set up and running of the experiments, for example:
Dropbox (http://www.dropbox.com) for sharing files without having to email files back and forth. Skype (http://www.skype.com) for video and audio calls.
Google Docs (http://docs.google.com) for collaborative document editing. Vimeo (http://www.vimeo.com) for video sharing.
Delicious (http://www.delicious.com) and Pinboard (http://pinboard.in) for sharing bookmarks.
All the experiments made use of some communication and collaboration tools (especially email, Skype and Dropbox). Though these tools aided the running of the experiment, the use of these tools was taken on as the experiment developed. In future, it would be useful to agree on a range of collaboration tools before the experiment begins so that all participants know what tools to use and what for (e.g. Dropbox to share files). No web-based available project management / 'to-do' list tool was set up for use by the experiment. Though actions were not missed, it usually meant relying on email and individuals project management tools to ensure work was done. In future, it would be useful for the Living Lab and SME to agree on a project management tool to use (such as Basecamp http://basecamphq.com/) before the experiment starts.
There has been some difficulty in using the above tools with some partners (e.g. Manchester Galleries staff,) as their corporate IT infrastructure was very locked down and is not flexible enough to allow staff to set up and use useful tools, such as Dropbox, without clearance by senior managers and IT policy officers. This has caused some slowdowns in working with those staff during the experiments as they were not able to directly access resources that were being freely shared between the Living Lab (MDDA) and the SME (IBBT). 5.6.2
Distributed application development
Described in the Methods and Tools document.
5.6.3
Cross-Border training
Described in the Methods and Tools document. 5.6.4
Observation, data collection and focus groups
Described in the Methods and Tools document ICT PSP APOLLON
44
Final Version
Apollon – D.5.5 Evaluation Report
5.6.5
Knowledge sharing
See 5.5.3 “Knowledge sharing”
5.7 Partners – End-Users User’s participation was very useful in producing information on what to concentrate on for the next development cycles of the applications. Apart from specific usability feedbacks gathered from users, one of the main lessons learned through the pilot concerned the role of new technologies in pilots aiming at enlarging the number of citizens participating to cultural activities. While the fact of using new technological solutions (as 3D and QR codes) has a positive impact in the participation of a wider (and generally younger) population, technologies can have the side effect of excluding users (especially older ones) that don’t have the right tools. This was the case in the first Issy pilot in which the Interactive Trail could only be fully experienced using an iPhone device. However, users without an iPhone could still scan QR codes with a mobile phone equipped with a camera and one of the many QR codes reader applications available. To solve this problem, in the second French pilot, the living lab lent mobile and tablet devices to the users in order to give everyone the chance of participating to the experiment. Users showed a positive attitude towards testing cross-border applications. In the case of Issy web reporters, the fact of being part of an international community (People Voice Media Reuters of the Community) contributed in a positive way the creation and the strengthening of the community.
In all experiments, the work with cultural associations and institutions was very important not only to define users targets but also in inviting visitor groups thus ensuring that a minimum number of participants was going to participate to the experiment and help attracting more users.
5.8 Partners – SME Benefits for SMEs and Living Labs are synthetized in 5.4 5.8.1
Air Graffiti – MuseUs
The cross-border experiment has yielded an interesting application as a consequence of the collaboration between the different parties. The pooling of the experICT PSP APOLLON
45
Final Version
Apollon – D.5.5 Evaluation Report tise of IBBT, Virdual and M HKA resulted in a product that would have been harder to achieve by any one of the involved partners. The cross-border experiment produced interesting lead contacts in the different markets. The cross-border pilots have provided Air Graffiti with access to markets in both Belgium, France and England, which we normally would never have achieved. -
-
-
-
-
-
5.8.2
Air Graffiti gained an insight in the perspectives from different stakeholders that they normally would never have gained, especially from working closely with M HKA.
Working with Virdual gave Air Graffiti access to technology which they normally would not have considered nor developed.
The cross-border trials allowed Air Graffiti to test the application in different cultural settings, with different languages and with different museum types. This made the application more generic and therefore useful in different settings. It made Air Graffiti realize the added value of our simple application: the concept is simple, which increased its applicability to different domains. The cross-border pilots led to a product that Air Graffiti is now developing into a new business.
Because of access though Apollon to Enoll, Air Graffiti is now discussing the possibility to test the application in other European countries, like Spain.
The living lab perspective has allowed the continuous evolution of the software prototype, based on user feedback. In addition, it has allowed for the formulation of insight concerning the way in which the app should be brought to the market, like for example what business model to use.
In the future, the experiments that are still to be conducted, but that Air Graffiti is currently setting up, will allow them to test the application’s concept in different types of museums. This will make the application more generic and therefore widely available. It will also increase the track record of the application and allow Air Graffiti to present results and testimonials in various markets.
Navidis
Navidis main contribution to Apollon was the 3D Urbadeus technology and the development of a data exchange protocol to allow third-party applications to send geolocalized media to Urbadeus. Navidis was a very active partner in Apollon, taking part to the majority of project meetings and dissemination events. Through Apollon Navidis was able to access to the Living Labs network, to reinforce its network of European contacts and to demo its technology (Urbadeus) at a number of European events. Even if no direct business opportunity was developed, Navidis ICT PSP APOLLON
46
Final Version
Apollon – D.5.5 Evaluation Report could benefit from Apollon to improve Urbadeus and to develop its network of European partners. During Apollon, Navidis joined a new European Project on Smart Cities (EPIC) including partners they met through Apollon. 5.8.3
Virdual
Virdual main contribution to Apollon was a mobile version of a 3D virtual gallery environment that museums can use to create virtual exhibitions. Apollon allowed Virdual to become aware of business models problems that arise when working with Cultural Institutions. The main regret for Virdual was that no partner in Apollon was positioned in the cross-media and interactive Television market and couldn’t help them accessing cross-border stakeholders. Virdual is willing to explore the opportunities for building a thematic network of Living Labs in the domain. 5.8.4
People Voice Media
People Voice Media contributed to Apollon with its program Community Reporters and with knowledge on how to set up and animate a community of citizen Web reporters. As a result of the pilot PVM realized that a partner model was more effective to achieve their goal of 10,000 community reporters. This has leaded them to explore alternative income generation models and change our business strategy. PVM started to offer “Social License” as a way forward. This approach encourages other organisations to run the program on their behalf either to add value to their existing service or as an income generating opportunity for themselves. PVM receives a “License fee” allowing to create better long term sustainability.
The Apollon pilot also allowed PVM to develop the “train trainer” program which is something they were thinking about but had not developed. As a result of this experiment PVM piloted the train trainer idea and received very positive feedback. The feedback from the experiment pilot also identified a number of weakness in the product particularly around clarity. These were corrected. PVM did have concerns about language issue with the program but found that these can be overcome through the train trainer program which is carried out in English as the tutor were able to translate these back into their languages.
This was a very important part for PVM in the project. Prior to Apollon they had strict control on the program and delivered it themselves. This was clearly not possible within the Apollon due to the language issue and therefore forced them to review the way they could operate and pilot a hands off approach. This experience was very successful and as a result has changed the way that PVM works with key stakeholders.
ICT PSP APOLLON
47
Final Version
Apollon – D.5.5 Evaluation Report
5.9 Partners – Local Authorities 5.9.1
ISSY MEDIA
For Issy Media, representative of the local authorities and manager of the local Living Lab, the eParticipation experiments have had a two-fold effect: -
-
the creation of two new services for the inhabitants of Issy (end-users of the Living Lab), as well as the improvement of the Living Lab methodology
the involvement of local institutions and local SMEs in foreign activities, which reinforced the links between the Living Lab and its stakeholders.
The dissemination activities undertaken during this period allowed the City and the Living Lab to gain national and international visibility, through the participation and organization of workshops 2, international events 3 and in several national competitions 4.In addition, the participation to APOLLON enabled Issy to increase its Living Lab activities. On another level, the visibility gained thanks to Apollon opened the door to new foreign partnerships and involvement in new EU funded projects in various domains 5, to improve existing city services and to create new ones benefiting the inhabitants of Issy. 5.9.2
Manchester Galleries
The experiments gave Manchester Galleries an opportunity to test new technologies and new forms of eParticipation. By being involved in a local Living Lab experiment, Galleries did not have to spend time purchasing or contracting new equipment or 'trial' software and expertise in or to test a new way or providing information about art works to gallery visitors.
It brought Galleries staff into contact with developers and gave them insight into the development process and the issues developers face when seeking scenarios to test their products in a real world setting. It also prompted discussion about the value of bespoke mobile content, the role of gaming in gallery spaces and how interpretive content could be used in ways not previously considered.
“Culture 2.0”, 24 June 2011, Issy-les-Moulineaux. Imagina, 1-3 February 2011, Monaco 4 Evillementiel (FR), 19 June 2011 – Best Award in the category “Culture” and Grand Prize “Evillementiel” for the interactive trail on the history of the Fort of Issy Festival Fimbacte (FR), 17 October 2011 – Special Distinction of the Jury for the interactive trail on the history of the Fort of Issy. 5 Cloud computing domain, with the EPIC project Open Data domain, with the CITADEL project 2 3
ICT PSP APOLLON
48
Final Version
Apollon – D.5.5 Evaluation Report 5.9.3
ISSY Museum of Playing Cards
The two experiments increased the visibility of the Museum on a local and regional level, offering in the same time the possibility to test beforehand new devices and technologies and experience new forms of citizen Participation.
The Living Lab environment made possible and eased the collaboration with a number of partners, local as well as foreign, which in normal conditions would have been difficult or long to achieve.
5.10
Living Lab Perspective
5.10.1 Some benefits for IBBT Apollon introduced IBBT to the digital business community in Manchester, which could potentially benefit them in the future when finding business partners. • •
•
Enabled them to test their product in a different cultural settings.
Provided them with real user feedback about their app, which could be used to guide the future development of it.
Gave them information of the type of business model to pursue for the application.
5.10.2 Some benefits for MDDA APOLLON helped MDDA raise its profile as a local Living Lab
The cross-border experiments provided MDDA with an opportunity to demonstrate how Living Lab processes work to local stakeholders, and being involved in a Living Lab could benefit them. APOLLON helped to highlight MDDA's relationship with the European Network of Living Labs to local stakeholders and raise the profile of the services provided by ENoLL and cross-border Living Lab networks. 5.10.3 Some benefits for LUTIN (UP8 Living Lab) Apollon has helped LUTIN Living Lab (UP8) to structure its Living Lab activity, to give it more visibility and to become a more active member inside the Living Lab community (participation to Summer Schools, Conferences, GNSS Living Lab prize, etc.). Thanks to the experience gained through Apollon, LUTIN could act as a consultant for the creation of a new Living Lab inside Universcience, Europe’s biggest science museum located in Paris. ICT PSP APOLLON
49
Final Version
Apollon – D.5.5 Evaluation Report Through Apollon, LUTIN promoted its Living Lab methodology and was able to apply it to many different projects that were funded during Apollon. Thanks to APOLLON, LUTIN was able to meet new European partners and to join a new European project on Science Education (Pri-Sci Net). 5.10.4 Benefits for ISSY MEDIA (as a Living Lab) Cf. section 5.9.1
ICT PSP APOLLON
50
Final Version
Apollon – D.5.5 Evaluation Report
6 Quantitative Evaluation of the Experiments Summary of the Quantitative Evaluation of Cross-Border Experiments Key Indicator
Sum:
No of cross-border activities
4
No of technology transfer activities
5
No of intellectual products (methodologies, know-how etc) transferred in the experiment No of Users that has been involved in the experiment No of user involvement activities
No of new ideas that emerged from the cross-border collaboration with users No of implementations of e.g. new functions as a result from the cross-border collaboration with users No of redesign of products, services, processes as a result from the cross-border collaboration with users No of SMEs involved in the experiment No SME engagement activities
No of new international partners for the SMEs involved
No of signed letter of intent between partners and/or customers for the SMEs No of new businesses generated in other countries for the SMEs involved No of new business proposals for the SMEs involved
No of new customers in other countries for the SMEs involved
Did the cross-border collaboration lead to increased turnover for the SMEs involved
Did the cross-border collaboration lead to increased customer retention for the SMEs involved No of Large Enterprises (LE) involved in the experiment No LE engagement activities
No of new international partners for the LEs involved ICT PSP APOLLON
51
4 120 (239 including remote users) 4 4 6 4 4
10 6
N/A N/A 7 0
No No 0 0 0 Final Version
Apollon – D.5.5 Evaluation Report No of signed letter of intent between partners and/or customers for the LEs involved for the LEs involved
0
No of new business proposals for the LEs involved
0
No of new businesses generated in other countries for the LEs involved No of new customers in other countries for the LEs involved
No of local authorities/public organizations (LA) involved in the experiment No LA engagement activities
No of new international partners for the LAs involved
No of signed letter of intent between partners and/or customers for the LAs involved
No of new businesses generated in other countries for the LAs involved No of new business proposals for the LAs involved
No of new customers in other countries for the LAs involved
Did the cross-border collaboration lead to increased turnover for the LAs involved
Did the cross-border collaboration lead to increased customer retention for the LAs involved No of products that has been transferred in the experiment
No of cross-border collaboration tools that has been used the experiment No of NEW (for the stakeholders) ICT-tools that has been used in the experiment No of distributed cross-border collaboration activities
Did the cross-border collaboration tools you used lead to increased access to relevant information Did the cross-border collaboration tools you used lead to increased effectiveness in communication Did the cross-border collaboration tools you used lead to increased co-creation of innovations among stakeholders
No of research activities that has been performed during the experiment No of authored journal articles
No of authored conference papers
No of research conference presentations ICT PSP APOLLON
0 0 2 2 3 0 0 1
N/A N/A N/A 4 4 6 5
yes yes yes 4 1 4
52
14
Final Version
Apollon – D.5.5 Evaluation Report No of new research projects initiated
6
No of Living Labs that has been involved in the experiment
4
Did the cross-border collaboration lead to increased comparability of Living Lab research No of new collaboration initiatives between Living Lab network members (planned, prepared or submitted) No of new Living Lab network members
Did the cross-border collaboration lead to increased access to user communities in other countries?
Did the cross-border collaboration lead to increased value proposition to the stakeholder community Did the cross-border collaboration lead to increased learning of Living Lab collaboration in networks
Did the cross-border collaboration lead to increased maturity of Living Lab management
ICT PSP APOLLON
53
Yes 4
14
Yes Yes Yes Yes
Final Version
Apollon – D.5.5 Evaluation Report
7 Lesson learned from specific pilot results 7.1 Contextual Factors WP5 pilot involved the usage of widespread eMedia and Mobile technologies that are widely used in the three participating becoming more and more widespread across Europe. In order to facilitate the appropriation of technologies by users, we developed a gaming approach and we worked with local institutions (City and Museums) that could help us facilitating the relation with users. Moreover, since all partners could speak English fairly well and the majority could also speak French the communication among partners and the localization of the technologies was not as difficult as it could have been. For all these reasons, we didn’t observe a strong impact of contextual factors on the results of the pilot. Some points deserve nevertheless a particular attention. 7.1.1
Technology and cultural barrier
Access to technology (smartphone and similar) is different across countries. The use of eMedia and Mobile technologies facilitates the involvement of a young audience but can be a drawback for older generations, The Living Lab should consider this aspect and try to make the access to technology as easy as possible, whether by using a mix of low-tech and hi-tech solution, or by making available the tools needed to participate (see 5.7 Partners involvement: End Users). Education is needed to help local stakeholders understand the reasons for supporting cross-border SMEs as well as “local” SMEs. Having a partner with a supervising role is important to make sure that cross-border objectives are not given less importance than local goals (5.3.1 Collaboration management).
The pilot showed that social media and low level “Technology in the pocket devices” are used across Europe and are not an issue for cross-border experiments. Similar issues with regard to lack of knowledge about the potential were similar across all countries. This was also true of social networking sites such as Youtube and Wordpress where there was some limited knowledge about distribution content for local needs for the developing of local content. Future experiments should not forget that working with partners with different skills implies a pedagogical approach of assisting them in the usage of new technologies (see 5.6.1). 7.1.2
Language barrier
This pilot was lucky in that all partners taking part in cross-border experiments spoke and read English and French fairly well. There was no need to contract English language translators to enable IBBT to work in Manchester, or to contract French language translator to enable IBBT and People Voice Media to work in France, nor to contract a Flemish language translator to enable Virdual to work in Belgium. In each case the hosting living lab has been responsible for translating the interfaces of foreign partners to its local language. ICT PSP APOLLON
54
Final Version
Apollon – D.5.5 Evaluation Report There may be a need for translators to be made available (potentially via the European Network of Living Labs or the European Commission?) in order to enable more SMEs to take part in future cross-border experiments. 7.1.3
Copyright issues
Future projects involving the usage of social media for sharing user generated contents will need to be aware of the copyright regulations related to the sharing of cultural contents on the web as this can limit the potential of using social media. For example, in Antwerp AirGraffiti wasn’t able to post the user-generated expositions on Facebook, because of copyright issues related to artistic production. According to M HKA personnel, copyright law does not seem to clear at the moment on how to deal with online reproduction of art works. This problem did not apply in the Manchester Art Gallery experiment, as the topic was Pre-Raphaelite art, on which copyrights have long expired. The same case applied in Issy-les-Moulineaux, for the “Alice in playing cards’ land” experiment.
7.2 Eco-Systems As the pilot evolved, all experiments needed to enlarge the ecosystem of involved stakeholders. This mainly consisted in Cultural Institutions (Museums), Local Authorities (Schools and city departments) and Associations (Youth associations, SMEs associations). The close connection with the local stakeholders ecosystem is a key element of Living Labs and seems crucial to be able to run a successful crossborder pilot. This involves a series of skills and practices that Living Labs should consolidate in order to develop cross-border activities. 7.2.1
Stakeholders identification
In order to be able to set up and manage cross-border pilots Living Labs should develop a good connection with the local ecosystem. This can be done by setting up a centralised "contact relationship management" system or "customer relationship management" system in your Living Lab organisation to keep details of contacts who are willing to act as testers, businesses who are potential business partners, organisations that can act as 'resources' during the testing of a scenario and product, and similar. 7.2.2
Goals definition
SMEs working with Living Labs should be clear what evaluation research they want to get out of the experiment. Even if the product / idea being tested is quite focused (such as an iPhone app), the reason for the carrying out the experiment should be clearly defined before the work required to set up the experiment proceeds. What one or two things is the SME looking to find out by carrying out the experiment? ICT PSP APOLLON
55
Final Version
Apollon – D.5.5 Evaluation Report Clarifying this would help to produce much more focused user testing sessions and identify the most pertinent stakeholders to help with the process.
In WP5 we observed an improvement between the first and the second French experiments. During the first experiment, stakeholders were contacted as the scenario for the experiment and the SMEs goals were defined. During the second French experiment on the contrary, all the stakeholders worked together since the beginning at the definition of the pilot. This allowed speeding up the process of preparing and running the experiment. For the first iteration of the pilot the preparation of the experiment took place in March 2011, 5 months after the first meeting about the scenario (November 2010). For the second experiment, the first meeting for the second took place in November and the experiment was carried out in January 201 (2 months later). 7.2.3
Openness of the consortium
As the pilot scenario evolved, new partners needed to be added to the consortium. This was the case for Mobexplore that became associated partner to momentarily replace Airgraffiti on the first iteration of the French pilot. The local Living Lab (Issy) had a crucial role in this process since it succeeded in finding a partner that could replace Airgraffiti in a very limited time. This is a good example of how SMEs can benefit from working with Living Labs that can find solutions to such situations through a good knowledge of their local ecosystem and a quick access to SMEs and stakeholders. One of the problems we had to face was that no budget could be allocated to this partner since he wasn’t part of the consortium from the beginning. The same happened for the partners SMEs that needed to provide additional ad hoc development to adapt to the scenario or to make sure that two applications could be integrated effectively (see also 5.5.4). Future cross-border pilots involving the integration of two technologies will have to consider this issue. Two solutions can be proposed to deal with this issue in future projects: - Keep a budget for additional partners that need to join the consortium of for extra development costs that need to be covered;
- Encourage the participations of SMEs that can adapt quickly to shifts in the scenario and have developed or are willing to develop solutions (such as APIs) that can facilitate the integration with third party applications;
7.3 Interoperability Issues During the pilot we had to face two kinds of interoperability issues: the technical interoperability of partners’ technologies and the harmonization of partners’ goals. Technical interoperability of the technologies has been reached through the collaboration of SMEs along the definition of the pilot scenario. In all cases this involved ICT PSP APOLLON
56
Final Version
Apollon – D.5.5 Evaluation Report some ad-hoc development to reach a level of integration that was sufficient for the pilot and useful to explore the interest of investing further effort to achieve a seamless integration. SMEs went through face-to-face meetings to introduce each other technologies and then worked remotely towards a common solution. All transfer and integration activities are described in 5.1. The process of harmonizing partners’ objectives is described in 5.5.1 Future projects will have to evaluate to which extent partners’ technology are interoperable asking SMEs to provide a precise description of their technology before the beginning of the pilot and planning early meetings among all technology partners. Once again, SMEs should be encouraged to develop an Application Programming Interface (API) as this facilitates the process of communicating with third party technologies. In addition to this, future projects should not underestimate the importance of the harmonization of partners’ objectives, especially when working with partners with different profile (SMEs, Startups, NGOs) as this is paramount for a fruitful collaboration inside the consortium.
7.4 Lead market opportunities Even if no SMEs increased its business directly because of Apollon (as highlighted in 5.5.2 and 5.8.1), Apollon allowed partners to be introduced to Belgian, French and English markets. Some partners (as AirGraffiti and People Voice Media) could benefit from Apollon to define their business model and identify their business target. Others (as Navidis) were faced with competitors and redefined their position. All partners could benefit from visibility and the opportunity of demoing their technology to European partners. Many of them are today engaged in new European projects with partners they met through Apollon.
Due to travel costs, it was not possible for the SME to engage in face-to-face meetings and communication with all the 'clients' (Issy, Manchester Galleries, MUKKA Museum) as much as would have been liked. This put the Living Labs in an odd position of having to be the SME's "sales representative" when the initial meetings took place with local partners in order to convince them to take part in the experiment. The Living Lab should be very clear on what the product / idea is that is being tested during the experiment. If the SME does not have clear marketing materials or an 'elevator pitch' ready to describe the product / idea, it should work with the LL until such marketing materials are available. Not having clear marketing makes it difficult for the LL to engage local stakeholders who may be able to provide material and facilities to support the experiment. Where possible, the SME's marketing materials should be made available electronically (e.g. password-protected video clip on Vimeo.com, PDF document giving overview) so that the LL can simply present the materials to local stakeholders instead of having to act as a "sales representative" for the SME when working to get local stakeholders interested in taking part in the experiment.
ICT PSP APOLLON
57
Final Version
Apollon – D.5.5 Evaluation Report As far as Business Models are concerned, most of the applications were developed for the cultural and public sectors, of which the subsidies are under threat of being cut due to the economical crisis. The development of business models that requires almost no investment by the cultural institutions seems a good solution for increasing the number of partners interested in these solutions. At the same time, eMedia and social media solutions business models usually avoid charging the service on end users and provide a basic service for free (at least the most popular as Facebook, Twitter, Youtube and similar) sometimes proposing premium features for a reduced fee. This is a challenge that future projects will also need to solve.
7.5 Collaboration Process
7.5.1
Matchmaking: Living Lab and SMES roles and profiles in technology integration pilots
As highlighted in 5.5.5 match-making between SMEs and Living Labs is a fundamental step in determining the success of collaboration between Living Lab and SMEs. When the Living Lab ecosystem is not adapted to the SMEs, opportunities are lost. For example, for SMEs as Virdual specialized in the domain of cross-media and Interactive Television, one of the main regrets for Apollon is the fact of not having been introduced to a network of Living Lab with experts in this domain. In the same way, when the SME solution is not flexible enough it can be hard to integrate it in a Living Lab ecosystem. For this reason, an agile approach (prototype, test, iterate) should be encouraged when working with Living Lab. Startups that can easily pivot if they found unforeseen opportunities can benefit from cross-border pilots more than SMEs with a closed solution. For SMEs that are looking for internationalization or extending their market, Living labs shouldn’t be used as salesmen but could nevertheless act as a single contact point for getting in touch with the local Living Lab Ecosystem. On the other hand, the risk of working with a startup which product is under development is to be delayed on the initial schedule. 7.5.2
Knowledge transfer
As described in 5.5.3 knowledge was shared in Apollon answering partner’s needs and based on each partner skills. In the future, we will need to set up and maintain a platform that will facilitate this process as this will speed up the process of finding and reusing existing methods and tools. Training sessions such as the one described in 5.5.4 are very effective ways of transferring knowledge among partners and can produce sustainable relations in the long term.
ICT PSP APOLLON
58
Final Version
Apollon – D.5.5 Evaluation Report 7.5.3
IPR
The fact of piloting the integration of eMedia technologies raised the questions of IPR for the solutions developed by combining two or more existing technologies. In the case of WP5, both IBBT Virdual have developed part of the M HKA PP prototype and therefore entitled to a part of the IPR. However, IBBT put much more effort in the development that Virdual and Virdual was not included in the development of the second prototype, called MuseUs. As the M HKA PP prototype has been discontinued, no further IPR issues seem to remain. For future projects, IPR and related licenses (Copyright, Creative Commons) will need to be defined at the beginning of the project in order to let each partner decide what to put in the project and understand what they take out. For their vocation to support open innovation, Living Labs should support hybrid models (as Creative Commons or alike) that allows the copyright holder to share some part of the knowledge generated through an innovation project with a wider community.
ICT PSP APOLLON
59
Final Version