Revised Requirements Specifications by CUbRIK Project

REVISED REQUIREMENTS SPECIFICATIONS Human-enhanced time-aware multimedia search

CUBRIK Project IST-287704 Deliverable D2.3 WP2

Deliverable Version 1.0 â&#x20AC;&#x201C; 30 September 2013 Document. ref.: cubrik.D2.3.TUD.WP2/V1.0

Programme Name: ...................... IST Project Number: ........................... 287704 Project Title:.................................. CUBRIK Partners:........................................ Coordinator: ENG (IT) Contractors: UNITN, TUD, QMUL, LUH, POLMI, CERTH, NXT, MICT, ATN, FRH, INN, HOM, CVCE, EIPCM Document Number: ..................... cubrik.D2.3.TUD.WP2.V1.0 Work-Package: ............................. WP2 Deliverable Type: ........................ Document Contractual Date of Delivery: ....................................... 30 September 2013 Actual Date of Delivery: .............. 30 September 2013 Title of Document: ....................... D2.3 Revised Requirements specifications Author(s): ..................................... Approval of this report ............... Summary of this report: .............. Document contains the revised requirements specifications for two vertical applications, including data models and success criteria as a firs step towards the evaluation of pipelines and V-Apps. History: .......................................... Keyword List: ............................... Availability .................................... This report is: public

This work is licensed under a Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported License. This work is partially funded by the EU under grant IST-FP7-287704

CUBRIK Revised Requirements Specifications

D2.3 Version 1.0

Disclaimer This document contains confidential information in the form of the CUbRIK project findings, work and products and its use is strictly regulated by the CUbRIK Consortium Agreement and by Contract no. FP7- ICT-287704. Neither the CUbRIK Consortium nor any of its officers, employees or agents shall be responsible or liable in negligence or otherwise howsoever in respect of any inaccuracy or omission herein. The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2011-7) under grant agreement n째 287704. The contents of this document are the sole responsibility of the CUbRIK consortium and can in no way be taken to reflect the views of the European Union.

CUBRIK Revised Requirements Specifications

D2.3 Version 1.0

Table of Contents EXECUTIVE SUMMARY

INTRODUCTION 1.1 THE ROLE OF V-APPS 1.2 HYBRID HUMAN CONVENTIONAL COMPUTATION 1.2.1 Introducing Hybrid Human Conventional Computation 1.2.2 An example of H2C2 from SME Innovation 1.2.3 An example of H2C2 from History of Europe 1.3 SPECIFYING REQUIREMENTS IN THE CUBRIK PROJECT 1.4 SPECIFYING USE CASES 1.5 SPECIFYING SUCCESS CRITERIA

INTRODUCTION TO THE HISTORY OF EUROPE (HOE)V-APP

2.1 INTRODUCTION 2.2 V-APP DESCRIPTION AND MOCK-UP 2.2.1 Background and rationale 2.2.2 Mock-up 2.3 DATA MODEL 2.4 OVERVIEW OF USE CASE GROUPS AND USE CASES 2.4.1 Indexation 2.4.2 Social graph construction 2.4.3 Uncovering who a person is 2.4.4 Expanding the context to find out more 2.5 FUNCTIONAL REQUIREMENTS FOR THE HISTORY OF EUROPE V-APP 2.6 NON-FUNCTIONAL REQUIREMENTS FOR THE HOE V-APP 2.7 TECHNOLOGY ACCEPTANCE CRITERIA

6 6 6 7 17 24 24 25 25 25 25 27 27

HISTORY OF EUROPE: INDEXATION 3.1 USE CASE: ACQUISITION OF CO-OCCURRENCES 3.1.1 Functional requirements 3.1.2 Non-functional Requirements 3.1.3 Success criteria 3.1.4 Description 3.1.5 Sequence Diagram 3.1.6 CUbRIK H-demos reference 3.1.7 Component List 3.2 USE CASE IDENTITY RECONCILIATION 3.2.1 Functional requirements 3.2.2 Non-functional requirements 3.2.3 Success criteria 3.2.4 Description 3.2.5 Sequence Diagram 3.2.6 CUBRIK H-demos reference 3.2.7 Component List 3.3 USE CASE 1.2 INDEXATION OF TEXTUAL INFORMATION 3.3.1 Functional requirements 3.3.2 Non-functional Requirements 3.3.3 Success criteria 3.3.4 Description 3.3.5 Sequence Diagram 3.3.6 CUBRIK H-demo reference 3.3.7 Component List

2 2 2 3 3 3 4 4

HISTORY OF EUROPE: SOCIAL GRAPH CONSTRUCTION

CUBRIK Revised Requirements Specifications

29 29 29 31 31 32 33 34 34 35 35 36 36 37 38 38 38 39 39 39 39 41 42 42 42 43 D2.3 Version 1.0

4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 5.

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENT LIST COMPONENT SPECIFICATION HISTORY OF EUROPE: WHO IS THIS PERSON?

5.1 USE CASE ‘VISUALIZATION OF THE SOCIAL GRAPH’ 5.1.1 Functional Requirements 5.1.2 Non-functional requirements 5.1.3 Success criteria 5.1.4 Description 5.1.5 Sequence Diagram 5.1.6 CUBRIK H-demos reference 5.1.7 Component List 5.2 USE CASE QUERY FOR ENTITIES 5.2.1 Functional requirements 5.2.2 Non-functional Requirements 5.2.3 Success criteria 5.2.4 Description 5.2.5 Sequence Diagram 5.2.6 CUBRIK H-demos reference 5.2.7 Component List 5.3 USE CASE EXPERT CROWD RESEARCH INQUIRY 5.3.1 Functional requirements 5.3.2 Non-functional Requirements 5.3.3 Success Criteria 5.3.4 Description 5.3.5 Sequence Diagram 5.3.6 CUBRIK H-demos reference 5.3.7 Component List 5.4 USE CASE 3.4 SOCIAL GRAPH NETWORK ANALYSIS TOOLKIT 5.4.1 Functional requirements 5.4.2 Non-functional requirements 5.4.3 Success criteria 5.4.4 Description 5.4.5 Sequence Diagram 5.4.6 CUBRIK H-demos reference 5.4.7 Component List 5.4.8 Component specification 6. 6.1 6.2 6.3 6.4 6.5 6.6 6.7 7.

43 43 43 44 45 46 46 46 47 47 47 48 48 49 50 50 50 51 51 51 52 52 53 53 53 53 53 55 55 56 57 57 57 58 58 59 59 59 60 61 61 61

USE CASE CONTEXT EXPANDER

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENT LIST

62 63 63 64 65 65 65

INTRODUCTION TO THE SME INNOVATION V-APP 7.1 V-APP DESCRIPTION AND MOCK-UP 7.1.1 Background and rationale 7.1.2 Mock-up

CUBRIK Revised Requirements Specifications

66 66 66 67 D2.3 Version 1.0

7.2 V-APPS DATA MODEL FOR FASHION 7.2.1 Twitter Image 7.2.2 Accessibility-related annotations 7.2.3 Sketchness Output 7.3 OVERVIEW OF THE USE CASES 7.4 FUNCTIONAL REQUIREMENTS FOR THE SME INNOVATION V-APP 7.5 NON-FUNCTIONAL REQUIREMENTS FOR THE SME INNOVATION V-APP 7.6 TECHNOLOGY ACCEPTANCE CRITERIA 7.7 OUTLOOK TOWARDS NEXT CHAPTERS 8.

SME INNOVATION: IMAGE CRAWLING FROM SN 8.1 8.2 8.3 8.4 8.5 8.6 8.7

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENTS LIST SME INNOVATION: TREND ANALYSIS FOR CATEGORY

9.1 9.2 9.3 9.4 9.5 9.6 9.7 10.

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENTS LIST SME INNOVATION: TREND ANALYSIS FOR SAMPLE

10.1 10.2 10.3 10.4 10.5 10.6 11.

SME INNOVATION: POPULARITY ASSESSMENT

11.1 11.2 11.3 11.4 11.5 11.6 11.7 12.

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENTS LIST

SME INNOVATION: FASHION MATCHING

12.1 12.2 12.3 12.4 12.5 12.6 12.7 13.

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM COMPONENTS LIST

FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENTS LIST

SME INNOVATION: PERSONALIZED QUERY SUGGESTION

13.1

FUNCTIONAL REQUIREMENTS

CUBRIK Revised Requirements Specifications

79 80 82 84 85 87 88 89 89 90 90 91 91 92 94 95 95 98 98 98 99 100 101 102 102 104 104 105 105 107 110 111 113 113 113 114 115 118 119 119 123 124 124 124 127 132 134 134 139 139

D2.3 Version 1.0

13.2 13.3 13.4 13.5 13.6 13.7 14.

NON-FUNCTIONAL REQUIREMENTS SUCCESS CRITERIA DESCRIPTION SEQUENCE DIAGRAM CUBRIK H-DEMOS REFERENCE COMPONENTS LIST

SUMMARY OF SUCCESS CRITERIA

14.1 USER-BASED PERFORMANCE INDICATORS 14.1.1 V-App level technology acceptance criteria 14.1.2 History of Europe 14.1.3 Fashion 14.2 TECHNICAL SUCCESS CRITERIA 14.2.1 History of Europe 14.2.2 Fashion 15.

CONCLUSION AND OUTLOOK

15.1 NEXT STEPS FOR THE DEVELOPMENT OF V-APPS 15.1.1 Fashion 15.1.2 History of Europe 15.2 NEXT STEPS FOR THE EVALUATION 16.

REFERENCES

CUBRIK Revised Requirements Specifications

139 140 140 142 142 142 144 144 144 144 147 150 150 153 159 159 159 159 160 161

D2.3 Version 1.0

Executive Summary This deliverable, D2.3, contains the refined requirements specifications. The main purpose of this refinement process was to define two vertical applications with a strong link between 1) Business requirements and end-user requirements, and 2) Technological progress achieved by pipelines that combine automatic processing with human intelligence. For that purpose: •

Two vertical applications have been described, consisting of a number of use cases. The applications both fulfil business and end-user needs and demonstrate ‘hybrid human conventional computation’-principles, the core innovation of the CUbRIK project;

•

The data model for both vertical applications has been refined and elaborated, while taking D2.1 Metadata models as the starting point.

Technical success criteria and technology acceptance criteria have been specified, as the first steps towards the evaluation that will take place in the third year of the project. This document consists of three parts:

•

The first part, starting with Section 2, describes the History of Europe V-App with an introductory section, which includes the data model and a description of the document. The subsequent sections each describe one use case; 2) The second part of the document describes the SME Innovation Fashion V-App. Section 7 provides an introduction to the V-App, followed by a description of the use cases; 3) The third part, Section 14, summarizes the technical success criteria and the technology acceptance criteria. The deliverable concludes with an outlook towards the evaluation of both pipelines and VApps in the third year of the project.

CUBRIK Revised Requirements Specifications

Page 1

D2.3 Version 1.0

Introduction

The CUbRIK project investigates the added value of a novel approach to multimedia querying and content processing: the combination of automatic processing with human computation.

1.1

The role of V-Apps

The final results of the project will be an open platform for multimedia search practitioners, researchers and end-users. The platform consists of external interfaces, pipelines (for content, querying, and feedback), components & executors, and platform services. Pipelines are a mixing of automatic operations – jobs – and human activities that are chained in a sequence. Applications can be built on top of the CUbRIK platform1. To demonstrate the added value of the CUbRIK platform, in this project two vertical applications are developed for two domains of practice (SME Innovation and History of Europe). In addition, a set of horizontal demos are developed aimed at providing proofs-ofconcept and demonstrating specific functionalities (components, pipelines, or apps that are not tied to a domain of practice).

1.2

Hybrid Human Conventional Computation

1.2.1

Introducing Hybrid Human Conventional Computation

Hybrid human conventional computation (H2C2) pipelines are the main innovation of the CUbRIK project. In H2C2 pipelines, automatic processing is combined with crowdsourcing in different configurations. In the vertical applications, the human is put "in the loop" in two different ways: to extend or verify the information that is generated by automatic processing (detection or ranking); • to verify the information that is introduced into the system by other users. The images with the additional trusted information obtained from crowdsourcing are then used in the application. A schematic overview of a Hybrid human conventional computation pipeline is displayed in Figure 1. •

Figure 1 Hybrid human conventional computation pipelines 1 A comprehensive description of the CUbRIK architecture can be found in D9.2 Delivery Management Plan and Testing Specification.

CUBRIK Revised Requirements Specifications

Page 2

D2.3 Version 1.0

1.2.2

An example of H2C2 from SME Innovation

The SME Innovation V-App focuses on the domain of Fashion. One of the use cases is focused on crawling fashion images from the web, detecting body parts in the images, and identifying clothing items. This use case prepares for the detection of fashion trends. It combines automatic detection components (detecting texture and color in the images and detecting the upper and lower body part within an image) with a crowd sourcing component: a Game With A Purpose (GWAP) called Sketchness (developed by POLMI), in which the players draw contours around the clothing items.

1.2.3

An example of H2C2 from History of Europe

Identifying people that played a part in European history is an important part of the V-App ‘History of Europe’. In the use case ‘Indexation’, source images from the archives are loaded in the pipeline. Automatic detection is used to detect faces. Crowdsourcing is used to verify the position of the faces in the images. Finally, an automatic face recognition process returns a list of potential identities for the faces. The examples demonstrate that in this project human hybrid conventional computation pipelines are employed that combine automatic components with crowdsourcing components in different orders and for different purposes.

1.3

Specifying requirements in the CUbRIK project

In WP2, the requirements are specified for both vertical applications and H-Demos. In the first year of the project, the results of the requirements specification process have been documented in two deliverables: D2.1 Metadata models and D2.2 Requirements specifications. In D2.1 Metadata models, a generic data model for the CUbRIK platform has been presented. In D2.2, delivered in M9, the preliminary functional and non-functional requirements have been specified for the vertical applications and for the H-Demos. D2.2 includes the usercentered design work that has been conducted in order to assess the needs of the target users in both domains of practice (i.e. user requirements). The user centered design efforts have been conducted in an inter-workpackage workgroup between Task 2.2 Functional and Non-Functional requirements, Task 10.1 Application requirements and coordination, and Task 7.1 Content collection and ground truth management. This work has led to the choice for two domains of practice: History of Europe and SME Innovation/Fashion. For both domains of practice, preliminary user requirements have been derived. This deliverable, D2.3, contains the refined requirements specifications. The main purpose of this refinement process was to define two vertical applications with a strong link between 1) business requirements and end-user requirements, and 2) technological progress achieved by pipelines that combine automatic processing with human intelligence. To achieve this goal, different components and pipelines have to be specified in detail and have to be aligned with the vertical applications. This deliverable reflects this purpose, and compared to D2.2, the following progress has been made: •

• •

The specification of use cases, pipelines, and components has been refined and directed towards the context of the vertical applications in the two domains of practice; The generic data models have been updated and tailored to the specific setting of the V-Apps; Success criteria have been specified. Two types of success criteria have been specified: technical success criteria, and user-based performance criteria. Both types of success criteria form the basis for the evaluation of CUbRIK pipelines (Task 10.4 Evaluation of CUbRIK Pipelines) and CUbRIK apps (Task 10.5 Evaluation of CUbRIK

CUBRIK Revised Requirements Specifications

Page 3

D2.3 Version 1.0

Apps), respectively. During the process of zeroing in on addressing concrete needs within the two domains of practice, we have prioritized the use cases of the V-Apps by choosing those that make the most critical use of H2C2 principles, in other words, those that most directly exploit the innovative contribution of CUbRIK.

1.4

Specifying Use Cases

As this deliverable is focused on the vertical applications and the needs of their users, user stories are the primary point of reference: pieces of functionality end-users in the domains of practice have expressed they need for their daily work. Thus, the use cases are rooted in the user-centered design studies that have taken place in the first year of the project. This specification document then bridges the gap between the high-level user needs and the detailed component specifications by describing use cases. They are specified to the extent that they impact the end users. The underlying components have been specified in detail, but for reasons of readability, are not part of this deliverable. These detailed specifications will be included in the updated version of D9.8 Architecture Design. The use cases contain a set of pipelines that are needed to achieve useful results for the end-user. Some of the use cases are preparatory for the other use cases. In the description of the use cases, the hybrid human conventional computation pipelines are emphasized.

1.5

Specifying Success Criteria

As stated in the previous section, we will assess the success of the V-Apps not only with regard to the technical performance, but also with regard to technology acceptance, since the V-Apps have the purpose of demonstrating the added value of CUbRIK technology to users in the two domains of practice. In preparation for assessing the technical performance, we have formulated technical success criteria for each use case. The success criteria are input to the system-oriented evaluation in Task 10.5 Evaluation of CUbRIK Pipelines. The success criteria are formulated as [measure] [comparison operator] [value]. Technology acceptance is assessed on two levels: the level of the V-App as a whole and the level of a use case. The V-App level needs to be addressed, as the perception of the users of the individual use cases is influenced by their perception of the application as a whole â&#x20AC;&#x201C; and vice versa. To assess technology acceptance on the level of vertical applications, we will take the Unified Theory of Acceptance and Use of technology (UTAUT; Venkatesh et al., 2003) as our starting point. The model is the validated result of integrating eight different user acceptance theories and their measurement instruments. In addition to these dimensions, we also address the task-technology fit: the extent to which the capabilities of the technology match with the tasks the user must perform. (Goodhue & Thompson, 1995). This aspect is not covered by the UTAUT model Apart from the high-level technology acceptance constructs, we specifically pay attention to usability considerations, as this is expected to be an important determinant for user adoption of the CUbRIK technology. The resulting set of technology acceptance indicators will be presented in the introduction sections for each V-App (Section 2 and 7). In addition to the V-App level, we also assess use-case specific aspects related to usability, usefulness and the usersâ&#x20AC;&#x2122; perception on the incentive models that are employed. The resulting set of user-based performance indicators will be presented in the use case sections. Both technology acceptance indicators and user-based success indicators do not have a target level, as target levels tend to be arbitrary for user-based success indicators. Therefore, in user-centered design research it is common practice to define what is going to be

CUBRIK Revised Requirements Specifications

Page 4

D2.3 Version 1.0

measured and then analyse the measurement results afterwards, without defining a threshold value a priori. Furthermore, Likert scales are not analysed with the purpose of providing means and standard deviations, but with the purpose of getting insight in how participantsâ&#x20AC;&#x2122; perceptions are distributed over the scale points.

CUBRIK Revised Requirements Specifications

Page 5

D2.3 Version 1.0

Introduction to the History of Europe (HoE)V-App

2.1

Introduction

In D2.2 Requirements specifications, History of Europe was introduced as a domain of practice to demonstrate the value of CUbRIK technology to help businesses achieve their goals. In their projects, CVCE’s history experts perform a number of tasks that are often complicated and time-consuming. Finding relevant materials, getting access to collections, dealing with copyright issues, and interpreting the results are examples of the challenges they are faced with. The application developed here provides computer support for a part of these challenges. Please refer to D2.2 for more information about the domain of practice.

2.2

V-App Description and Mock-up

2.2.1

Background and rationale

Four different user stories were developed in the beginning of the requirement definition phase for the History of Europe application: • •

• •

User story 1: Collecting materials at the archive; User story 2: o A: Where was this photo taken? o B: Who is this person? Tell me more! User story 3: Can I publish this? User story 4: Documenting the results.

The scenario sketched in User story 2 B: Who is this person? Tell me more! will be used as a point of departure for the implementation of the History of Europe application. The modified user story reads as follow: A newly scanned photo shows a few people at a dining table. Amidst familiar faces, Ibrahim notices one person that is unknown to him. He uploads the image to the History of Europe application to find out who this person is. He selects the person’s faces in the scan and provides the names of some of the people he already recognized. After a while, the image analyser returns suggestions for the identity of the missing persons. One of the suggestions proposes that the unknown person might be a Dutch politician. Based on the reference images that the application offers him, Ibrahim is able to confirm this identity. Ibrahim wants to find out more details about the photo and starts the exploration of the social graph. There, he finds more photos of the same event. Based on the number of cooccurrences of persons in these and other photos, he is able to identify possible candidates who, to his surprise, could also have known the politician he identified in the initial picture. For every person Ibrahim selects in the graph he sees the results of the context expander, which in addition to photos provides him with a list of media items such as newspaper articles, interviews, commentaries and videos.

CUBRIK Revised Requirements Specifications

Page 6

D2.3 Version 1.0

2.2.2

Mock-up

Based on the modified user story, a mock-up of the History of Europe V-App was created to help define a set of functional requirements and to be the basis for implementation. The mock-up was presented in a second focus group with historians to refine the requirements and success criteria. The envisioned application will provide the following main functionalities: •

•

The application will contain a set of existing and well-established media collections, including the CVCE and Europeana collection. Different types of media documents will be available to the users: photographs, videos, audio recordings and text documents; Users will be able to upload their own media documents Figure 2), to organize and bookmark documents by creating their own collection folders, and to share their uploads and collections with the History of Europe community; Different research tools are available to users to enable them to work with the available media documents: Text-based Search, Social Graph Exploration, Document Annotation and Research Inquiry.

CUBRIK Revised Requirements Specifications

Page 7

D2.3 Version 1.0

Figure 2 The homepage: overview of available documents, research tools and recent activity

CUBRIK Revised Requirements Specifications

Page 8

D2.3 Version 1.0

Figure 3 Media harvesting and uploading

CUBRIK Revised Requirements Specifications

Page 9

D2.3 Version 1.0

With the Text-based Search, users can search for media documents and known entities (persons, places, events).

The social graph explorer A more explorative approach to working with media documents is the Social Graph Explorer (Figure 4). It depicts a graph of links between actors that were identified in the processed media documents from the History of Europe collections. The nodes represent the actors, while the graphâ&#x20AC;&#x2122;s edges represent the media documents that connect two actors. Different filters are available to users to explore the graph, including a timeline, an events- and placesfilter to be able to retrieve time-, location- and event-based sub graphs. With such a graph tool, historians can explore co-occurrences of different actors in the media documents and get an overview of the complex structure, which can generate leads for new research perspectives. The birds-eye-view can give new insights into seemingly well-known knowledge (recapitulating) and lead to surprises, e.g. identifying until then unknown persons documents by finding unexpected links. Patterns and paths between actors can be discovered as well as overlaps between several well-known networks (aggregation of information). In order to compute the Social Graph and accumulate knowledge about the various media documents (identification of actors, places, events, time and additional information that help to provide context), the media documents need to undergo both automatic processing techniques as well as general-purpose and expert crowdsourcing. It is the combination of these methods that sets the application apart from similar works.

Figure 4 Visualization of the Social Graph. The "min" and "max" controls allow to analyse weak and strong links

CUBRIK Revised Requirements Specifications

Page 10

D2.3 Version 1.0

Automatic face detection and face position validation Collections and media documents that are added to the History of Europe application first go through an automated process. To identify possible actors in photographs, automatic face detection is combined with general-purpose crowd face position validation to rule out falsepositives and false-negatives. This validation is usually done through the Microtask plattform, but for single user uploads the History of Europe application will provide a built-in editor for users to manually delete bounding boxes and add new faces (Figure 5 and Figure 6). Only after this step is completed, automatic face recognition is applied to provide confidence scores of the top identification matches for each previously marked face. For more accurate results, expert crowdsourcing is a key element of the History of Europe application, as a high level of domain knowledge is required to verify most of the automatic recognition results and to enrich the media documents with additional information such as metadata.

Figure 5 CROWD face position validation, manual annotation by user for single images, page 7

CUBRIK Revised Requirements Specifications

Page 11

D2.3 Version 1.0

Figure 6 CROWD face position validation, manual confirmation by user for single images Expert-crowdsourcing is realized with the two tools: Document Annotation for implicit expertcrowdsourcing and Research Inquiry for explicit expert-crowdsourcing. The Document Annotation tool can be used by users of the application, who are experts in European Integration and similar fields, whenever they are viewing a document such as a photograph. With the tool, they can annotate metadata of a photograph, e.g. the caption, the date or event the photograph was taken, and data related to depicted persons, e.g. the name, affiliation or birth date (Figure 7, Figure 8, Figure 9).

CUBRIK Revised Requirements Specifications

Page 12

D2.3 Version 1.0

Figure 7 Entity annotation and verification

CUBRIK Revised Requirements Specifications

Page 13

D2.3 Version 1.0

Figure 8 Expert CROWD verification

CUBRIK Revised Requirements Specifications

Page 14

D2.3 Version 1.0

Figure 9: Expert CROWD verification However, such annotations are often ambiguous, and historians need to be able to comprehend others’ reasoning. Therefore, users can propose a suggestion for a particular annotation field and also provide a short explanation of their chain of thought to make suggestions transparent for other users. Author-based majority voting is also included, i.e. it is possible for users to vote on others’ suggestions to identify more likely ones. The automatic face recognition confidence scores are also displayed in the Document Annotation tool to provide users with an initial guess that they can use if they are trying to identify unknown persons. Requirements analysis revealed that historians are already using existing social media networks, e.g. Twitter, to distribute mostly image-related queries such as ”Who is this person?” among colleagues. With the Research Inquiry tool, users can address their professional social media networks and the History of Europe community to ask specific questions about media documents or particular annotation fields (Figure 10). With the tool, they can easily retrieve and manage the answers their colleagues provide.

CUBRIK Revised Requirements Specifications

Page 15

D2.3 Version 1.0

Figure 10 Expert CROWD Research Inquiry, New Research Inquiry The activity monitor on the application homepage helps users keep track of incoming Expert Inquiry responses and new Expert Inquiries from other users. Additionally, new annotations and new media documents are tracked there as well (Figure 2).

CUBRIK Revised Requirements Specifications

Page 16

D2.3 Version 1.0

Figure 11: Relationship between user stories and use cases

2.3

Data model

In D2.1 an extensive data model has been presented for the CUbRIK platform as a whole. The diagram displayed in Figure 12 contains an adapted version of the data model that is adjusted to the History of Europe vertical application. In particular, this model makes use of a subset of the elements in the Content and Content Description Model described in Section 1.2 of D2.1. This model is split into three main sub-models, the Entity Model (which models objects or events in the real world), the Content Model (which models content objects), and the Content Description Models (which models Annotations, and mediates between the content world and the real world). Figure 12 shows that refinements have been added (for example, the specification of a location as a street address), but that also elements representing other CUbRIK data models have been incorporated, especially the meta-entity class below is used to express elements from the Provenance and Rights Model (source). As in the Annotation model, confidence is specified as a value associated with an annotation, here designated â&#x20AC;&#x153;Accuracyâ&#x20AC;?. This simple

CUBRIK Revised Requirements Specifications

Page 17

D2.3 Version 1.0

representation of confidence makes it possible to track uncertainty according to the needs of the HoE V-App. Key

Value

Description

id number (long)

The number we use to identify the entities. (unique)

Subclass

Text (String)

The name of the entity subclass.

Name

Text (MString)

The name of the entity

Description

Semantic text (semString)

A description of the entity

Start

date

The date when the entity starts

End

date

The date when the entity ends

Duration

Time period (Mduration)

The period of time between start and end

part of

Link to an entity (MEntity<entity>)

Describe the entity as a part of the other entity

Homepage

URL (MURL)

The URL of a webpage related to the entity

externalId

id number (long)

An id referencing the entity on an external database

Entity

Entity (MEntity< Entity >)

The entity this MetaEntity references

Attribute

Id number (Mlong)

The id of the attribute it references

Value

MLong

Source

Entity (MEntity< Entity >)

The entity representing the source of this data

Accuracy

Score (MDouble)

A number representing the accuracy of this data

Street

Text (MString)

The name of the street

civicNumber

Text (MString)

The number assigned to the address.

postalCode

Text (MString)

The postal code of the city where this address is located.

City

Entity (MEntity<location>)

The city where this address is located.

Entity

MetaEntity

Address

CUBRIK Revised Requirements Specifications

Page 18

D2.3 Version 1.0

State

Entity (MEntity<location>)

The state where address is located.

this

Country

Entity (MEntity<location>)

The country where address is located.

this

Type

Text (MString)

The type of keypoint

Number (MInteger)

The X coordinate of the keypoint

Number (MInteger)

The Y coordinate of the keypoint

Type

Text (MString)

The type of tag

Image

Entity (MEntity<image>)

The entity of the image on which this tag is made.

ImageOfTheBoundingBox

Entity (MEntity<image>)

The part of the image where we tag it.

Entity

Entity (MEntity<entity>)

The entity tagged.

Left

Number (MInteger)

Left coordinate of the tag (relative to the image)

Top

Number (MInteger)

Top coordinate of the tag (relative to the image)

Right

Number (MInteger)

Right coordinate of the tag (relative to the image)

Bottom

Number (MInteger)

Bottom coordinate of the tag (relative to the image)

Keypoint

Entity (MEntity<keypoint>)

The keypoint representing the tag on the image.

School

Entity (MEntity<Organization>)

The establishment where this education can be taken

Field

Semantic text (MSemString)

The field of education

Degree

Semantic text (MSemString)

The academic degree delivered with this education

Creator

Entity (MEntity<entity>)

The entity representing the photographer

Artefact

Entity (MEntity<Image>)

The entity representing the picture itself

Source

Semantic text (MSemString)

The source of the photo

deviceModel

Semantic text (MSemString)

The model of the device

Keypoint

Tag

Education

Photo

CUBRIK Revised Requirements Specifications

Page 19

D2.3 Version 1.0

used deviceID

Text (MString)

The id of the device used

creationTime

Date (MDate)

The date when this photo was taken

sceneType

Text (MString)

The type of scene described by the photo

exposureTime

Number (MDouble)

The time of exposure used for this photo in milliseconds

focalNumber

Number (MDouble)

The focal number used for this photo

focalLength

Number (MInteger)

The length of the focal used for this photo in millimeters

ExposureProgram

Semantic text (MSemString)

The exposure program used for this photo

SpectralSensitivity

Text (MString)

The spectral sensitivity of the device used for the photo

isoSpeedRatings

Number (MInteger)

The ISO speed rating used for this photo

Oecf

Number (MInteger)

The OECF (opto electronic conversion function) of the device used for the photo

exposureBiasValue

Number (MDouble)

The value of the exposure bias used for the photo

SubjectDistance

Number (MDouble)

The distance of the subject in meters

subjectLocationX

Number (MInteger)

The X coordinate of the subject

subjectLcationY

Number (MInteger)

The Y coordinate of the subject

lightSourceMode

Semantic text (MSemString)

The mode of light source used for the photo

meteringMode

Text (MString)

The metering mode used for the photo

Flash

Boolean (MBoolean)

0 : no flash, 1 : flash

flashEnergy

Number (MDouble)

The energy released by the flash

flashReturn

Semantic String (MSemString)

Indicates flash return status

redEyeReduction

Boolean (MBoolean)

0 : no red eye reduction, 1: red eye reduction

Orientation

Number (MInteger)

The orientation of the photo

CUBRIK Revised Requirements Specifications

Page 20

D2.3 Version 1.0

Location

Entity (MEntity<location>)

The place where the photo was taken

directionReference

Semantic String (MSemString)

A reference to the direction of the photo

directionOrientation

Number (MDouble)

The orientation of direction of the photo

universalTime

Date (MDate)

The date when the photo was taken at UT

harvestingDetail

Text (String)

Details about the photo

Filename

Text (MString)

The name of the file

url

URL (MURL)

The URL of the file

Format

Text (MString)

The format (extention)

Size

Number (MLong)

The size of the file (in octet)

deviceModel

Semantic String (MSemString)

The model of the device used to create this file

deviceId

Text (MString)

The Id of the device used to create this file

creationTime

Date (MDate)

The Date of creation of the file

Source

Semantic String (MSemString)

The sources of the file

Creator

Entity (MEntity<entity>)

The entity representing the creator of the file

Tag

Semantic String (MSemString)

Tags related to this file

mindProduct

Entity (MEntity<entity>)

References derived mind products

Height

Number (MInteger)

The height of the image

Width

Number (MInteger)

The width of the image

bytePerPixel

Number (MInteger)

Number of bytes per pixel of the image

colorModel

Semantic String (MSemString)

The color space of the image

lowLevelFeature

Entity (MEntity<file>)

The low level description of the image

Semantic String (MSemString)

Nationality of the person

the

File

the

file

Image (sub class of file)

Person Nationality

CUBRIK Revised Requirements Specifications

Page 21

D2.3 Version 1.0

lifeSpan

Duration (MDuration)

Life span of the person

DateOfBirth

Date (MDate)

Date of birth of the person

dateOfDeath

Date (MDate)

Date of death of the person

placeOfBirth

Entity (MEntity<Location>)

Place of birth of the person

placeOfDeath

Entity (MEntity<Location>)

Place of person

Role

Entity (MEntity<role>)

Entity representing the role of the person

Education

Entity (MEntity<education>)

Entity representing the education of the person

Seat

Entity (MEntity<Location>)

The place where the organization takes seat

dateOfEstablishment

Date (MDate)

Date of establishment of the organization

dateOfDisestablishment

Date (MDate)

Date of disestablishment of the organization

Language

Semantic String (MSemString)

Language used within the organization

Participant

Entity (MEntity<Entity>)

Entity representing event participants

Location

Entity (MEntity<Location>)

The place where the event takes place

Latitude

Number (MDouble)

Latitude of the location

Longitude

Number (MDouble)

Longitude of the location

Altitude

Number (MDouble)

Altitude of the location

Source

Entity (MEntity<Entity>)

Entity representing provenance source

Accuracy

Number (MDouble)

Accuracy of the automatic recognition process

Vote

Boolean (MBoolean)

0: negative vote 1: positive vote

Comment

Semantic String (MSemString)

Comment free text

death

the

Organization

Event the

Location

Provenance

CUBRIK Revised Requirements Specifications

Page 22

the

D2.3 Version 1.0

Figure 12: HoE data model for Entitypedia CUBRIK Revised Requirements Specifications

Page 23

D2.3 Version 1.0

2.4

Overview of Use Case Groups and Use Cases

The History of Europe app implements User story 2 B: Who is this person? Tell me more! as four different use case groups that may contain a number of separate use cases. An overview of the use case groups is displayed in Figure 13. Image Indexation Clickworkers

Acquisition of co-occurrences

Media harvesting and upload

Provenance checker

Met adat a Entity ex traction

License checker

Crowd Face position validation

Face detection

Face identifi cation

Content provider tools

Data Set Creation

Face recognition

Identity reconciliation CROWD prefi ltering

Entity verifi cation & annotation

Entitypedia Integration

Expert Crowd

Indexation of textual information

Connection to the CVCE collection

Entity anntation and ex traction

Entity extraction

Social Graph construction

Entitypedia data provisioning

Ex pert CROWD verifi cation

Entitypedia Integration

Entity reconciliation

Uncovering who a person is Visualization of the social graph

Social Graph Creat ion

Expert CROWD Research Inquiry CROWD Research Inquiry

Graph Visualization

Expanding the context […] Expert Crowd Ex pansion through docum ents

Ex pansion through im ages

Ex pansion through videos

Ex pansion through related entities

Social Graph Network Analysis Tk Analysis of the social graph

Graph Visualization

Query for entities

Query ex ecution

Graph Visualization

Figure 13 Overview of use case gropus

2.4.1

Indexation

The Indexation use case group breaks down into two use cases. The indexation of images starts with the Acquisition of Co-Occurrences sub use case. This use case acquires cooccurrences by identifying the individual actors in historical images. Single images or bulks of images are uploaded directly by the user or crawled from connected online archives in the Data Set Creation process step. The optional Content Provider Tool process allows owners of content archives to assign specific usage rights to the images in their archives. During the crawling process a license checker consumes these usage rights and stores the corresponding information. The provenance identification component assures that all images that have been analysed before are identified and associated to the corresponding CUBRIK Revised Requirements Specifications

Page 24

D2.3 Version 1.0

annotations. All images that haven't been analysed before are now exposed to the Face Recognition process. In this process the position of faces in the images is detected algorithmically and verified by crowd-workers. Detected and verified faces are then analysed by a face identification process that associates different candidates for a personâ&#x20AC;&#x2122;s identity to entities in the Entitypedia. In Year3, contextual information of the social graph as well as entities provided in the metadata (person names, places, organizations, events and time) will be taken into account during the identification process. In the Identity Reconciliation use case, humans verify the results of the automatic identification processes. In the Crowd pre-filtering component a generic crowd tries to verify the identity of the depicted persons. Identities that could not be verified in the previous step are forwarded to the Entity Verification and Annotation component where expert users can verify the associations between faces and entities but also provide additional annotations to images, which are in turn stored in the Entitypedia. The Text Indexation use case provides verified entities for textual documents in the CVCE collection. Once a document is added to the CVCE collection, the existing entities are extracted and then verified by a crowd process. The resulting references are stored back in the Entitypedia.

2.4.2

Social graph construction

The social graph construction process generates the social graph based on the cooccurrences of persons in images.

2.4.3

Uncovering who a person is

In the use case group Uncovering who a person is, users can get a visualization of the social graph, interact with it through queries, start research inquiries for the expert crowd to verify uncertain/unclear information or apply analytical tools to the social graph. The aforementioned features are all separate use cases under the â&#x20AC;&#x2DC;uncovering who a person isâ&#x20AC;&#x2122; use case group.

2.4.4

Expanding the context to find out more

The Expanding the context to find out more use case takes entities as a point of departure to expand them with documents, videos or images. In the case of images, both already indexed images as well as publicly available images, e.g. from Flickr, are shown. Images from external data sources such as Flickr are fed back into the image indexation process and become in turn a part of the social graph.

2.5

Functional Requirements for the History of Europe V-App

In this section we present high-level functional requirements that explain the core functionality of the History of Europe vertical application. The requirements will be elaborated in later Sections, which each deals with a separate use case. #

Functional requirement

HFR1

The application must enable the bulk upload of images containing people Related use cases - Acquisition of co-occurrences

HFR2

The application must be able to detect faces in the image

CUBRIK Revised Requirements Specifications

Page 25

D2.3 Version 1.0

Functional requirement Related use cases - Acquisition of co-occurrences

HFR2

The application must have the crowd verify the position of faces in the images Related use cases - Acquisition of co-occurrences

HFR4

The application must enable users to identify the faces of people known to the user Related use cases - Acquisition of co-occurrences

HFR5

The application must enable users to identify the faces of people known to the user Related use cases - Acquisition of co-occurrences

HFR7

The application must enable users to identify the faces of people known to the user Related use cases - Acquisition of co-occurrences

HFR8

The application must be able to provide suggestions for the identity of the persons that were not identified by the user yet. Related use cases - Identity reconciliation

HFR9

The application must display reference images of the same person that enables the user to confirm or reject the identity of unknown persons Related use cases - Image indexation, identity reconciliation

HFR10

The application should contain a social graph that enables users to explore the relationship between persons, places and events through: the 1) Co-occurence in media sources (images, text documents, video, audio), 2) Identification of weak ties; Related use cases - Uncovering who a person is (visualization of the social graph)

HFR11

The application must enable users to start a research inquiry to supplement or verify information in the social graph by submitting questions to one or more of the following crowds: generic crowds, the application community of experts, social media contacts, and the userâ&#x20AC;&#x2122;s e-mail contacts.

CUBRIK Revised Requirements Specifications

Page 26

D2.3 Version 1.0

Functional requirement

HFR12

Related use cases - Uncovering who a person is (expert crowd research inquiry) The application should enable users to retrieve a list of media items (newspapers, interviews, commentaries, and videos) related to a selected person or media item Related use cases - Expanding the context to find out more (Context expander) Table 1 Functional requirements for History of Europe

The technical success criteria will be discussed in the sections that deal with individual use cases.

2.6

Non-functional requirements for the HoE V-App

Non-functional requirement

HNFR1

The application shall be accessible and useable for non-technical audiences.

HNFR2

The application shall be well documented and described

HNFR2

The application shall be transparent, allowing researchers e.g. to understand when results are modified by algorithms

HNFR3

The application shall be extensible to any domain within the field of History and beyond

HNFR3

The application shall be scalable in the sense that it can be used by a large group of users from different domains (professionals, students, teachers, politicians, journalists) and that it can be extended with additional data collections

HNFR3

The application shall be easy to use Table 2 Non-functional requirements for History of Europe

2.7

Technology acceptance criteria

As explained in the introduction, we use the UTAUT framework (Venkathesh et al., 2003) as our frame of reference for the assessment of technology acceptance criteria. The following indicators that were derived from the UTAUT framework will be used: • • •

Performance expectancy: the degree to which an individual believes that using the system will help him or her to attain gains in job performance; Effort expectancy: the degree of ease associated with the use of the system; Attitude towards using technology: an individual's overall affective reaction to using a system.

Type of indicator

Type of user

Indicator

Type of measurement

Technology acceptance indicators

Expert historian

Performance expectancy

Questionnaire (self reported, likert scale)

Expert historian

Effort expectancy

Expert historian

Attitude toward using

Questionnaire (self reported, likert scale) Questionnaire (self reported,

CUBRIK Revised Requirements Specifications

Page 27

D2.3 Version 1.0

Type of indicator

Overall Usability

Type of user

Indicator

Type of measurement

technology

likert scale)

Expert historian

Task-technology fit

Questionnaire (self reported, likert scale)

Expert historian

Ease of Use

Questionnaire (self reported, likert scale)

Expert historian

Interface Errors

Analysis of logs

Expert historian

Critical Incidents

Analysis of logs

Table 3 Technology Acceptance Criteria We address ease of use by means of self-reporting via Likert scales and by asking respondents to describe critical incidents. Additionally, we analyse the logs in order to detect interface errors that might point out to usability issues.

CUBRIK Revised Requirements Specifications

Page 28

D2.3 Version 1.0

History of Europe: Indexation

The Indexation use case is split up into different use cases: •

•

3.1

Indexation of images: o Acquisition of Co-Occurrences; o Identity reconciliation; Indexation of textual information.

Use case: Acquisition of Co-Occurrences

The identification of persons in historical images is the foundation of the social graph and the basis for the analysis of co-occurrences. In this use case, source images are loaded in the pipeline, and in these source images faces are identified and verified by a generic crowd before an automatic face recognition process returns a list of potential identities for the faces. •

• • • •

3.1.1

The existing images of the CVCE collection will be used as an initial set: o to identify and recognize faces; o to associate these faces with Entitypedia entities in order to finally identify cooccurrences' Source images might be uploaded or can be chosen from one of the imported archives; Content providers can assign usage rights for different user groups to their content; Indexation takes place either for individual images or for bulk imports of images; The use case is covered in the interface mock-up (see Figure 3).

Functional requirements

Functional requirement

The application must contain the CVCE collection data set

The application must enable users to upload and manage new media items individually and in bulks

The application must integrate meta information about a media object, such as captions, sources, date, location and event, photographer, copyright and relevance

The application must integrate information about persons appearing in the media objects, such as name, lifetime and affiliation

The application must provide an indicator for the estimated duration of automatic recognition tasks

The application must give access to content in accordance with the permission levels that are assigned to different user groups; content providers must be able to determine the usage rights (such as the right to re-publish, analyse, distribute to generic crowds, expert crowds) per user group.

The application must integrate a reliability indicator that shows the estimated reliability for results that come from automatic recognition tasks Table 4 Functional requirements for 'Acquisition of co-occurrences’

CUBRIK Revised Requirements Specifications

Page 29

D2.3 Version 1.0

Figure 14: Component diagram for the use case ‘Acquisition of Co-Occurrences’

CUBRIK Revised Requirements Specifications

Page 30

D2.3 Version 1.0

(optional) license information related to images references

meta-data related to image references

Related tasks 10.2

References to image files

Task 5.3

DEMO NONE

UNITN

EMPOLIS

CVCE

Partners

READY

Task 5.3

DEMO NONE

FRH

Partners

Entity extraction from metadata

Task 8.1

Task 5.3

Task 4.1

DEMO News history

POLMI

FRH

Partners

License checker

Image file references from the CVCE collection OR from external archives OR user uploaded images

DEMO NONE

L3S

FRH

CERTH FRH

Partners

Provenance checker

Data Set Creation – owner: CERTH Copyright aware crawler

Content provider tool

Media harvesting and upload

* IMPLEMENTED AND INTEGRATED Current use in a sandbox, use of full crowd after Como. Adding faces: Will be added till the end of the month

person entities

social graph

READY

Task 8.1

DEMO media entity annotation

FRH

POLMI

Partners

Face identification

face positions with associated Entitypedia entities

Ready *

Task 5.2

L3S

MICT

POLMI

Partners

Image files

READY

Task 8.1

DEMO human enhanced trademark logo detection

FRH

POLMI

Partners

CROWD face position validation

Face recognition – owner: POLMI Face detection

3.1.2

Non-functional Requirements

In the next section, measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address processing time, memory usage, precision and recall. From the user point of view, perception of usefulness and perception of effort are addressed.

3.1.3

Success criteria

In Table 4 the technical success criteria are displayed. Component Content provider tool

Indicator avg processing time

Target level < 100 ms;

Content provider tool Copyright aware Crawler

heap memory usage

< 30 MB

usedBandwidth / availableBandwidth

> .7

Provenance checker

avg processing time < 100 ms

< 100 ms

Provenance checker License checker

heap memory usage < 80 MB avg processing time < 20 ms

< 80 MB

Face detection

heap memory usage < 80 MB Face detection: Recall

Face identification

Precision (top-1) Precision (top-10)

< 20 ms

> .50 (@FP rate < .05)

> .15 > .35

Remarks includes file operations (read/write), hashing, XML document validation and XMLSignature creation (regardless of size and number of processed content items) bandwidth efficiency: assumption: storage is fast enough; if lower: limiting end must be provider bandwidth limiting) includes file operations (read/write), hashing, database query based on hash (against empty HSQL reference database), provenance info XML creation (regardless of size and number of processed content items) includes file operations (read/write), XML document validation, XML-Signature validation, mapping license to platform permissions, creating permissions-info XML (regardless of size and number of processed content items) The figure refers to unrestricted conditions, including non-frontal faces, varying illumination, varying size, occlusions, film grain, etc.). FP rate .05 means that .05X wrongly detected faces were found in X tested images. An algorithm that performs face identification based on a random guess would achieve top-1 precision equal to .004 (ground truth with 250 different classes â&#x20AC;&#x201C; individuals)

Table 5 Technical success criteria for 'Acquisition of co-occurrences'

CUBRIK Revised Requirements Specifications

Page 31

D2.3 Version 1.0

In Table 5 the user-based performance indicators are displayed for acquisition of cooccurrences. Human-computer transaction

Type of user

Indicator

Type of measurement

User extends the existing collection with own photographs/uploads

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

User validates/defines the position of faces in images

Expert Historian / generic crowd

Perception of effort of manual face validation

Questionnaire (self reported, likert scale)

Table 6 User-based performance indicators for 'Acquisition of co-occurrences’

3.1.4

Description

Use case #

1.1.1 Acquisition of Co-Occurrences

Goal in Context

Identify and recognize faces from historical images then associate the faces with Entitypedia entities.

Scope & Level

Component Summary

Role of Human Hybrid Conventional Computation

Automatic processing is used to detect faces in historical images. Subsequently, the generic crowd verifies the detected face positions.

Preconditions

Entitypedia entities for persons, places and organizations Image files showing historical photos

Success End Condition

Images with faces that are associated to a result list. Every element of the list is associated to an Entitypedia Entity

Failed End Condition

No faces can be detected on the images, •

The detected faces were not recognized,

•

The recognized faces cannot be associated to Entitypedia entities,

•

The association between faces and Entitypedia entities cannot be verified

Primary, Secondary Actors

Content injector Crowd experts, Crowd workers

Trigger

An image is selected for processing

DESCRIPTION

Step

Action

Media harvesting and uploading

Provenance checker

CUBRIK Revised Requirements Specifications

Page 32

D2.3 Version 1.0

Use case #

1.1.1 Acquisition of Co-Occurrences

EXTENSIONS

License checker

Entity extraction from metadata

Face detection

CROWD face position validation

Face identification

Step

Branching Action

If the image had been analysed before, the rest of the pipeline is skipped and the user sees the results for the image that had already been processed

SUB-VARIATIONS

3.1.5

Branching Action 1

Content providers can annotate the content of their archives with different kind of permitted uses on a per user/group base through the content provider tool

Images can be â&#x20AC;˘

Harvested from imported archives or

â&#x20AC;˘

Uploaded as bulk or as single files

Sequence Diagram

Figure 15: Image indexation sub-variation Image upload

CUBRIK Revised Requirements Specifications

Page 33

D2.3 Version 1.0

Figure 16: Image indexation sub-variation Archive indexation

3.1.6 â&#x20AC;˘ â&#x20AC;˘

3.1.7

CUbRIK H-demos reference News History H-Demo for provenance identification People Identification H-Demo for face identification

Component List

Process steps

Component Name

Data set creation

Media harvesting and upload

owner CERTH

Partner responsible CERTH L3S FRH

Patrick Aichroth patrick.aichroth@idmt.fraunhofer.de

FRH

Patrick Aichroth patrick.aichroth@idmt.fraunhofer.de

FRH POLIMI

Patrick Aichroth patrick.aichroth@idmt.fraunhofer.de

FRH

Patrick Aichroth patrick.aichroth@idmt.fraunhofer.de

License checker

owner POLMI

Theodoros Semertzidis theosem@iti.gr

Content Provider Tool

Provenance checker

Face recognition

Contact Point

Entity extraction from metadata

CVCE EMPOLIS UNITN

Face detection

POLMI FRH

Marco Tagliasacchi marco.tagliasacchi@polimi.it

Crowd face position validation

POLMI MICT L3S

Marco Tagliasacchi marco.tagliasacchi@polimi.it

Face identification

POLMI FRH

Marco Tagliasacchi marco.tagliasacchi@polimi.it

CUBRIK Revised Requirements Specifications

Lars Wieneke lars.wieneke@cvce.eu

Page 34

D2.3 Version 1.0

3.2

Use case Identity reconciliation

The use case is covered in the interface mock-up (see Figure 7)

3.2.1

Functional requirements

Functional requirement

The application must provide the ability to annotate meta-information for the image

The application must provide the ability to annotate persons that are depicted in the image

I10

Meta-information refer in this case to the image as a source (e.g. date and location where the image was taken) whereas person-related information refer to the identity of the person depicted in the image.

I11

The application must provide an extensible ontology for entities (persons, places and events) that allows to normalize new entities with existing entities and to add new entities

I12

The application must enable users to correct errors from automatic face detection, i.e. mark undetected faces (false negatives) and delete bounding boxes not containing a face (false positives)

I13

The application must provide provenance information for all annotations

I14

The application must integrate a confidence indicator that shows the estimated confidence for results from crowdsourcing tasks and expert annotations Table 7 Functional requirements for 'Identity reconciliation'

CUBRIK Revised Requirements Specifications

Page 35

D2.3 Version 1.0

Identity reconciliation – owner: CVCE CROWD pre-filtering

Entity verification & annotation

Entitypedia integration

Partners

L3S

HOM

UNITN

POLMI

CERTH

EIPCM

CVCE

ENG

EIPCM

DEMO NONE

Task 3.4

Task 10.2

Task 4.2

Ready till end of July

Part Y2 Part Y3

Ready till mid of July

GUI Component

image files

verified identities for face positions on images related to Entitypedia entities

face positions with associated entities

results stored in Entitypedia

Figure 17: Component diagram for the use case ‘Identity reconciliation’

3.2.2

Non-functional requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address precision, rendering time and validation accuracy. The user-based performance indicators refer to comprehension, and perception of incentive to provide annotations, and the perception of effort.

3.2.3

Success criteria

In Table 7 the technical success criteria for identity reconciliation are displayed. Component CROWD pre-filtering

Indicator Crowd Validation Accuracy

Target level > .5

Entity verification & annotation Entity verification & annotation Entity verification & annotation

No. of annotated faces per photograph Manual face annotation precision Component rendering time (ms)

< 200

Remarks Percentage of images correctly validated by the crowd

> .9 < 1000

Table 8 Technical success criteria for Identity reconciliation

CUBRIK Revised Requirements Specifications

Page 36

D2.3 Version 1.0

In Table 8 the user-based performance indicators for identity reconciliation are displayed. Human-computer transaction

Type of user

Indicator

Type of measurement

User sees a list of automatic face recognition results with reliability indicators

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Expert Historian

Comprehension

Questionnaire (self reported, likert scale)

Expert Historian Expert Historian Expert Historian

Perception of effort to provide annotations Perception of usefulness Number of Annotations

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale) Analysis of logs

User annotates metadata and person-based data of documents (e.g. photographs)

Table 9 User-based performance-indicators for Identity reconciliation

3.2.4

Description

Use case #

1.1.2 Identity Reconciliation

Goal in Context

Verify assigned identities or assign other identities to faces.

Scope & Level

Component Summary

Role of Hybrid Human Conventional Computation

First, automatic processing results in a hypothesis about the identity of the person in the image. However, the results of the face recognition process are erroneous and need to be verified by humans. Identity is verified by members of a generic crowd and by experts in case the generic crowd failed. Experts can annotate the images with additional information. Results are stored in Entitypedia.

Preconditions

Image files and face positions in these images with associated Entitypedia entities

Success End Condition

Verified identities for face positions in images related to Entitypedia entities. Storage of the results in Entitypedia.

Failed End Condition

Identities are not verified. Identities do not relate to Entitypedia entities. The results are not stored in Entitypedia

Primary, Secondary Actors

Generic crowd, CVCE researchers â&#x20AC;&#x201C;

Trigger

New results of the face recognition process arrive

DESCRIPTION

Step

Action

CROWD pre-filtering

Entity verification and annotation

Entitypedia Integration

CUBRIK Revised Requirements Specifications

Page 37

D2.3 Version 1.0

Use case #

1.1.2 Identity Reconciliation

EXTENSIONS

Step

Branching Action

â&#x20AC;&#x201C; SUB-VARIATIONS

Branching Action 2

3.2.5

Optional verification for identities that have been verified in step 1

Sequence Diagram

Figure 18: Sequence diagram identity reconciliation

3.2.6

CUBRIK H-demos reference

â&#x20AC;˘

Media Entity Annotation H-Demo

3.2.7

Component List

Process steps

Component Name

Partner responsible

Contact Point

Process Step

Component Name

Partner responsible

Contact Point

Identity reconciliation

CROWD pre-filtering

L3S UNITN EIPCM

Sergiu Chelaru chelaru@l3s.de

Entity verification & annotation

HOM POLMI CVCE EIPCM

Javier Garcia javiergarcia@homeria.com

owner CVCE

CUBRIK Revised Requirements Specifications

Page 38

D2.3 Version 1.0

Process steps

3.3 • • • •

3.3.1

Component Name

Partner responsible

Contact Point

Entitypedia integration

UNITN

Uladzimir Kharkevich uladzimir.kharkevich@disi.unit n.it

Use case 1.2 Indexation of textual information The CVCE collection contains various textual information related to person names, places and time The goal of this use case is to index documents and to extract existing Entitypedia entries. The use case is NOT covered in the interface mock-up This use case will be further refined in Y3

Functional requirements

Functional requirement

I15

Entities must be extracted from text documents in the CVCE collection

I16

The crowd must be able to verify the entities

I17

Verified entities must be entered into Entitiypedia Table 10 Functional requirements for 'Indexation of textual information'

3.3.2

Non-functional Requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements.

3.3.3

Success criteria

Component

Indicator

Entity annotation & Extraction

F-measure

Target level > .85

Remarks

Table 11 Technical success criteria for Indexation of textual information

CUBRIK Revised Requirements Specifications

Page 39

D2.3 Version 1.0

Entity extraction– owner: CVCE

Entity reconciliation – owner: CVCE

Entity annotation & extraction

Expert CROWD verification

Entitypedia integration

Partners

CVCE

EMPOLIS

CVCE

UNITN

ENG

UNITN

ENG

CVCE

POLMI(?)

Connection to the CVCE collection

HOM

DEMO NONE

DEMO Media entity annotation

DEMO NONE

Task 10.2

Task 4.2

GUI Component

text documents

extracted entities related to text elements

entities related to text elements

verified Entitypedia entities related to text elements

Figure 19: Use case 1.2 Indexation of textual information

CUBRIK Revised Requirements Specifications

Page 40

D2.3 Version 1.0

3.3.4

Description

Use case #

Indexation of textural information

Goal in Context

Extract entities (person names, places, organizations and time) from CVCE documents are validated and consolidate with Entitypedia entities

Scope & Level

Component Summary

Role of hybrid human conventional computation

Entities are automatically extracted from CVCE documents. However, they are verified by CVCE experts before they are stored in Entitypedia.

Preconditions

Text documents with unstructured information about persons, events and organisations in time

Success End Condition

Extracted and consolidated Entitypedia entities for person names, places, organizations and time related to text elements in documents

Failed End Condition

Entitypedia entities for person names, places, organizations and time are NOT extracted and consolidated from CVCE documents The entities are NOT related to text elements in documents

Primary, Secondary Actors

CVCE researchers –

Trigger

pre-processing starts

DESCRIPTION

Step

Action

Connection to the CVCE collection

Entity annotation & extraction

Expert CROWD verification

Entitypedia Integration

Step

Branching Action

EXTENSIONS

– SUB-VARIATIONS

Branching Action –

–

CUBRIK Revised Requirements Specifications

Page 41

D2.3 Version 1.0

3.3.5

Sequence Diagram

Figure 20: Sequence diagram indexation of textual information

3.3.6 â&#x20AC;˘

3.3.7

CUBRIK H-demo reference Media Entity Annotation H-Demo

Component List

Process steps

Component Name

Partner responsible

Contact Point

Process step

Component Name

Partner responsible

Contact Point

Entity extraction

Connection to the CVCE collection

CVCE ENG

Ghislain SILLAUME <ghislain.sillaume@cvce.eu>

owner CVCE

Entity annotation & extraction

EMPOLIS UNITN CVCE

Mathias Otto <mathias.otto@empolis.com>

Entity reconciliation

Expert CROWD verification

CVCE ENG POLMI HOM

Lars Wieneke (will be updated) <lars.wieneke@cvce.eu>

Entitypedia integration

UNITN

Uladzimir Kharkevich <uladzimir.kharkevich@disi.unitn.it>

owner CVCE

CUBRIK Revised Requirements Specifications

Page 42

D2.3 Version 1.0

History of Europe: Social Graph Construction

The social graph is constructed by combining verified information from the image and text indexation tasks to build a social graph based on the co-occurrence of persons in images. The idea of co-occurrence could be extended to other types of documents later (either in year 3 or in a follow-up project). The use case can be found in the mock up in Figure 8.

4.1

Functional Requirements

Functional requirement

The application must compute a social graph based on the co-occurrence of person in images

The application must enrich this graph with place-, time-, event-related information. Table 12 Functional requirements for â&#x20AC;&#x2DC;Social graph constructionâ&#x20AC;&#x2122;

4.2

Non-functional Requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address the number of notes and entities, and the time needed for updating Entitypedia and the generation of the social graph.

4.3

Success Criteria

In Table 13 the technical success criteria for social graph construction are displayed. Component

Indicator

Social graph creation

SG Generation time after receiving data from Entitypedia

Social graph creation Social graph creation

Max. number of generated graph nodes Number of entities that can be handled simultaneously Entitypedia updates can be handled in realtime

Social graph creation

Target level < O(n)

Remarks Response Time to a query = Entitypedia response time + SG Generation time n is the number of requested nodes by query

Unlimited > 100,000

1 x RT

Table 13 Technical success criteria for social graph cosntruction

CUBRIK Revised Requirements Specifications

Page 43

D2.3 Version 1.0

Social graph construction – owner: EMPOLIS Entitypedia data provisioning

Social graph creation

Partners

UNITN

QMUL

CERTH

EMPOLIS

ENG

POLMI L3S

DEMO NONE

DEMO X-domain recognition in consumer photos

Task 4.2

Task 6.1, 3.3

Ready

verified entities for face positions on images

Social Graph

Figure 21: Component diagram for the use case ‘Social graph construction’

4.4

Description

Use case #

Social Graph construction

Goal in Context

A social graph is constructed based on the co-occurrence of persons in the historical images.

Scope & Level

Component Summary

Preconditions

Verified Information about co-occurence of persons in historical images, entity definitions in Entitypedia about persons, places, organizations and events

Success End Condition

A social graph is updated and stored

Failed End Condition

The social graph is not updated

Primary,

CUBRIK Revised Requirements Specifications

Page 44

D2.3 Version 1.0

Use case #

Social Graph construction

Secondary Actors Trigger

Previously unknown identities in a image have been verified

DESCRIPTION

Step

Action

Entitypedia data provisioning

Social graph creation

Step

Branching Action

EXTENSIONS

SUB-VARIATIONS

Branching Action -

4.5

Sequence Diagram

Figure 22: Sequence Diagram Social Graph Management

CUBRIK Revised Requirements Specifications

Page 45

D2.3 Version 1.0

4.6 â&#x20AC;˘

4.7

CUBRIK H-demos reference None

Component List

Process steps

Component Name

Social graph construction owner: EMPOLIS

Entitypedia Integration and data provisioning

UNITN CERTH ENG

Uladzimir Kharkevich <uladzimir.kharkevich@disi.unitn.it>

Social graph creation

QMUL POLIMI EMPOLIS L3S

Navid Hajimirza <navid.hajimirza@eecs.qmul.ac.uk>

4.8

Partner responsible

Contact Point

Component specification

Component description

Version

Date

Author

Entitypedia data provisioning

0.1

10.06.2013

Uladzimir Kharkevich

Social graph creation

0.1

04.06.2013

Navid Hajimirza

CUBRIK Revised Requirements Specifications

Page 46

D2.3 Version 1.0

History of Europe: Who is this person?

The use case contains four sub use cases: • • • •

5.1

Visualization of the social graph Query for entities Expert crowd research inquiry Social Graph network analysis toolkit

Use case ‘Visualization of the Social Graph’

In this use case, users can start a visualization of the social graph. It is covered in the mockup.

5.1.1

Functional Requirements

Functional Requirement

The application must visualize persons as nodes and co-occurrences as edges that connect persons through images

The application must represent the amount of images that connect persons through a mutual co-occurence

The application must enable users to access an ego-graph for a specific person

The application must enable users to specify a time range for images that should be taken into account for the visualization of the graph

The application must enable users to select weak ties by defining upper and lower limits for the amount of connecting images in the collection

The application must enable users to select weak ties by defining upper and lower limits for the amount of times a person appears in an image collection

The application must enable users to filter images by giving a upper and lower limit for the amount of persons in an image

The application must enable users to review only images that show two specific persons

All information in the social graph must contain provenance information

W10 The application must focus on the domain of European History W11 The application can provide the ability to export network data Table 14 Functional requirements for 'Visualization of the social graph'

CUBRIK Revised Requirements Specifications

Page 47

D2.3 Version 1.0

Graph Visualization

Partners HOM EIPCM

DEMO NONE Task 10.2

GUI Component

Social Graph

Visualization of the social graph

Images with face positions References to Entitypedia entities

Figure 23: Component diagram for the use case â&#x20AC;&#x2DC;Visualization of the social graphâ&#x20AC;&#x2122;

5.1.2

Non-functional requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address the number of images, the number of nodes shown and rendering time. The user-based indicators concern usefulness and comprehension.

5.1.3

Success criteria

In Table 15 the technical success criteria for visualization of the social graph are displayed. Component

Indicator

Graph visualization Graph visualization Graph visualization Graph visualization

Max. number of processed images Max. number of of shown nodes at once Max. number of shown links per node Initial graph rendering time

CUBRIK Revised Requirements Specifications

Target level > 100,000 > 500

Remarks

> 100 < 15,000 ms Page 48

D2.3 Version 1.0

Component

Indicator

Graph visualization

Dynamic graph rerendering time

Target level < 2000 ms

Remarks

Table 15 Technical success criteria for visualization of the social graph In Table 16 the user-based performance indicators for visualization of the social graph are displayed. Human-computer transaction

Type of user

Indicator

Type of measurement

User views document-based connections between persons

Expert Historian, students Expert Historian, students Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Comprehension of navigation

Questionnaire (retelling), Analysis of logs Questionnaire (self reported, likert scale)

Expert Historian

Perception of efficiency

Expert Historian Expert Historian

Trustworthiness of annotations Perception of relevance

User discovers new information (e.g. identity of a person) by looking at available annotations for documents in the graph

User profiles link to usersâ&#x20AC;&#x2122; professional background and fields of expertise

Perception of usefulness

self reported comparison of time spent on task in comparison to traditional work processes Questionnaire (self reported, likert scale) Analysis of logs

Table 16 User-based performance indicators for visualization of the social graph

5.1.4

Description

Use case #

Visualization of the social graph

Goal in Context

The social graph is visualized

Scope & Level

Component Summary

Preconditions

Images with face positions and associated entities, Social graph, Entitypedia entities for persons, places, organizations and events

Success Condition

End

The graph is visualized

Failed Condition

End

The graph is not visualized

Primary,

CVCE researchers

CUBRIK Revised Requirements Specifications

Page 49

D2.3 Version 1.0

Use case #

Visualization of the social graph

Secondary Actors

Trigger

The user starts the visualization of the social graph

DESCRIPTION

Step

Action

Graph Visualization

Step

Branching Action

EXTENSIONS

SUB-VARIATIONS

Branching Action –

5.1.5

–

Sequence Diagram

Figure 24: Sequence Diagram 3.1 Visualization of the social graph

5.1.6

CUBRIK H-demos reference

None

5.1.7

Component List

Process steps

Component Name

Partner responsible

Contact Point

Visualization of the social graph

Graph visualization

HOM EIPCM

Javier Garcia Morón <javiergarcia@homeria.com>

CUBRIK Revised Requirements Specifications

Page 50

D2.3 Version 1.0

5.2

Use case Query for entities

This component allows users to query for specific components and to visualize the results.

5.2.1 #

Functional requirements Functional requirement

W12 The social graph must be addressable by source (image, text, video, etc.), person, place, event and time/date

W13 The application must allow the user to select the data collections that should be used for the creation of the social graph Table 17 Functional requirements for 'Query for entities' Query execution

Graph visualization

Partners

EMPOLIS

HOM

ENG

EIPCM

HOM CVCE DEMO NONE

DEMO NONE

Task tbd

Task 10.2

READY

GUI Component

Social Graph

Visualization of the results

Images with face positions References to Entitypedia entities

Figure 25: Component diagram for the use case â&#x20AC;&#x2DC;Query for entitiesâ&#x20AC;&#x2122;

5.2.2

Non-functional Requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address the number of processed images and the perception of usefulness.

CUBRIK Revised Requirements Specifications

Page 51

D2.3 Version 1.0

5.2.3

Success criteria

In Table 18 the technical success criteria are displayed. Component

Indicator

Graph visualization

Max. number of processed images

Target level > 100,000

Remarks

Table 18 Technical success criteria for Query for entities In Table 19 the user-based performance indicators for query for entities is displayed. Human-computer transaction

Type of user

Indicator

Type of measurement

User filters the graph by time, location and event

Expert Historian, students

Perception of usefulness

Questionnaire (self reported, likert scale)

Table 19 User-based performance indicators for query for entities

5.2.4

Description

Use case #

Query for entities

Goal in Context

Users can query the social graph for specific entities and the result is then visualized.

Scope & Level

Component Summary

Preconditions

Images with face positions and associated entities, Social graph, Entitypedia entities for persons, places, organizations and events

Success Condition

End

The query is executed and the result is visualized

Failed Condition

End

Either the query is not executed or the result is not visualized

Primary, Secondary Actors

CVCE researchers -

Trigger

The user starts the query

DESCRIPTION

Step

Action

Query execution

Graph visualization

Step

Branching Action

EXTENSIONS

SUB-VARIATIONS

Branching Action â&#x20AC;&#x201C;

â&#x20AC;&#x201C;

CUBRIK Revised Requirements Specifications

Page 52

D2.3 Version 1.0

5.2.5

Sequence Diagram

Figure 26: Sequence Diagram 3.2 Query for entities

5.2.6

CUBRIK H-demos reference

None

5.2.7

Component List

Process steps

Component Name

Partner responsible

Contact Point

Query for entities

Query execution

EMPOLIS ENG HOM CVCE

Mathias Otto <mathias.otto@empolis.com>

Graph visualization

HOM EIPCM

Javier Garcia Mor贸n <javiergarcia@homeria.com>

5.3

Use case Expert CROWD Research Inquiry

In this use case users can verify information by starting research inquiries to a network of Experts. The use case is part of the mock-up (see Figure 10).

5.3.1 #

Functional requirements Functional requirement

W14 The application must integrate the ability to delegate tasks to generic crowds W15 The application must integrate the ability to delegate tasks to the community of expert users provided that work with the HoE app

W16 The application must integrate the ability to delegate tasks to the e-mail contacts of CUBRIK Revised Requirements Specifications

Page 53

D2.3 Version 1.0

Functional requirement the user

W17 The application must integrate the ability to delegate tasks to contacts of the user on social media

W18 For explicit crowd research inquiries the application must allow responding crowd members to indicate that they don't know the correct answer

W19 The application must provide templates for the management of crowdsourcing tasks This includes • • • •

Crowd control (which crowd to ask), Time management (how long will the request remain active), Urgency and Incentives

W20 The application must be able to analyse the occurrence of persons that were not identified

W21 The application must enable users to manage crowdsourcing inquiries in the following way: -

browse, close, select preferred answer

W22 User names must be identifiable and users must have the ability to connect their account with additional information about their background (e.g. public profiles or websites) Table 20 Functional requirements for 'Expert CROWD Research Inquiry’

CUBRIK Revised Requirements Specifications

Page 54

D2.3 Version 1.0

Figure 27: Component diagram for the use case â&#x20AC;&#x2DC;Expert CROWD Research Inquiryâ&#x20AC;&#x2122;

5.3.2

Non-functional Requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. The technical success criteria address the no. of false positives in verified information. The user-based indicators address comprehension, efficiency, usefulness, and trustworthiness.

5.3.3

Success Criteria

In Table 21 the technical success criteria for expert crowd research inquiry are displayed. Component CROWD Research Inquiry

Indicator No. of false positives in verified information for images and entities

Target level

Remarks

< 10%

Table 21 Technical success criteria for expert crowd research inquiry

CUBRIK Revised Requirements Specifications

Page 55

D2.3 Version 1.0

In Table 22 the user-based performance indicators are displayed for expert research inquiry. Human-computer transaction

Type of user

Indicator

Type of measurement

User initiates explicit expertcrowd-sourcing inquiries for uncertain information

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Expert Historian

Perception of efficiency

Self reported comparison of time spent on task in comparison to traditional work processes

Expert Historian

Number of inquiries initiated p. user

Analysis of logs

Expert Historian, social network of experts Expert Historian, social network of experts

Perception of incentive to respond to others' inquiries

Questionnaire (self reported)

Return time < specified urgency (< 1 day for urgent requests, < 2 weeks for all other)

Analysis of logs

Perception of effort

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale)

User responds to explicit expert-crowd-sourcing inquiries

Expert Historian

Trustworthiness of response

Table 22 User-based performance indicators for expert crowd research inquiry

5.3.4

Description

Use case #

Expert CROWD Research Inquiry

Goal in Context

Verification of entity or image information

Scope & Level

Component Summary

Role of human hybrid conventional computation

Experts can verify Entitypedia information by starting a research inquiry and asking other experts in their network to verify information. No automatic processing occurs in this step as this was already part of the indexation use cases.

Preconditions

Images with face positions and associated entities, Social graph, Entitypedia entities for persons, places, organizations and events

Success Condition

End

The information is verified

CUBRIK Revised Requirements Specifications

Page 56

D2.3 Version 1.0

Use case # Failed Condition

Expert CROWD Research Inquiry End

The verification is not completed

Primary, Secondary Actors

CVCE researchers -

Trigger

The user starts a research inquiry

DESCRIPTION

Step

Action

Crowd Research Inquiry

Step

Branching Action

EXTENSIONS

SUB-VARIATIONS

Branching Action –

5.3.5

–

Sequence Diagram

Figure 28: Sequence Diagram 3.1 Visualization of the social graph

5.3.6

CUBRIK H-demos reference

None

5.3.7

Component List

Process steps

Component Name

Who is this person

Crowd Research Inquiry

Partner responsible POLIMI CVCE EIPCM L3S

CUBRIK Revised Requirements Specifications

Contact Point Tech: Marco Tagliasacchi <marco.tagliasacchi@polimi.it> User: Carine Lallemand <Carine.Lallemand@tudor.lu>

Page 57

D2.3 Version 1.0

5.4

Use case 3.4 Social Graph Network Analysis Toolkit

This use case describes the analysis of the social graph and the visualization of the results of this analysis. The use case is part of the mock-up (see Figure 4).

5.4.1 #

Functional requirements Functional requirement

W23 The application can provide a toolkit to explore network analysis features, e.g. – Calculation of shortest path between persons, places and events – Centrality Table 23 Functional requirements for 'Social Graph Network Analysis Toolkit’

Analysis of the social graph

Graph visualization

Partners

CERTH

HOM

EMPOLIS

EIPCM

DEMO NONE

Task 4.2

Task 10.2

GUI Component

Social Graph

Visualization of the social graph

Images with face positions

Results of the analysis

References to Entitypedia entities

Figure 29: Component diagram for the use case ‘Social Graph Network Analysis Toolkit’

CUBRIK Revised Requirements Specifications

Page 58

D2.3 Version 1.0

5.4.2

Non-functional requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. The technical success criteria address network analysis accuracy, and the criteria that were reported in the use case â&#x20AC;&#x2DC;visualization of the social graphâ&#x20AC;&#x2122;.

5.4.3

Success criteria

In Table 24 the technical success criteria for Social graph network analysis toolkit. Component

Indicator

Social graph network analysis toolkit Graph visualization Graph visualization Graph visualization Graph visualization Graph visualization

Accuracy for centrality and shortest pat Max. number of processed images Max. number of of shown nodes at once Max. number of shown links per node Initial graph rendering time Dynamic graph rerendering time

Target level = 100%

Remarks

> 100,000 > 500 > 100 < 15,000 ms < 2000 ms

Table 24 Technical success criteria for social graph network analysis toolkit In Table 25 the user-based performance indicators for Social graph network analysis toolkit. Human-computer transaction

Type of user

Indicator

Type of measurement

User analyzes the graph with a set of network analysis tools

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Table 25 User-based performance indicators for social graph network analysis toolkit

5.4.4

Description

Use case #

3.4 Social Graph Network Analysis Toolkit

Goal in Context

Relationships in the social graph are analysed and visualized

Scope & Level

Component Summary

Preconditions

Images with face positions and associated entities, Social graph, Entitypedia entities for persons, places, organizations and events

Success End Condition

The results of the analysis are shown and the graph is visualized

CUBRIK Revised Requirements Specifications

Page 59

D2.3 Version 1.0

Use case #

3.4 Social Graph Network Analysis Toolkit

Failed End Condition

The results of the analysis are NOT shown and the graph is NOT visualized

Primary, Secondary Actors

CVCE researchers -

Trigger

The user starts the analysis of the social graph

DESCRIPTION

Step

Action

Analysis of the social graph

Visualization of the social graph

Step

Branching Action

EXTENSIONS

SUB-VARIATIONS

Branching Action 1

5.4.5

Different analytical tools are provided

Sequence Diagram

Figure 30: Sequence Diagram 3.1 Social graph analysis toolkit

CUBRIK Revised Requirements Specifications

Page 60

D2.3 Version 1.0

5.4.6

CUBRIK H-demos reference

None

5.4.7

Component List

Process steps

Component Name

Partner responsible

Contact Point

Who is this person

Social Graph network analysis toolkit

CERTH EMPOLIS L3S HOM CVCE

Theodoros Semertzidis theosem@iti.gr

Graph visualization

HOM EIPCM

Javier Garcia Mor贸n javiergarcia@homeria.com

5.4.8

Component specification

Component description

Version

Date

Author

Social Graph network analysis toolkit (aka Graph exploration toolkit, aka: Analysis of the social graph)

1.0

10.06.2013

Dimitris Rafailidis

Graph Visualization

0.1

05.06.2013

Javier Garcia Mor贸n

CUBRIK Revised Requirements Specifications

Page 61

D2.3 Version 1.0

Use case Context Expander

Selected entities are expanded with related media such as newspaper articles, interviews, commentaries and videos.

6.1

Functional requirements

Functional requirement

The application must integrate at least one external data collection

Besides images the application must integrate other document types such as text documents (e.g. letters, newspaper articles), videos and audio recordings

The application must provide links to other archival resources that are related to a specific person Table 26 Functional requirements for 'Context Expander'

Expansion through documents

Expansion through videos

Expansion through images

Partners

EMPOLIS

FRH

L3S UNITN

will be updated with L3S contribution

L3S

DEMO NONE

DEMO News history

DEMO Monuments

Task tbd

Task 5.3

Task 6.3

READY

GUI Component

social graph with related entities

tags

Location of the user tweeted this tweet Timestamp of the tweet (unix timestamp)

(array

Array of hashtags contained in the tweet The name of the user (string)

Table 29 Original Twitter schema New schema Based on the specifications, the new schema follows: Key

Value

Description

String

Id of the ImageObject within CUbRIK

mediaLocator

String

The expanded url of the retrieved image

Descriptions

Array of ContentDescription

ContentDescription used to store the annotations

Provider

ContentProvider

Platform from which the object has been retrieved

Permissions

UsagePermission

Licence that can be used to work with the object

Provenance

CubrikUser

User that has submitted the object

Table 30 New Twitter schema Each ContentDescription returned by Sketchness is characterized by: Key

Value

Description

String

Id of the ContentDescription within CUbRIK

itemAnnotations

TextAnnotation[0..*] LowLevelFeature[1]

The item annotation contains the query-string, the hashtags contained in the tweet, a text and a lowlevelfeature representing

CUBRIK Revised Requirements Specifications

Page 81

D2.3 Version 1.0

the geo coordinates. mediaSegment

TemporalDescription[1]

An annotation containing the date that the object refers to (submission)

Table 31 SketchNess Content Description

7.2.2

Accessibility-related annotations

Original schema The accessibility-related metadata that should be available for the images are the following. These are a set of elements from the Content Description Model, specifically Annotations. Here, they have been refined and specified at the level of detail necessary in order to support the modeling of information that is necessary for accessibility. Metadata for the whole image: Key

Value

Description

Width

image width (pixels)

The width of the image in pixels.

Height

image height (pixels)

The height of the image in pixels.

Brightness

image brightness (number [0-1])

The brightness of the whole image, normalized in the range [0: very dark – 1: very bright].

contrast

image contrast (number [0-1])

The contrast of the image, normalized in the range [0: low contrast – 1: high contrast].

Sharpness

image sharpness (number [0-1])

The sharpness of the image, normalized in the range [0: very blurred – 1: very sharp].

dominant_color

color (RGB)

The dominant color of the image.

color_list

colors (RGB)

All colours appearing in the image.

dominant_color_combination

pair of colors (pair of RGB triples)

The combination of the N most dominant color of the image, exceeding a custom threshold.

red_percentage

percentage (number [01])

The percentage of the red tone in the depicted colors.

color_saturation

color saturation (number [0-1])

The average color saturation in the image, normalized in the range [0: low saturation (grayscale) – 1: high saturation].

contains_bright_backgrounds

boolean (true / false)

A logical value describing if there are bright backgrounds in

CUBRIK Revised Requirements Specifications

Page 82

D2.3 Version 1.0

the image. contains_bright_objects

boolean (true / false)

A logical value describing if there are bright objects in the image.

contains_bright_spots

boolean (true / false)

A logical value describing if there are bright spots in the image.

contains_small_details

boolean (true / false)

A logical value describing if the image contains small objects / details of importance.

contains_text

boolean (true / false)

A logical value describing if the image contains text.

text_size

font size (pixels)

Average size of the diagonal of the detected fonts in pixels

min_object_is_centered

boolean (true / false)

A logical value describing if the main object (i.e. fashion item) in the image is centered in the image.

objects

array of objects, each containing objectspecific metadata (see below)

An array of the objects that are detected in the image. Each object is described by the metadata presented in the table below.

Table 32 Image metadata (original schema) Object-specific metadata: Key

Value

object (pixels)

coordinate

The x-coordinate of the center of the object in the image, in pixels.

object (pixels)

coordinate

The y-coordinate of the center of the object in the image, in pixels.

Width

object width (number)

The width of the object in pixels.

Height

object height (number)

The height of the object in pixels.

Brightness

object brightness (number [0-1])

The brightness of the object, normalized in the range [0: very dark – 1: very bright].

Contrast

object contrast (number [0-1])

The contrast of the object, normalized in the range [0: low contrast – 1: high contrast].

Sharpness

object sharpness (number [0-1])

The sharpness of the object, normalized in the range [0: very blurred – 1: very sharp].

CUBRIK Revised Requirements Specifications

Description

Page 83

D2.3 Version 1.0

color_histogram

RGB color histograms (arrays of numbers for the R, G and B color components)

The R, G and B color histograms of the colors of the object.

texture_descriptor

texture descriptor (array of numbers)

The texture descriptor of the object, as a vector of numbers.

Table 33 Object-specific metadata (original schema) New Schema: Based on the specifications, the new schema follows. The annotation can be represented with a ContentDescription. In the case of image_metadata it is characterized by: Key

Value

Description

String

Id of the ContentDescription within CUbRIK

Name

String

image_metadata, identifying that the ContentDescription is related to the metadata of the whole image

itemAnnotations

LowLevelFeature[0..*] LogicalFeature[0..*]

The item annotation contains width ,height,brightness,contrast,sharpness dominant_color, color_list, dominant_color_combination, red_percentage, color_saturation, text_size as LowLevelFeature. contains_bright_backgrounds, contains_bright_objects,contains_bright_spots, contains_small_details,contain_text, main_object_is_centered as LogicalFeature

Table 34 Image metadata (new schema) In the case of object_specific information it is characterized by: Key

Value

Description

String

Id of the ContentDescription within CUbRIK

Name

String

object_specific, identifying that the ContentDescription is related to the metadata of an object within the image

itemAnnotation s

LowLevelFeature[0.. *]

The item annotation contains x,y,width ,height,brightness,contrast,sharpness,color_histogr am ,texture_descriptor as LowLevelFeature.

Table 35 Object-specific metadata (new schema)

7.2.3

Sketchness Output

The following schema represents the output that Sketchness is able to provide after the intervention of the human operators. It associates to each image object one or more ContentDescription containing the tag of the object recognized in the image and the polyline associated with that tag, representing the contour of the object. CUBRIK Revised Requirements Specifications

Page 84

D2.3 Version 1.0

The output returned by Sketchness is an Object Image with the following fields: Key

Value

Description

String

Id of the ImageObject within CUbRIK

Width

Number (Integer)

Width of the image

Height

Number (Integer)

Height of the image

Filesize

Number (Integer)

Size of the image

Filemime

String

Image format

Filename

String

Name of the original image

mediaLocator

String

The expanded url of the retrieved image

Descriptions

Array of ContentDescription

ContentDescription used to store the annotations

Table 36 Sketchness Object Image Output Each ContentDescription returned by Sketchness is characterized by: Key

Value

Description

Number (Integer)

Id of the ContentDescription within CUbRIK

itemAnnotations

TextAnnotation[1]

The tag associated to the object described by the ContentDescription

mediaSegment

PolylineDescription[1]

An annotation containing points, as specified in PolylineDescription.json

Table 37 Sketchness Content Description

7.3

Overview of the use cases

The list for the Fashion V-App presented below has been defined starting from the user stories that were described in D2.2 Requirements Specifications. The original use cases have been detailed to a large extent and the actions performed by the user and the functionalities available have been described. Moreover, some variations have been introduced as compared to deliverable D2.2. New ideas and possible new interesting functionalities to be provided to the users of the Fashion V-App were introduced. This led to the adaptation of existing user stories and the definition of new ones. The following changes have been made with respect to D2.2: • •

The consortium agreed to discard the use scenario “Funny games” and thus not to proceed with its implementation; New use cases have been added: o “Image crawling from SN”: it is a use case, needed to populate the database and gather information to be used for the trend analysis; o “Trend analysis for image sample” has been added, because it was judge of

CUBRIK Revised Requirements Specifications

Page 85

D2.3 Version 1.0

•

interest the possibility offered to a SMEs user to have insight of trends starting from a sample image; o “Personalised query suggestion” has been added in order to offer additional functionalities to the user of the V-App Some use case have changed slightly the name: o The “Trend Analyser” of the D2.2 changed the name into: “Trend Analysis for category” o The “Search similar images 1a-1” of the D2.2 changed the name into: “Popularity Assessment” o The “What do I wear today?” and “Search similar images 1a-2” of the D2.2 changed the name and were integrated into: “Fashion matching”

A summary of the changes is presented in Table 38. Final use case

Reference in the D2.2

Image crawling from SN

Not present

Trend analysis for category

Present with the name of: “Trend Analyser”

Trend analysis for image sample

Not present: it is a variation of the “Trend Analyser”

Popularity Assessment

Present with the name of Search similar images 1a-1

Fashion matching

Present with the same name of What do I wear today?

Personalised query suggestion

Not present

Table 38 Summary of changes in use case compared to D2.2 The use cases that will be presented in the next section of the document are the following: Image crawling from SN Crawling fashion related images from social networks in order to extract trends on the images and the preferences of the social network users. Crowdsourcing is used for the annotation of the images. The annotations refer to body parts and categories of clothing items, while texture and colour is derived from the images automatically. Trend analysis for category This use case aims at providing the SMEs users with a trend analysis of clothing preferences based on the category the SME user has selected. Crowdsourcing is used for the annotation of the images for body parts and clothing items if automatic body parts detection failed. The trend analysis function analyses the output from Image crawling from social networks in order to present the SME user with a trend analysis report. Trend analysis for image sample The use case aims at providing the SME users with a trend analysis of current clothing preferences using as reference the image (and the clothing category represented there) uploaded by the user himself.The descriptors extractor component extracts the color and the texture from the image the SME user has uploaded, while body parts detector and router classifies the image according to body part. In Image similarity detection, the similarity between images in the database and the uploaded images is computed based on clothing category, texture, and color. Crowdsourcing is used to collect relevance feedback on the search results that come from the image similarity calculations. The refined results are presented to the SME user.

CUBRIK Revised Requirements Specifications

Page 86

D2.3 Version 1.0

Popularity Assessment The use case aims at providing the users with the possibility to upload their own images and ask other users (fan/followers in Social Network) to vote them. Crowd workers verify the image on appropriateness and correctness. An image is appropriate when it does not contain sex or violence. It is correct when the provided annotations for clothing category and colour are correct. If considered appropriate and correct, the Entity Recognition & extraction functionality is used to extract color and texture. If the confidence value is low, the image is then introduced in the Sketchness GWAP in order to extract segments based on human perception. Otherwise, the automatic processing results are used. Finally, the image is annotated with a accessibility parameters. The user can then make the image s/he uploaded available to other V-App users or to friends in social networks. Fashion matching ‘Fashion matching’ retrieves ideas and feedback from the community or from the user’s about an image in which clothes are matched: does this shirt match well with the skirt? The user can either select an image or upload one. After the user uploads a picture, it is checked for appropriateness and the correctness of the user’s annotations (see ‘Request image votes’). The user then selects the lower or the upper part of the body in the image to search for similar images. Those images are shown that are similar to the one uploaded or selected, but that can be an alternative (in texture and style) to the one uploaded or selected by the user. The body part detector and router is used to extract segments related to the upper and lower body part of the body. If the confidence of the results is low, the image is then analysed by Sketchness, otherwise the image is used to provide similar images using the automatic detected segments as input. Users can share their selection of similar images with their network in social media to ask their opinion about the match: does this skirt go with this shirt or not? Personalised query suggestion This technical scenario registers a user profile in order to analyse the behaviour of the V-App user and provide suggestions on what to search for in the app, based on similarity in age, city, and gender. The analysis is performed by the behaviour analyser component.

7.4

Functional Requirements for the SME Innovation V-App

In this section we present high-level functional requirements that explain the core functionality of the SME Innovation vertical application. The requirements will be elaborated in Section and thereafter, which deals with a use case each. #

Functional requirement

SFR1

The application must crawl social images from a social network. Related use cases - Image crawling from social networks

SFR2

The application must identify trends over a time period up to six months Related use cases - Trend analysis for category - Trand analysis for sample

CUBRIK Revised Requirements Specifications

Page 87

D2.3 Version 1.0

Functional requirement

SFR2

The application must reveal aspects of these trends to users such as color and texture. Related use cases - Trend analysis for category - Trend analysis for sample

SFR3

The application must allow the user to upload a photo showing a particular fashion item, and annotate it with a fashion category. Related use cases - Trend analysis for sample

SFR4

The application must allow users to search for images that are similar to an image that is already in the application or to a new image the user uploads. Related use cases - Trend analysis for sample - Fashion matching

SFR5

The application must allow users to match clothes for the upper body parts with clothes for the lower body part and vice versa. Related use cases - Fashion matching

SFR6

The application must allow users to ask for feedback on combinations of garments for the upper and lower body (matches). Users must be able to ask their friends on social networks for feedback. Related use cases - Fashion matching

SFR7

The application must provide the user with suggestions for trend searches, based on the userâ&#x20AC;&#x2122;s past behavior in the application. Related use cases - Personalized query suggestion Table 39 Functional requirements for SME Innovation

The technical success criteria will be discussed in the subsequent sections that deal with individual use cases.

7.5

Non-functional requirements for the SME Innovation V-App

Non-functional requirement

SNFR1

The application shall be accessible and useable for non-technical audiences.

SNFR2

The application shall be well documented and described

SNFR3

The application shall be extensible: it must allow for the addition of other image sources and other social networks to collect feedback on images

SNFR3

The application shall be scalable in the sense that it should guarantee the same performances with the growth of the users and of the database size (number of images)

CUBRIK Revised Requirements Specifications

Page 88

D2.3 Version 1.0

Non-functional requirement

SNFR3

The application shall analyse images in order for the trend analysis to produce valid results.

SFNR4

The application shall have access to a sufficient number of crowdworkers for the Sketchness game, and for the appropriateness verification Table 40 Non-functional requirements for SME Innovation

7.6

Technology acceptance criteria

Overall Usability

Type of user

Indicator

Type of measurement

SME User

Performance expectancy

Questionnaire (self reported, likert scale)

SME User

Effort expectancy

SME User SME User

Attitude toward using technology Task-technology fit

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale)

SME User

Ease of Use

SME User SME User

Interface Errors Critical Incidents

Questionnaire (self reported, likert scale) Analysis of logs Analysis of logs

Table 41 Technology acceptance criteria

7.7

Outlook towards next chapters

The next sections each present a use case story with respect to the functional requirements, sequence diagrams and workflow steps.

CUBRIK Revised Requirements Specifications

Page 89

D2.3 Version 1.0

SME Innovation: Image crawling from SN

The use case aims at crawling fashion related images from social networks in order to extract trends on the images and the preferences of the social network users. Crowdsourcing is used for the annotation of the images for body parts and clothing items. The following image depicts the use case workflow fragment. It is arranged in steps (grey boxes). Each step, in general, involves more than one component usage (orange boxes).

Figure 45: Component diagram for the use case ‘Image crawling from SN’

8.1

Functional requirements

Functional requirement

The application must extract URL’s to images from Social Network that are related to previously-defined fashion item categories.

The application must be able to detect segments related to the upper and lower body of the person wearing the clothes, automatically or through human computation.

The application must be able to extract for a given image the features related to colour and texture belonging to segments of the image.

The application must be able to extract key frames that are most interesting to human viewers from videos by using LikeLines to collect explicit and implicit feedback from the crowd. Table 42 Functional requirements for 'Image crawling from social networks '

CUBRIK Revised Requirements Specifications

Page 90

D2.3 Version 1.0

8.2

Non-functional requirements

In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. They address the number of images, computation time, and the number of clothing categories.

8.3

Success criteria

In Table 43 the technical success criteria for Image crawling from social networks are displayed. Component Image extraction from social network

Indicator No. of domains parsed

Target level > 10

Image extraction from social network

No. of images downloaded per week.

> 100000

Descriptors extractor

Computation time for dominant colors for each image

< 100 msec.

Descriptors extractor

Computation time for texture descriptor for a 100x100 No. of stored URLâ&#x20AC;&#x2122;s to images No. of clothing categories supported

< 100 msec.

Clothing item identification Clothing item identification Relevance feedback

Remarks The component should be able to parse at least 10 most popular photo sharing domains in order to extract shared images from the retrieved image URLâ&#x20AC;&#x2122;s. The component should be able download at least 100000 images per week to have a significant sample of the relevant content shared through the social network .

>= 100,000 >= 6

Each category that we add in the VApp adds a significant amount of data that we collect and process.

< 1 week Time required to collect implicit feedback on the most interesting key frames in a video from the crowd

Table 43 Technical success for image crawling from social networks

CUBRIK Revised Requirements Specifications

Page 91

D2.3 Version 1.0

8.4

Description

Use case #

Image crawling from SN

Goal in Context

The use case aims at crawling images fashion related from the Social Network in order to extract trends on the images and the preferences of the Social Network users.

Scope & Level

Detailed at component level Summary of the components used.

Role of Hybrid Human Conventional Computation

Automatic processing takes place in two fors: the descriptors extractor component extracts the color and the texture from fashion images, while body parts detector and router classifies the image according to body part. In Skecthness, the crowd is involved: a drawing game (a GWAP) is used to identify and annotate clothing types in images.

Preconditions

The images must be CC licensed, and they can be retrieved to gather analysis on the metadata and shown in the web application. To ensure respecting the image license, the images extracted from Twitter have to be presented with a reference to the source, while the images extracted from Flickr will be filtered previously using the API available.

Success End Condition

The images crawled have been successfully annotated, segmented and their features extracted

Failed End Condition

The images cannot be segmented or the feature extracted because of their low quality

Primary, Secondary Actors

No users of the application are involved. Social network users providing popularity info for the images User coming from the crowd are involved in the annotation and in the manual segmentation

Trigger

None

DESCRIPTION

Step

Action

o Extraction of images from SN: the functionality aims at extracting images from Social Network in order to gather information useful to analyse trends. The functionality is performed through the usage of the following components: o Image extraction from Social Network component: the component crawls the Social Network (Twitter) and extracts the images fashion related and the information associated for each category defined. The images are put in the database for the following processing of the segments and the features. o Implicit filter feedback Likelines: the component extracts keyframes (images) fashion related starting from videos uploaded in YouTube crawled by the

CUBRIK Revised Requirements Specifications

Page 92

D2.3 Version 1.0

Use case #

Image crawling from SN “Image extraction from Social Network component”. It is based on Likelines, a tool that collects implicit feedback from users concerning portions of a video that are of particular interests 2

• The Entity Recognition & extraction functionality aims at analyzing the images extracting the features and the segments identifying the category of the clothes present in the images. The functionality is performed by three components: o Body part detector and router component: it analyses the image using a software algorithm extracting the segments related to the upper and lower body part of the body. o Sketchness: it analyses the image using human actions and extracting segments done by clothes contours. o The Descriptors extractor analyses the images and the segments, extracting for each segment the features of the color and the texture. • Accessibility annotation worlflow step analyses the images in order to extract accessibility related metadata: o Accessibility component extract the accessibility-related annotation from the uploaded images and it associates the images to their own annotations.

EXTENSIONS

The images are then put in the database and made available to the vertical application

Step

Branching Action

SUB-VARIATIONS

CUBRIK Revised Requirements Specifications

Branching Action

Page 93

D2.3 Version 1.0

8.5

Sequence Diagram

Figure 46: " Image crawling from SN "

CUBRIK Revised Requirements Specifications

Page 94

D2.3 Version 1.0

Note: Sequence diagram naming is following the same convention of use case naming.

8.6

CUBRIK H-demos reference

The H-demos designed and used to demonstrate the capability of implementing some of functionalities needed by the V-Application are: Likelines: it demonstrated the viability of using video fashion related in order to extract key frames fashion related. The H-demos is used by the Implicit Feedback filter component Accessibility aware Relevance feedback: it aims to enhance the process of multimedia content harvesting in such a way that it will be later able to provide an accessibility related indicator (i.e. confidence factor), regarding the level of accessibility and usability it offers to specific groups of users.

8.7

Components List

In the following table the association between workflow steps and components involved in the given use case “Image crawling from SN” is reported. Workflow Step Name

Component Name

Extraction of images from S.N. Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Image extraction from Social Network

CERTH

Theodoros Semertzidis' <theosem@iti.gr>

Implicit feedback filter LIKELINES

TUD

‘Martha Larson’ <M.A.Larson@tudelft.nl>

Entity recognition & extraction Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Descriptors extractor

CERTH

Theodoros Semertzidis <theosem@iti.gr>

Body parts detector and router

QMUL

‘Markus Brenner’ <markus.brenner@eecs.qm ul.ac.uk>

Sketchness

POLIMI

Luca Galli <luca.galli@polimi.it>

Accessibility

CERTH

Anastasios Drosou drosou@iti.gr

Accessibility Annotation workflow step Owner: CERTH Resp: Anastasios Drosou drosou@iti.gr

Partner responsible

Contact Point

Extraction of Images from SN The “Extraction of Images from SN” workflow step is responsible of coordinating the “extraction of images from SN” (EISN) component and “Implicit feedback filter - LikeLines” (LL) component in order to collect multimedia content shared through twitter. The EISN component will fetch a stream of multimedia URLs and metadata shared through twitter. From them, only the youtube URLs will be rooted to the LL component were keyframe images will be extracted as the most representative/interesting keyframes of the video. This CUBRIK Revised Requirements Specifications

Page 95

D2.3 Version 1.0

workflow step will provide a never ending stream of multimedia content URLs and the corresponding metadata.

Figure 47: Extraction of Images from SN sequence diagram

Entity recognition and Extraction The “Entity recognition and Extraction” workflow step will be the SMILA pipeline that will orchestrate the steps for pre-processing the images coming in to the system. First each image will be passed from the “Upper and lower body parts detector” component to extract upper and lower body part bounding boxes as well as confidence scores about the computed bounding boxes. These confidence levels will then be used to decide whether this image should be discarded as very noisy, sent directly to the descriptor extraction component to compute multimedia features, or pushed to the Sketchness component. In case of Sketchness component the outcome segmentation will also be pushed to the descriptor extraction component. The output of the workflow step is a set of multimedia descriptors, regions of interest as well as annotations for the processed image.

CUBRIK Revised Requirements Specifications

Page 96

D2.3 Version 1.0

Figure 48: Entity recognition and Extraction sequence diagram

Accessibility annotation The Accessibility Annotation workflow step is responsible for the extraction of accessibilityrelated features from images, in order for the accessibility-aware reranking of the search results to be possible. During accessibility annotation, the images pass through the Accessibility component, which contains methods that extract accessibility-related information from the images (brightness, contrast, dominant color etc. of the whole image as well as of recognized objects in the image). This information is encoded as a set of accessibility score values, one for each of the supported impairments. The accessibility scores take values in the [0, 1] range, with 0 meaning that the image is not accessible for users having the respective impairment, while 1 means that the image is accessible. The accessibility scores are added as annotation to the image record.

CUBRIK Revised Requirements Specifications

Page 97

D2.3 Version 1.0

SME Innovation: Trend analysis for category

The use case aims at providing the SMEs users with a trend analysis of clothing preferences related to a category to be selected. The following image depicts the use case workflow fragment. The trend analysis function analyses the output from Image crawling from social networks in order to present the SME user with a trend analysis report based on the popularity of the images, the most popular colours and textures in the social Network.

Figure 49: Component diagram for the use case â&#x20AC;&#x2DC;Trend analysis for categoryâ&#x20AC;&#x2122; This use case is visualized in the mock-up (Figure 34 and Figure 35).

9.1 #

Functional requirements Functional requirement

The application must enable users to select a category of clothes for which they request a trend analysis. The application must provide users with an analysis of fashion trends based on the analysis of social network images, providing the users with the information about the preferred color and texture, the most popular images and a time trend of the colour for the category selected. Table 44 Functional requirements for 'Trend analysis for category'

9.2

Non-functional Requirements

In the next section measurable technical and user-based success criteria are formulated that are perceived as operationalized non-functional requirements. They address response time and computation time. From the user perspective, comprehension and quality of the results are taken into account. CUBRIK Revised Requirements Specifications

Page 98

D2.3 Version 1.0

9.3

Success Criteria

Component

Indicator

Trend analysis

Time granularity

Target level < 1 week

Trend analysis

Maximal trend analysis period

>3 months

Trend analysis

Response time

<1s

Trend analysis

The computation time of the analysis

<= 1 day

Descriptor extraction

PSNR

> 25 dB for each image

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image

< 100 msec

Descriptor extraction

Explanation The analysis component should be able to get information for small periods of time that are less than one week to provide information that is not possible to be extracted without automated processes using only e.g. fashion magazines or cool hunters Due to the large volumes of data to be processed and possible data bursts due to e.g. social events the processing and storage costs for periods larger that few months rise critically. This is the time needed for the component to return results to the UI. No computations will be performed in this step but precomputed results will be fetched. The trend analysis component should be able to compute fresh content or existing content to give answers to new queries posed by an SME. This criterion should be applicable to any trend analysis granularity from the minimum to the maximum. Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability of the final results.

< 100 msec

Table 45 Technical success criteria for Trend analysis for category Human-computer transaction V-Apps user selects clothing category and time span selection for trend prediction V-Apps user interprets the results:

Type of user

Indicator

SME user

Comprehension

SME user

Comprehension

CUBRIK Revised Requirements Specifications

Page 99

Type of measurement Questionnaire (retelling)

Log analysis (time spent, critical

D2.3 Version 1.0

Human-computer transaction 1. Analysis 2. Popular photos

Type of user

Indicator

Perceived quality of the results 1. Timeliness 2. Usefulness for SMEuser's goals

Type of measurement incidents in UI) Questionnaire (retelling) Questionnaire (selfreported; likert scale) Absolute and relative to other systems

Table 46 User-based performance criteria for trend analysis for category

9.4

Description

Use case #

Trend analysis for category

Goal in Context

The use case aims at providing the SMEs users with a trend analysis of clothing preferences related to a category to be selected.

Scope & Level

Detailed at component level. Summary of the components used.

Preconditions

“Image crawling from SN” use case is a precondition of the present use case, because the database of the V-App has to contain images extracted from Social Network with preferences associated (i.e. popularity of the images, preferences on the color and the textures of the clothing categories )

Success Condition

End

The trend analysis is performed and the SME user is provided with a trend analysis.

Failed Condition

End

The trend analysis doesn’t provide any results.

Primary, Secondary Actors

SMEs users of the application.

Trigger

The SME users select the option to have a trend insight on a category

DESCRIPTION

Step

Action

• SME users ask for an insight on fashion trend: the SME users of the V-App select the category of the clothes on which they want to have an insight regarding the fashion trend.

• The Trend Analysis functionality performs the analysis of the trends on the selected category through the following component: • The Trend Analyser component uses the category selected in order to perform a trend analysis using the information already retrieved from the Twitter along with the corresponding images (i.e. metadata) as processed in the use case “extraction of the images

CUBRIK Revised Requirements Specifications

Page 100

D2.3 Version 1.0

Use case #

Trend analysis for category from SNâ&#x20AC;?, providing to the V-App the information about the preferred color and texture, the most popular images and a time trend of the color for the category selected.

EXTENSIONS

SUB-VARIATIONS

9.5

â&#x20AC;˘ The Report to SMEs functionality shows the report to the users through the user interface.

Step

Branching Action

Sequence Diagram

Figure 50: Trend analysis for category Note: Sequence diagram naming is following the same convention of use case naming.

CUBRIK Revised Requirements Specifications

Page 101

D2.3 Version 1.0

9.6

CUBRIK H-demos reference

No H-Demos are related to this use case.

9.7

Components list

This section reports a table with the association between workflow steps and components involved in the given use case â&#x20AC;&#x153;Trend analysis for categoryâ&#x20AC;?. Workflow STEP Name

Component Name

Trend Analysis Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Trend analyser

Partner responsible CERTH

Contact Point Theodoros Semertzidis' <theosem@iti.gr>

Trend analyser The Trend analyser step will be the pipeline that will get input from previous pipelines in the workflow and trigger trend analysis component computations. Finally the computed results will be requested via this pipeline.

CUBRIK Revised Requirements Specifications

Page 102

D2.3 Version 1.0

Figure 51: Trend analyser sequence diagram

CUBRIK Revised Requirements Specifications

Page 103

D2.3 Version 1.0

10.

SME Innovation: Trend Analysis for sample

The use case aims at providing the SME users with a trend analysis of current clothing preferences using as reference the image (and the clothing category represented there) uploaded by the user himself.

Figure 52: Component diagram for the use case ‘Trend Analysis for sample’

10.1 Functional requirements #

Functional requirement

The application must allow SME-users to upload an image. The image should be related to fashion and should contain clothes

The application must allow SME-users to annotate the images with the category of the clothes

The application must be able to search for images that are similar to the sample the user has uploaded. Images are similar when their clothing category, colour, and texture correspond

The application must allow crowdworkers to assign a relevance score to each of the images; the application must present the re-ordered set of images to the users.

The application must provide users with an analysis of fashion trends, similar to ‘Trend analysis for category’, but calculated on the basis of the sample image uploaded by the user.

Table 47 Functional requirements for Trend analysis for sample' CUBRIK Revised Requirements Specifications

Page 104

D2.3 Version 1.0

10.2 Non-functional requirements In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements. The technical criteria address computation time, response time, precision and scalability. The user-based criteria address the perceived incentive, perceived usefulness, and perceived quality of the results that are returned.

10.3 Success criteria Component

Indicator

Target level

Explanation

Image crawling from SN Image crawling from SN

No. of images/second

>= 10

The component should be able download at least 10 images per second

No. of domains to track

>10

Trend analysis

Time granularity

< 1 week

Trend analysis

Maximal trend analysis period

>3 months

Trend analysis

Response time

<1s

Trend analysis

The computation time of the analysis

<= 1 day

Descriptor extraction

PSNR

> 25 dB for each image

The component should be able to parse at least 10 photo sharing domains in order to extract shared images from the retrieved image URLs The analysis component should be able to get information for small periods of time that are less than one week to provide information that is not possible to be extracted without automated processes using only e.g. fashion magazines or cool hunters Due to the large volumes of data to be processed and possible data bursts due to e.g. social events the processing and storage costs for periods larger that few months rise critically. This is the time needed for the component to return results to the UI. No computations will be performed in this step but precomputed results will be fetched. The trend analysis component should be able to compute fresh content or existing content to give answers to new queries posed by an SME. This criterion should be applicable to any trend analysis granularity from the minimum to the maximum. Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability to the final results.

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image

< 100 msec

Descriptor extraction

CUBRIK Revised Requirements Specifications

< 100 msec

Page 105

D2.3 Version 1.0

Component

Indicator

Target level

Body parts detector and router Body parts detector and router Relevance feedback Relevance feedback

Runtime complexity

<5s

Computed detection score

Precision along key dimension No. of key dimensions implemented Response time

> .75

<= 1 m

In case of a pre-scheduled crowd.

Time for estimation of response time

<= 1 m

If no crowd is pre-scheduled, the system will return an estimate of the response time within one minute. The length of the response time is based on an estimation of the size of the available crowd.

No. of annotated images

> 100,000

Server reply time

<3s

No. of users that can use the component simultaneously

> 1000

Relevance feedback Relevance feedback

Image similarity detection Image similarity detection Image similarity detection

Explanation

A detection score below 10 indicates a very unlikely detection, and above 20-30 a very likely detection.

>= 3

Table 48 Technical success criteria for trend analysis for sample Humancomputer transaction V-Apps user annotates image

Type of user

Indicator

Type of measurement

SME user

Perceived effort of annotating images

Questionnaire (self-reported, Likert scale)

V-Apps user interprets results

SME user

Comprehension

Questionnaire (retelling)

Perceived quality of the results

Questionnaire (self-reported: absolute + relvative to quality of existing platforms - Google, fashion sites): Pipeline evaluation Questionnaire (self-reported; Likert scale)

Usefulness for SMEuser's goals

Table 49 User-based performance criteria for trend analysis for sample

CUBRIK Revised Requirements Specifications

Page 106

D2.3 Version 1.0

10.4 Description Use case #

Trend Analysis for sample

Goal in Context

Role of Hybrid Human Conventional Computation

Automatic processing takes place in two forms: the descriptors extractor component extracts the color and the texture from the image the SME user has uploaded, while the body parts detector and router classifies the image according to body part. In Image similarity detection, the similarity between images in the database and the uploaded images is computed based on clothing category, texture, and color. Crowdsourcing is used to collect relevance feedback on the search results that come from the image similarity calculations.

Scope & Level

Detailed at component level Summary of the components used

Preconditions

Success End Condition

The search similarity produces a set of similar images The trend analysis is performed and the SME user is provided with a trend analysis.

Failed End Condition

The trend analysis doesn’t provide results.

Primary, Secondary Actors

SMEs users of the application.

Trigger

The user selects the option to have a trend insight on category and images that match in terms of similarity the ones to be uploaded

DESCRIPTION

Step

Action

• SME user uploads an image A: the V-App SME user uploads an image A, with the aim to have an insight of both the trends related to the category represented by the image A and the trends related to other images representing the category of clothes similar to the one uploaded.

• Image annotation: the V-App SME users annotate the image they have just uploaded with the clothing categories present in the picture, then they provide information about the category they are interested in.

• Analysis of Lower/Upper part and features extraction: the image is processed with the aim of extracting the segments and the features contained in the category

CUBRIK Revised Requirements Specifications

Page 107

D2.3 Version 1.0

Use case #

Trend Analysis for sample represented. The process is performed by the:

•

The Lower and Upper Body part component: it analyses the image using a software algorithm extracting the segments related to the upper and lower body part of the body.

•

The Descriptors extractor analyses the images and the segments, extracting for each segment the features about the color and the texture.

• The system retrieves similar images and shows them to the user through the step Search Similar Images filtered by the crowd. The functionality is performed by the following components: o

Image similarity detection: it identifies images similar to the one uploaded by the users, by searching into the database of the application. The utilized criteria are: (a) same category of clothes contained, (b) similar color and similar texture of the category selected.

Relevance feedback: starting from the set of similar images identified by the “Image Similarity Detection” component, the “Relevance Feedback” component filters and reorders the set of images, ranking the similarity according the human perception about similarity.

•

• The Report to SMEs functionality shows the report to the user through the user interface.

The Trend Analysis functionality performs the analysis of the trend on the selected category through the following component: o The Trend Analyser component uses the category selected in order to perform a trend analysis using the information already retrieved from Twitter and the corresponding images extracted and processed in the use case “Extraction of the Images from SN”, providing to the V-App information like the preferred color and texture, the most popular images and the time trend of the color for the category selected. o Using the images resulting from the “Search Similar Images” workflow step, as selected by the SME users, the component also retrieves information about the images previously crawled from twitter along with their traces associated information about popularity. Thus, the component, provides to the VApp information about such images and their popularity.

CUBRIK Revised Requirements Specifications

Page 108

D2.3 Version 1.0

Use case #

Trend Analysis for sample

EXTENSIONS

Step

SUB-VARIATIONS

Branching Action

Branching Action 4a

The image is put in the database

CUBRIK Revised Requirements Specifications

Page 109

D2.3 Version 1.0

10.5 Sequence Diagram The sequence diagram for ‘trend analysis for sample’ is displayed in Figure 53.

Figure 53: Sequence diagram for ‘trend analysis for sample’ Note: Sequence diagram naming is following the same convention of use case naming. CUBRIK Revised Requirements Specifications

Page 110

D2.3 Version 1.0

10.6 Components List This section reports a table with the association between workflow steps and components involved in the given use case “Trend Analysis for sample”. Workflow Step Name

Component Name

Analysis of Lower/upper part and features extraction Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Descriptors extractor

CERTH

Theodoros Semertzidis <theosem@iti.gr>

Body parts detector and router

QMUL

‘Markus Brenner’ <markus.brenner@eecs.qm ul.ac.uk>

Search similar images filtered by the crowd Owner: EMPOLIS Resp: Mathias Otto mathias.otto@empolis .com

Image detection

EMPOLIS

Mathias Otto mathias.otto@empolis.com

Trend Analysis Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Trend analyser

similarity

Relevance Feedback

Partner responsible

TUD

CERTH

Contact Point

‘Martha Larson’ <M.A.Larson@tudelft.nl>

Theodoros Semertzidis' <theosem@iti.gr>

Table 50 Components List

Analysis of Lower/upper part and features extraction The “Entity recognition and Extraction” workflow step will be the SMILA pipeline that will orchestrate the steps for pre-processing the images coming in to the system. First each image will be passed from the “Body parts detector and router” component to extract upper and lower body part bounding boxes as well as confidence scores about the computed bounding boxes. These confidence levels will then be used to decide whether this image should be discarded as very noisy or sent directly to the descriptor extraction component to compute multimedia features. The output of the workflow step is a set of multimedia descriptors, regions of interest as well as annotations for the processed image.

CUBRIK Revised Requirements Specifications

Page 111

D2.3 Version 1.0

Figure 54: Extraction of Images from SN sequence diagram

Search similar images filtered by the crowd The workflow step aims at providing the user with a set of similar images to the one he selected. The functionality is performed through 2 steps: firstly a set of similar images on category, color and texture basis is extracted from the database; in the second step such set of images is processed by the crowd in order to provide the user with a reordered and reranked set of images, through the usage of human perception of similarity.

Trend analyzer The Trend analyser step will be the pipeline that will get input from previous pipelines in the workflow and trigger trend analysis component computations. Finally the computed results will be requested via this pipeline.

CUBRIK Revised Requirements Specifications

Page 112

D2.3 Version 1.0

11.

SME Innovation: Popularity Assessment

The use case aims at providing the users with the possibility to upload their own images and ask other users (fan/followers in Social Network) for their feedback.

Figure 55: Component diagram for the use case ‘Popularity assessment’

11.1 Functional Requirements #

Functional requirement

The application must allow users to upload images. The images must be related to fashion and must contain clothes.

The application must enable users to annotate the uploaded images with clothing category and color.

The application must request crowd workers to verify the image on appropriateness and correctness. An image is appropriate when it does not depict sex or violence. It is correct when the provided annotations for clothing category and colour are correct

The application must collect unstructured feedback by publishing the link to the images via social networks through user’s own social network profile or to other users of the V-App. Table 51 Functional requirements for 'Popularity Assessment'

11.2 Non-functional requirements In the next section measurable technical and user-based success criteria are formulated that are perceived as operationalized non-functional requirements. The technical success criteria CUBRIK Revised Requirements Specifications

Page 113

D2.3 Version 1.0

address computation time, recall, and accuracy. The user-based indicators address perception of usefulness, effort, privacy, and the incentive to provide feedback.

11.3 Success Criteria Component

Indicator

Target level

Explanation

Descriptor extraction

PSNR

> 25 dB for each image

Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability to the final results.

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image Runtime complexity

< 100 msec

Descriptor extraction Body parts detector and router Upper/lower body parts detector Sketchness Sketchness

Conflict manager

< 100 msec

<5s

Computed detection score

Garment recognition: recall Garment segmentation: average pixel accuracy Answer/query ratio per day

> .5

Conflict manager

A detection score below 10 indicates a very unlikely detection, and above 2030 a very likely detection. @FP rate < .1

> .6

< 0.8

< 2.5

Expected Responding Time: Answer/Query Ratio per day: 80% (Based on Answering search queries with CrowdSearcher) Average Answer to Question ratio: 2.5 replies within 1 day (Based on Answering search queries with CrowdSearcher)

Table 52 Technical success criteria for popularity assessment Human-computer transaction V-Apps user extends the existing collection with own uploads

Type of user

Indicator

SME user

Perceived usefulness

V-Apps user annotates the image

SME User

Type of measurement Questionnaire (self reported, likert scale)

No. of uploaded images Perception of effort

Questionnaire (self reported, likert scale)

No. of annotated images Crowd worker verifies images wrt

Crowd worker

CUBRIK Revised Requirements Specifications

Comprehension

Page 114

Log analysis (time spent, critical D2.3 Version 1.0

Human-computer transaction appropriateness and the provided annotations

Type of user

Indicator

Error rate Perception of incentive to provide annotations Return time V-Apps user provides feedback

SME User

Perception of incentive to provide feedback No. of feedback requests

Type of measurement incidents in UI) Questionnaire (retelling) No. of false positives/negatives Questionnaire (self reported, likert scale) Log analysis Questionnaire (self reported, likert scale)

Table 53 Technical success criteria for popularity assessment

11.4 Description Use case #

Popularity assessment

Goal in Context

The use case aims at providing the users with the possibility to upload their own images and ask other users (fan/followers in Social Network) to vote them.

Scope & Level

Detailed at component level Summary of the components used.

Role of Hybrid Human Conventional Computation

First, the fashion image s/he uploaded is verified- by the crowd on two points: the correctness of the annotations (clothing type, color, and texture), and the appropriateness of the image with regard to the occurrence of sexual explicitness and violence. The image can then broadcast to other V-App users and social networks to collect feedback. Automatic processing occurs after this step. The Entity Recognition & extraction functionality is used to extract color and texture. If the confidence value is low, the image is then introduced in the Sketchness GWAP in order to extract segments based on human perception. Otherwise, the automatic processing results are used.

Preconditions

The V-App user have to be registered and logged in: the uploading of the images is allowed just to logged users.

Success Condition

End

The image uploaded is judged appropriate, the images are voted by other V-App user.

Failed Condition

End

The image uploaded is judged not appropriate.

Primary, Secondary Actors

Individual users of the application. Users coming from the crowd are involved in the verification of the images and other users (fan/followers of the individual user in the social networks) are involved in voting on the images.

Trigger

The V-App individual users select the possibility to upload images

CUBRIK Revised Requirements Specifications

Page 115

D2.3 Version 1.0

Use case #

Popularity assessment and ask vote

DESCRIPTION

Step

Action

• Image upload: the V-App user uploads a picture fashion-related in the V-Application.

• Image Annotation: the V-App user annotates the image specifying which clothing category can be seen in the image and what its color is (the color is represented by a 8-bit color palette).

• Verification of the image: this functionality consists in the verification of appropriateness of the images and of the correctness of the annotation provided by the user. This functionality is performed by the component: • Crowd verification: the component checks the image and through the professional crowd verifies if the image contains violence or sexual scenes and if the category and the color specified by the initial uploading user are correct.

• After the recognition of the image, provided the image is judged appropriate, the user can make his own image available to the V-App users. Using the V-App users’ feedback functionality he can ask the opinion about the match he selected to: o o

EXTENSIONS

other users of the application to his friends and contacts of the social network through the Conflict Manager component. In that case the user has to provide his credentials of Twitter and Facebook.

Step

Branching Action

CUBRIK Revised Requirements Specifications

Page 116

D2.3 Version 1.0

Use case #

Popularity assessment and the segments, extracting for each segment the features of the color and the texture. â&#x20AC;˘ The image is put in the qualified database. 5a

SUB-VARIATIONS

â&#x20AC;˘ Accessibility annotation workflow step analyses the images in order to extract accessibility related metadata: o Accessibility component extracts from the uploaded images the accessibility-related annotation (brightness, contrast, dominant color etc. of the whole image as well as of recognized objects in the image), associating to the images the corresponding accessibility-metadata. It will support the reranking of the images for the supported impairments. Branching Action

CUBRIK Revised Requirements Specifications

Page 117

D2.3 Version 1.0

11.5 Sequence Diagram

Figure 56 Sequence diagram for â&#x20AC;&#x2DC;Popularity assessmentâ&#x20AC;&#x2122; Note: Sequence diagram naming is following the same convention of use case naming.

CUBRIK Revised Requirements Specifications

Page 118

D2.3 Version 1.0

11.6 CUBRIK H-demos Reference The H-demo designed and used to demonstrate the capability of implementing some of functionalities needed by the V-Application is: Accessibility aware Relevance feedback: it aims to enhance the process of multimedia content harvesting in such a way that it will be later able to provide an accessibility related indicator (i.e. confidence factor), regarding the level of accessibility and usability it offers to specific groups of users.

11.7 Components List This section reports a table with the association between workflow steps and components involved in the given use case “Popularity assessment”. Workflow Step Name

Component Name

Verification of the Image Owner: ENG Resp: Marilena Lazzaro <Marilena.Lazzaro@e ng.it>

Crowd verification

Accessibility Annotation workflow step Owner: CERTH Resp: Anastasios Drosou drosou@iti.gr Entity recognition & extraction Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

V-App users’ feedback Owner: POLIMI Resp: Marco Tagliasacchi <marco.tagliasacchi@ polimi.it>

Partner responsib le

Contact Point

TUD

‘Otto Chrons’ < otto.chrons@microtask.com >

Accessibility

CERTH

‘Anastasios Drosou’ drosou@iti.gr

Descriptors extractor

CERTH

‘Theodoros Semertzidis’ <theosem@iti.gr>

Body parts detector and router

QMUL

‘Markus Brenner’ <markus.brenner@eecs.qmul. ac.uk>

Sketchness

POLIMI

‘Luca Galli’ <luca.galli@polimi.it>

Conflict Manager

POLIMI

‘Marco Tagliasacchi’ <marco.tagliasacchi@polimi.it >

CUBRIK Revised Requirements Specifications

Page 119

D2.3 Version 1.0

Verification of the image Verification of the image: this functionality consists of the verification of appropriateness of the images and of the correctness of the annotation provided by the user. This functionality is performed by the component Crowd Verification. Images that are evaluated as appropriate can be stored to be further processed. Otherwise they are discarded.

Entity recognition and Extraction The “Entity recognition and Extraction” workflow step will be the SMILA pipeline that will orchestrate the steps for pre-processing the images coming in to the system. First each image will be passed from the “Body parts detector and router” component to extract upper and lower body part bounding boxes as well as confidence scores about the computed bounding boxes. These confidence levels will then be used to decide whether this image should be discarded, sent directly to the descriptor extraction component to compute multimedia features, or pushed to the Sketchness component. In case of Sketchness component the outcome segmentation will also be pushed to the descriptor extraction component. The output of the workflow step is a set of multimedia descriptors, regions of interest as well as annotations for the processed image.

CUBRIK Revised Requirements Specifications

Page 120

D2.3 Version 1.0

Figure 57: Entity recognition and Extraction sequence diagram

V-App user feedback The V-App user feedback is the last workflow step of the use case. Its role is to retrieve the set of images and their associated metadata, as processed by the â&#x20AC;&#x153;Entity Recognition and Extraction workflow stepâ&#x20AC;? (either automatic or derived from the contribution of the crowd), and provide human feedback over them. To gather feedback from the crowd, the Conflict Manager has first to be configured to specify the kind of question that has to be submitted (Binary - Like/Dislike, Textual description Preferences over a Colour...) to the crowd and where it has to be propagated. The questions can be propagated to several different social networks at the same time or by generating a special link that has to be sent to specific list of users by email (e.g. based on competences); the configuration step must be triggered by another component. Once the evaluation task has been configured, it can be started by the other components by calling the appropriate API (Open Task). The Conflict Manager will automatically take care of collecting the results from the crowd and

CUBRIK Revised Requirements Specifications

Page 121

D2.3 Version 1.0

will update the information contained in the database that has been chosen to store the metadata associated to the images if enquired about it. Since the conflict manager uses REST calls, the calls to and from the component will be handled by a SMILA pipelet.

CUBRIK Revised Requirements Specifications

Page 122

D2.3 Version 1.0

12.

SME Innovation: Fashion matching

The following image depicts the use case workflow fragment. It is arranged in steps (grey boxes). Each step, in general, involves more than one component usage (orange boxes).

Figure 58: Component diagram for the use case ‘Fashion Matching’ 1/2

Figure 59: Component diagram for the use case ‘Fashion matching’ 2/2

CUBRIK Revised Requirements Specifications

Page 123

D2.3 Version 1.0

12.1 Functional requirements #

Functional requirement I1

The application must enable users to select an image from the database, based on which a set of similar images is returned.

The application must allow users to upload images. The images should be related to fashion and should contain clothes and/or accessories.

The application must allow users to annotate the uploaded images with clothing category.

The application must allow users to select the body part for which they would like to receive matching images

The application must provide the users with similar images based on the body part they selected

The application must enable users to ask for votes for this selection of images via the V-App publishing the link via social networks through their own social network profile.

The application must accept a set of accessibility-annotated images as its input

The application must output a reranking of the set of images that promotes those that are most accessible for the specific user

The application must allow the user to provide feedback concerning the accessibility either of the individual results, or of the whole reranking

I10

The application must be able to update the user impairment profile by using the user feedback Table 54 Functional requirements for 'Fashion matching'

12.2 Non-functional requirements In the next section measurable technical and user-based success criteria are formulated that are perceived as operationalized non-functional requirements. The technical success criteria cover precision, response time, scalability, and changes in user profile values with regard to accessibility. The user-based criteria cover comprehension, perceived usefulness, and the perception of the incentives for SME users and crowdworkers.

12.3 Success criteria Component

Indicator

Target level

Relevance feedback

Precision along key dimension No. of key dimensions implemented Response time

> .75

Relevance feedback Relevance feedback Relevance feedback

Time for estimation of response time

CUBRIK Revised Requirements Specifications

Explanation

>= 3 <= 1 m <= 1 m

In case of a pre-scheduled crowd. If no crowd is pre-scheduled, the system will return an estimate of the response time within one minute. The length

Page 124

D2.3 Version 1.0

Component

Indicator

Target level

Explanation of the response time is based on an estimation of the size of the available crowd.

Image similarity detection Image similarity detection Image similarity detection Descriptor extraction

Descriptor extraction

Body part detector and router Body part detector and router

No. of annotated images Server reply time No. of users that can use the component simultaneously PSNR

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image Runtime complexity

> 100,000 <3 seconds > 1000

> 25 dB for each image <2s

<2s

<5s

Computed detection score

> .5

Conflict manager

Garment recognition: recall Garment segmentation: average pixel accuracy Answer/query ratio

<= 80%

Conflict manager

Answer/question ratio

<= 2.5

Accessibility filtering

Change in impairment related parameters after between 10 and 20 feedback loops

< 15%

Accessibility filtering

Change in values in user profile values after 50 feedback loops Percentage of accessibility issues covered by the top 20 search results

< 5%

Sketchness Sketchness

Accessibility filtering

CUBRIK Revised Requirements Specifications

Dominant Colors extraction should keep quantization noise low

A detection score below 10 indicates a very unlikely detection, and above 20-30 a very likely detection. @FP rate < .1

> .6

> 50%

Expected Responding Time: Answer/Query Ratio per day: <80% (Based on Answering search queries with CrowdSearcher) Average Answer to Question ratio: 2.5 replies within 1 day (Based on Answering search queries with CrowdSearcher) An initial estimation of these parameters will be provided at the registration phase. Their convergence to the actual impairment level of the user will be achieved via a relevant feedback procedure.

The 20 first results returned by the submitted query should meet the requirements of at least one the two impairments,

Page 125

D2.3 Version 1.0

Component

Indicator

Accessibility filtering

Target level

Processing for changing between different rankings Time required for initial accessibility related filtering

< 2 s.

Explanation so as to be visible by the user. i.e. percentage of the accessibility issues. After relevance feedback

< 1 s.

Table 55 Technical success criteria for fashion matching Human-computer transaction Crowd worker identifies clothing item via Sketchness

Type of user

Indicator

Crowd worker

Comprehension

User satisfaction / affective response

Crowd worker verifies images wrt appropriateness and the provided annotations

Crowd worker

Comprehension

Perception of incentive

Return time

Type of measurement Log analysis on time spent, critical incidents in UI Avg. no of hits played by individual players Questionnaire (selfreported, Likertscale) Log analysis on time spent, critical incidents in UI Questionnaire (selfreported, Likert scale) Questionnaire (selfreported, Likert scale) Log analysis

V-Apps user chooses the upper or lower body part of his/her own image

SME user

Comprehension

Log analysis on time spent, critical incidents in UI Questionnaire (retelling)

V-Apps user searches images similar to the clothes in the selected body part of their uploaded image

SME user

Comprehension

Log analysis on time spent, critical incidents in UI

Perceived quality of the search results

Questionnaire (selfreported: absolute + relvative to quality of existing platforms Google, fashion

CUBRIK Revised Requirements Specifications

Page 126

D2.3 Version 1.0

Human-computer transaction

Type of user

Indicator

Perceived usefulness of the matching V-Apps user reviews the systemâ&#x20AC;&#x2122;s outfit match suggestion for their uploaded image, i.e. clothes in tagged body part

SME user

Type of measurement sites): evaluation at pipeline level Questionnaire (selfreported)

Comprehension

Log analysis on time spent, critical incidents in UI Questionnaire (retelling)

Perceived quality of the search results Perceived usefulness of the matching

Questionnaire (selfreported) Questionnaire (selfreported)

Table 56 User-based performance indicators for fashion matching

12.4 Description Use case #

Expert CROWD Research Inquiry

Goal in Context

The user wants to retrieve ideas and feedback from the community or from his friends/fans about a match of clothes or other fashion images in the database.

Scope & Level

Detailed at component level. Summary of the components used.

Human hybrid conventional computation

First, the fashion image the V-App user uploaded and annotated is verified- by the crowd on two points: the correctness of the annotations (clothing type, dominant color, and texture), and the appropriateness of the image with regard to the occurrence of sexual explicitness and violence. Alternatively, the user can select an image within the application. Automatic processing occurs after this step. The Entity Recognition & extraction functionality is used to extract color and texture. If the confidence value is low, the image is discarded. Otherwise, the automatic processing results are used. In Image similarity detection, the similarity between images in the database and the selected image is computed based on clothing category, texture, and color. Crowdsourcing is used to collect relevance feedback on the search results that come from the image similarity calculations.

Preconditions

The user is registered; The database has to contain images to allow the V-App to perform similar images searches. The fashion database contains images with lower/upper body part of the body clothes; The user logged in.

CUBRIK Revised Requirements Specifications

Page 127

D2.3 Version 1.0

Use case #

Expert CROWD Research Inquiry

Success Condition

End

The user find a match of interest and ask the preferences of the other users

Failed Condition

End

The user doesn’t find a match of lower/upper part of the body clothes, the user don’t ask other users their feedback The search similarity doesn’t return any results. The image is not voted for.

Primary, Secondary Actors

Individual users. Individual users friends/fans in Social Network, other individual users of the application

Trigger

The user select the possibility to create a match

DESCRIPTION

Step

Action

1 (branch A)

• Image selecting: the V-App user select a picture already present in the database of the V-Application. The picture can contain an image of a complete part of the body clothes (a match of lower and upper part of the body clothes) or a lower or upper part of the body clothes.

1.1 B)

• Image upload: the V-App user uploads a picture in the V-Application. The picture can contain an image of a complete part of the body clothes (a match of both lower and upper part of the body clothes) or a lower or upper part of the body clothes.

(branch

1.2

• Image Annotation: the V-App user annotates the image specifying which the category of the clothes represented is and which the related color is (the color is represented by a 8-bit color palette)

(branch B)

1.3 B)

(branch

• Verification of the image: this functionality consists in the verification of appropriateness of the images and of the correctness of the annotation (category and color) provided by the user. This functionality is performed by the component: • Crowd verification: the component checks the image and through the professional crowd verifies if the image depicts violence or sexual scenes and if the category and the color specified by the user are correct. Images that are valuated as appropriate can be stored to be further processed (1.3B.a), otherwise they are discarded (1.3B.b) The user does not have to wait for the response to get feedback before s/he can continue with the next step. However approval is necessary before step 8.

CUBRIK Revised Requirements Specifications

• Upper/Lower part selection by the user: the user selects the lower or the upper part of the bodies of the image to be used for similar search. He asks the Page 128

D2.3 Version 1.0

Use case #

Expert CROWD Research Inquiry system to provide similar images of the selected part of the body. In order to do this the analysis of Lower/upper part is needed.

Analysis of Lower/Upper part and features extraction: the image provided by the user is processed with the aim of extracting the segments related to the upper /lower body part of the body selected by the user and the relative features. This process is done online, The extraction is performed by the o The Body part detector and router component: it analyses the image using a software algorithm extracting the segments related to the upper and lower body part of the body. If the confidence is low the image will be discarded. o The Descriptors extractor analyses the images and the segments, extracting for each segment the features of the color and the texture.

â&#x20AC;˘

â&#x20AC;˘ The system retrieves similar images and shows them to the user through the functionality Search Similar Images. The functionality is performed by the following components: o Image similarity detection: it identifies image similar to the one upload by the user or to the fragments extracted as described in step 4.1, finding them into the database of the application. The criteria used are: same category, similar color. The objective is providing images similar but that can represent an alternative (in texture and style) to the one selected by the user. o Relevance feedback: starting from the set of similar images identified by the image similarity detection component, the Relevance Feedback component filters and reorders the set of images, ranking the similarity through the usage of human perception of similarity.

CUBRIK Revised Requirements Specifications

â&#x20AC;˘

The Accessibility filtering step provides the reordering function of the images through the Page 129

D2.3 Version 1.0

Use case #

Expert CROWD Research Inquiry following component: •

Accessibility (component) rerank the set of retrieved results, so as to promote those query results that are most accessible for the user. For this purpose, the system retrieve an impairment related profile for each user that has activate the accessibility mode.

• Matching of the retrieved images functionality allow the user to match the lower and the upper body part he prefers using his image and the set of images provided by the system search functionality.

• The user makes available his own image to the other user and using the V-App users’ feedback functionality he can ask the opinion about the match he selected to the other users of the application and/or: o To his friends and contacts of the social network through the Conflict Manager component (the user has to provide his credentials of Twitter and Facebook)

EXTENSIONS

Step

Branching Action

1.3B.a

Condition: the image is judge appropriate. • The image is put in the database, to be analyzed through the Entity Recognition & extraction functionality. The functionality is performed by three components: • The Body part detector and router component: it analyses the image using a software algorithm extracting the segments related to the upper and lower body part of the body. If the confidence of the results is low, the image is then analysed by Sketchness, otherwise directly by the Full image identification component. • Sketchness: it analyses the image using human actions and extracting segments done by clothes contours. • The descriptors extractor analyses the images and the segments, extracting for each segment the features of the color and the texture. The image is analized in order to extract accessibility related metadata (Accessibility use case): The Accessibility component extract the accessibilityrelated annotation from the uploaded images and it

CUBRIK Revised Requirements Specifications

Page 130

D2.3 Version 1.0

Use case #

Expert CROWD Research Inquiry associates the images to their own annotations. 1.3B.b

SUBVARIATIONS

The image is discarded

Branching Action 1A

â&#x20AC;˘

CUBRIK Revised Requirements Specifications

Select Images: The user may have an image with just a lower OR an upper part of the body clothes and wants to search any other clothes possibly matches with the one of his image. So the user select the images by category criterion asking to the system to provide the images present in the database for the chosen category

Page 131

D2.3 Version 1.0

12.5 Sequence Diagram In Figure 60 and Figure 61 the sequence diagrams for ‘fashion matching’ are displayed.

Figure 60: Sequence Diagram for ‘fashion matching’ 1/2

CUBRIK Revised Requirements Specifications

Page 132

D2.3 Version 1.0

Figure 61: Sequence Diagram for â&#x20AC;&#x2DC;fashion matchingâ&#x20AC;&#x2122; 2/2 Note: Sequence diagram naming is following the same convention of use case naming.

CUBRIK Revised Requirements Specifications

Page 133

D2.3 Version 1.0

12.6 CUBRIK H-demos reference The H-demo designed and used to demonstrate the capability of implementing some of functionalities needed by the V-Application is: Accessibility aware Relevance feedback: it aims to enhance the process of multimedia content harvesting in such a way that it will be later able to provide an accessibility related indicator (i.e. confidence factor), regarding the level of accessibility and usability it offers to specific groups of users.

12.7 Components List This section reports a table with the association between workflow steps and components involved in the given use case “Fashion matching”. Workflow STEP Name

Component Name

Verification of the Image Owner: ENG Resp: Marilena Lazzaro <Marilena.Lazzaro@eng.it>

Crowd verification

Accessibility Annotation workflow step Owner: CERTH Resp: Anastasios Drosou drosou@iti.gr Entity recognition & extraction Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Partner responsible

Contact Point

TUD

Otto Chrons <otto.chrons@microtask.com>

Accessibility

CERTH

Anastasios Drosou drosou@iti.gr

Descriptors extractor

CERTH

Theodoros Semertzidis <theosem@iti.gr>

Body parts detector and router

QMUL

‘Markus Brenner’ <markus.brenner@eecs.qmul.a c.uk>

Sketchness

POLIMI

Luca Galli <luca.galli@polimi.it>

Analysis of Lower/upper part and features extraction Owner: CERTH Resp: Michalis Lazaridis <lazar@iti.gr>

Descriptors extractor

CERTH

Theodoros Semertzidis <theosem@iti.gr>

Body parts detector and router

QMUL

‘Markus Brenner’ <markus.brenner@eecs.qmul.a c.uk>

search similar images filtered by the CROW Owner: EMPOLIS Resp: Mathias Otto mathias.otto@empolis.com

Image detection

V-App users’ feedback

Conflict Manager

CUBRIK Revised Requirements Specifications

similarity

EMPOLIS

POLIMI

Page 134

Mathias Otto mathias.otto@empolis.com

Marco Tagliasacchi

D2.3 Version 1.0

Workflow STEP Name

Component Name

Partner responsible

Owner: POLIMI Resp: Marco Tagliasacchi <marco.tagliasacchi@polimi.it>

Contact Point <marco.tagliasacchi@polimi.it>

CUBRIK Revised Requirements Specifications

Page 135

D2.3 Version 1.0

Figure 62: Entity recognition and Extraction sequence diagram

Verification of the image This functionality consists in the verification of appropriateness of the images and of the CUBRIK Revised Requirements Specifications

Page 136

D2.3 Version 1.0

correctness of the annotation provided by the user. This functionality is performed by the component Crowd verification. Images that are valuated as appropriated can be stored to be further processed, otherwise they are discarded

Analysis of Lower/upper part and features extraction The Analysis of Lower/upper part and features extraction” workflow step will be the SMILA pipeline that will orchestrate the steps for pre-processing the images coming in to the system. First each image will be passed from the “Body parts detector and router” component to extract upper and lower body part bounding boxes as well as confidence scores about the computed bounding boxes. These confidence levels will then be used to decide whether this image should be discarded as very noisy or sent directly to the descriptor extraction component to compute multimedia features. The output of the workflow step is a set of multimedia descriptors, regions of interest as well as annotations for the processed image.

Figure 63: Analysis of Lower/upper part and features extraction sequence diagram

Search similar images The workflow step aims at providing the user with a set of similar images to the one he selected. The functionality is performed extracting similar images on category, color and texture basis from the database, in order to provide a set of images similar.

Accessibility Filtering The Accessibility Filtering workflow step is responsible for reranking a set of search results, so that the most accessible ones for the specific user are promoted. In order for the reranking to be performed, two kinds of information are used: the accessibility-related annotation of the presented images (which is extracted from the images during image upload and annotation) and the user impairment profile. The user impairment profile is a vector of values in the range [0, 1] encoding the user’s CUBRIK Revised Requirements Specifications

Page 137

D2.3 Version 1.0

degree of impairment in each of the supported types of disabilities. A value of 0 means that the user does not have the respective disability, while a value of 1 means that the user is fully disabled in the respective impairment. Using this information, the Accessibility Filtering step calculates a set of Pareto-optimal rankings for the results for the various impairments and selects the one which is most appropriate for the user, according to his/her profile. The results are presented to the user according to the selected reranking. As a last step, the user is able to provide feedback concerning the accessibility either of the individual results, or of the whole reranking. The system uses this feedback to update the user impairment profile, so that it is closer to the actual profile of the user, and to eventually present a more appropriate reranking of the results.

V-App users’ feedback The V-App users’ feedback is the last workflow step of the What I wear today use case. Its role is to retrieve the set of images and their associated metadata, as processed by the “Entity recognition and Extraction workflow step” (either automatic or derived from the contribution of the crowd), and provide human feedback over them. To gather feedback from the crowd, the Conflict Manager has first to be configured to specify the kind of question that has to be submitted (Binary - Like/Dislike, Textual description Preferences over a Colour...) to the crowd and where it has to be propagated. The questions can be propagated to several different social networks at the same time or by generating a special link that has to be sent to specific list of users by email (e.g. based on competences); the configuration step must be triggered by another component. Once the evaluation task has been configured, it can be started by the other components by calling the appropriate API (Open Task). The Conflict Manager will automatically take care of collecting the results from the crowd and will update the information contained in the database that has been chosen to store the metadata associated to the images if enquired about it. Since the conflict manager uses REST calls, the calls to and from the component will be handled by a SMILA pipelet.

CUBRIK Revised Requirements Specifications

Page 138

D2.3 Version 1.0

13.

SME Innovation: personalized query suggestion

This technical scenario registers a user profile in order to analyse the behaviour of the V-App user and provide suggestions on what to search for in the app, based on similarity in age, city, and gender.

Figure 64: Component diagram for the use case ‘ Personalized query suggestion’

13.1 Functional requirements #

Functional requirement

The application must allow users to create a user profile, consisting of username, password, and accessibility-related information.

The system must provide the user with suggestions for searches, based on the behaviour of the user in the past, other users that use the application at that time, and other users from the same city. Table 54 Functional requirements for ‘Personalized query suggestion’

13.2 Non-functional requirements In the next section measurable technical success criteria are formulated that are perceived as operationalized non-functional requirements.

CUBRIK Revised Requirements Specifications

Page 139

D2.3 Version 1.0

13.3 Success criteria Component

Indicator

Target level

Explanation

Behaviour analyser

No. of suggestions provided to the user % suggestions used by users

Behaviour analyser

Reponse time

< 2 s.

Behaviour analyser

No. of input queries needed before providing suggestions

The number of suggested queries after In a global statistical analysis the users should be found to use at least 20% of the suggested queries. The additional delay that is allowed to be inserted by the processing for the recommended queries should not exceed 2 sec The behaviour analyzer should be able to return recommendation results, based on the global user trends, even when the user is logged in with a totally untrained profile.

Behaviour analyser

> 20%

Table 57 Technical success criteria for personalized query suggestion Human-computer transaction User inspects the search suggestions presented in the dashboard

Type of user

Indicator

SME User

Perceived quality of the query suggestions

Type of measurement Questionnaire (selfreported; Likert scale)

Table 58 User-based performance indicators for personalized query suggestion

13.4 Description Use case #

Personalized query suggestion

Goal in Context

The use case aims at providing the individual users with the most preferred actions previously selected by him or by other individual users having similar characteristics (similar age, same city, same gender).

Scope & Level

Detailed at component level Summary of the components used.

Preconditions

The user is registered, The user logged in

Success Condition

End

The user choices one of the action suggested

Failed Condition

End

The user doesnâ&#x20AC;&#x2122;t select any of the preferred choices presented.

CUBRIK Revised Requirements Specifications

Page 140

D2.3 Version 1.0

Use case #

Personalized query suggestion

Primary, Secondary Actors

Individual users of the application.

Trigger

The user logs in

DESCRIPTION

Step

Action

• User login: the V-App individual user makes login in the VApplication. The step is based on the following component: o The User profiling component checks the credentials of the user and allow him to access to the app. If the user is not registered, the component allow him to register himself into the application, storing into the database the information related to the user profile, including the accessibility-related ones.

• The User behaviour prediction step suggests to the users searches that could be of interest to them. The step is based on the following component: o The Behavior analyzer component analyses the behaviours of the users according the information gathered on their past activities and on the activities of other related users; The component analyse the log file in order to trace the behaviour of: - the user in his past access to the application - other users accessing at the same time - other users accessing from the same place (city, country)

• The Human-computer transaction step registers the actions the user is taking and sends the information to the Behaviour analyser for the further analysis. The step is based on the following component: • The CMS for Human Computation, analyses and store the actions performed by the user (select a category, select a specific image) and send it to the Behaviour analyser component for further analysis.

CUBRIK Revised Requirements Specifications

Page 141

D2.3 Version 1.0

13.5 Sequence Diagram

Figure 65: Personalized query suggestion Note: Sequence diagram naming is following the same convention of use case naming.

13.6 CUBRIK H-demos reference No H-Demos are related to this use case.

13.7 Components List This section reports a table with the association between the workflow steps and components involved in the given use case â&#x20AC;&#x153;Personalized query suggestionâ&#x20AC;?. Workflow Name

STEP

Component Name

User Login Owner: POLIMI Resp: Luca Galli luca.galli@polimi.it

User component

User Behavior Prediction Owner: CERTH Resp: Anastasios Drosou drosou@iti.gr

Behavior analyzer

CUBRIK Revised Requirements Specifications

profiling

Partner responsib le

Contact Point

CERTH

Luca Galli luca.galli@polimi.it

CERTH

Anastasios Drosou drosou@iti.gr

Page 142

D2.3 Version 1.0

Human-computer transaction Owner: POLIMI Resp: Luca Galli luca.galli@polimi.it

CMS for Human Computation

POLIMI

Luca Galli luca.galli@polimi.it

User login The user Login workflow step allows the user to access to the application, providing him the functionalities to register himself or to login if already registered. The functionalities are provided through the component User profiling component.

User behaviour prediction The User Behaviour Prediction workflow step is responsible for providing query recommendations to users based on time and location context. As soon as the user logs in to the system, the system retrieves stored history of this user or other users and displays recommendations based on the searches that this user or the other users have committed at similar times and places as the current user.

Human-computer transaction The Human-computer transaction workflow step receipts the choice of the users and sends the information to the Behaviour analyser for the further analysis to detect the query to be suggested.

CUBRIK Revised Requirements Specifications

Page 143

D2.3 Version 1.0

14.

Summary of success criteria

14.1 User-based performance indicators User-based performance indicators are defined on two levels: • •

The level of the V-App The level of the use case

14.1.1 V-App level technology acceptance criteria Technology acceptance is the focus of the evaluation on the level of the V-App. In Table 59 an overview of the technology acceptance criteria is displayed. As discussed in Section 1 the indicators are based on the UTAUT framework (Venkatesh et al., 2003). Type of indicator Technology acceptance indicators

Overall Usability

Indicator

Type of measurement

Performance expectancy

Questionnaire (self reported, likert scale)

Effort expectancy Attitude toward using technology Task-technology fit

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale)

Ease of Use Interface Errors Critical Incidents

Questionnaire (self reported, likert scale) Analysis of logs Analysis of logs

Questionnaire (self reported, likert scale)

Table 59 Technology acceptance indicators The indicators are the same for History of Europe and for SME Innovation. In History of Europe the target users are the expert historians, while the SME’s are targeted in the SME Innovation application.

14.1.2 History of Europe Use case Acquisition of Co-Occurences Human-computer transaction

Type of user

Indicator

Type of measurement

User extends the existing collection with own photographs/uploads

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

User validates/defines the position of faces in images

Expert Historian / generic crowd

Perception of effort of manual face validation

Questionnaire (self reported, likert scale)

CUBRIK Revised Requirements Specifications

Page 144

D2.3 Version 1.0

Use case Identity reconciliation Human-computer transaction

Type of user

Indicator

Type of measurement

User sees a list of automatic face recognition results with reliability indicators

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Expert Historian

Comprehension

Questionnaire (self reported, likert scale)

Expert Historian Expert Historian Expert Historian

Perception of effort to provide annotations Perception of usefulness Number of Annotations

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale) Analysis of logs

User annotates metadata and person-based data of documents (e.g. photographs)

Use case Indexation of textual information None. Use case Social graph construction None. Use case Visualization of the social graph Human-computer transaction

Type of user

Indicator

Type of measurement

User views document-based connections between persons

Expert Historian, students Expert Historian, students Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Comprehension of navigation

Questionnaire (retelling), Analysis of logs Questionnaire (self reported, likert scale)

Expert Historian

Perception of efficiency

Expert Historian Expert Historian

Trustworthiness of annotations Perception of relevance

User discovers new information (e.g. identity of a person) by looking at available annotations for documents in the graph

User profiles link to usersâ&#x20AC;&#x2122; professional background and fields of expertise

CUBRIK Revised Requirements Specifications

Perception of usefulness

Page 145

self reported comparison of time spent on task in comparison to traditional work processes Questionnaire (self reported, likert scale) Analysis of logs

D2.3 Version 1.0

Use case Query for entities Human-computer transaction

Type of user

Indicator

Type of measurement

User filters the graph by time, location and event

Expert Historian, students

Perception of usefulness

Questionnaire (self reported, likert scale)

Use case Expert crowd research inquiry Human-computer transaction

Type of user

Indicator

Type of measurement

User initiates explicit expertcrowd-sourcing inquiries for uncertain information

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Expert Historian

Perception of efficiency

Self reported comparison of time spent on task in comparison to traditional work processes

Expert Historian

Number of inquiries initiated p. user

Analysis of logs

Expert Historian, social network of experts Expert Historian, social network of experts

Perception of incentive to respond to others' inquiries

Questionnaire (self reported)

Return time < specified urgency (< 1 day for urgent requests, < 2 weeks for all other)

Analysis of logs

Perception of effort

Questionnaire (self reported, likert scale) Questionnaire (self reported, likert scale)

User responds to explicit expert-crowd-sourcing inquiries

Expert Historian

Trustworthiness of response

Use case Social graph network analysis toolkit Human-computer transaction

Type of user

Indicator

Type of measurement

User analyzes the graph with a set of network analysis tools

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

CUBRIK Revised Requirements Specifications

Page 146

D2.3 Version 1.0

Use case Context expander Human-computer transaction

Type of user

Indicator

Type of measurement

User finds media items (photographs, videos, documents) from various digital collections

Expert Historian

Perception of usefulness

Questionnaire (self reported, likert scale)

Expert Historian

Perception of Provenance (public sources like Flickr or Youtube vs. renowned digital collections)

Questionnaire (self reported, likert scale)

14.1.3 Fashion Image crawling from social networks None Trend analysis for category Human-computer transaction V-Apps user selects clothing category and time span selection for trend prediction V-Apps user interprets the results: 1. Analysis 2. Popular photos

Type of user

Indicator

SME user

Comprehension

SME user

Comprehension

Type of measurement Questionnaire (retelling)

Perceived quality of the results 1. Timeliness 2. Usefulness for SMEuser's goals

Log analysis (time spent, critical incidents in UI) Questionnaire (retelling) Questionnaire (selfreported; likert scale) Absolute and relative to other systems

Table 60 User-based performance criteria for trend analysis for category Trend analysis for sample Humancomputer transaction V-Apps user annotates image

Type of user

Indicator

Type of measurement

SME user

Perceived effort of annotating images

Questionnaire (self-reported, Likert scale)

V-Apps user interprets results

SME user

Comprehension

Questionnaire (retelling)

CUBRIK Revised Requirements Specifications

Page 147

D2.3 Version 1.0

Perceived quality of the results

Usefulness for SMEuser's goals

Questionnaire (self-reported: absolute + relvative to quality of existing platforms - Google, fashion sites): Pipeline evaluation Questionnaire (self-reported; Likert scale)

Table 61 User-based performance criteria for trend analysis for sample Popularity assessment Human-computer transaction V-Apps user extends the existing collection with own uploads

Type of user

Indicator

SME user

Perceived usefulness

V-Apps user annotates the image

SME user

No. of uploaded images Perception of effort

Type of measurement Questionnaire (self reported, likert scale)

Questionnaire (self reported, likert scale)

No. of annotated images

Crowd worker verifies images wrt appropriateness and the provided annotations

Crowd worker

Perception of incentive to provide annotations Return time

Log analysis (time spent, critical incidents in UI) Questionnaire (retelling) No. of false positives/negatives Questionnaire (self reported, likert scale) Log analysis

Perception of incentive to provide feedback

Questionnaire (self reported, likert scale)

Comprehension

Error rate

V-Apps user provides feedback

Fashion consumer/ SME User

No. of feedback requests

Table 62 Technical success criteria for popularity assessment Fashion matching Human-computer transaction Crowd worker identifies clothing item via Sketchness

Type of user

Indicator

Crowd worker

Comprehension

User satisfaction / affective response

CUBRIK Revised Requirements Specifications

Page 148

Type of measurement Log analysis on time spent, critical incidents in UI Avg. no of hits played by individual players Questionnaire (selfD2.3 Version 1.0

Human-computer transaction

Type of user

Indicator

Type of measurement reported, Likertscale)

Crowd worker verifies images wrt appropriateness and the provided annotations

Crowd worker

Comprehension

Log analysis on time spent, critical incidents in UI Questionnaire (selfreported, Likert scale) Questionnaire (selfreported, Likert scale) Log analysis

Perception of incentive

Return time V-Apps user chooses the upper or lower body part of his/her own image

SME user

Comprehension

Log analysis on time spent, critical incidents in UI Questionnaire (retelling)

V-Apps user searches images similar to the clothes in the selected body part of their uploaded image

SME user

Comprehension

Log analysis on time spent, critical incidents in UI

Perceived quality of the search results

Questionnaire (selfreported: absolute + relvative to quality of existing platforms Google, fashion sites): evaluation at pipeline level Questionnaire (selfreported)

Perceived usefulness of the matching V-Apps user reviews the systemâ&#x20AC;&#x2122;s outfit match suggestion for their uploaded image, i.e. clothes in tagged body part

SME user

Comprehension

Log analysis on time spent, critical incidents in UI Questionnaire (retelling)

Perceived quality of the search results Perceived usefulness of the matching

Questionnaire (selfreported) Questionnaire (selfreported)

Table 63 User-based performance indicators for fashion matching

CUBRIK Revised Requirements Specifications

Page 149

D2.3 Version 1.0

Personalized query suggestion Human-computer transaction User inspects the search suggestions presented in the dashboard

Type of user

Indicator

SME User

Perceived quality of the query suggestions

Type of measurement Questionnaire (selfreported; Likert scale)

Table 64 User-based performance indicators for personalized query suggestion

14.2 Technical success criteria 14.2.1 History of Europe In the tables below we summarize the technical success criteria for History of Europe. Acquisition of co-occurrences Component

Indicator

Target level < 100 ms;

Content provider tool

avg processing time

Content provider tool Copyright aware Crawler

heap memory usage usedBandwidth / availableBandwidth

Provenance checker

avg processing time < 100 ms

< 100 ms

Provenance checker License checker

heap memory usage < 80 MB avg processing time < 20 ms

< 80 MB

Face detection

heap memory usage < 80 MB Face detection: Recall

Face identification

Precision (top-1) Precision (top-10)

> .15 > .35

< 30 MB > .7

< 20 ms

> .50 (@FP rate < .05)

Remarks includes file operations (read/write), hashing, XML document validation and XML-Signature creation (regardless of size and number of processed content items) bandwidth efficiency: assumption: storage is fast enough; if lower: limiting end must be provider bandwidth limiting) includes file operations (read/write), hashing, database query based on hash (against empty HSQL reference database), provenance info XML creation (regardless of size and number of processed content items) includes file operations (read/write), XML document validation, XML-Signature validation, mapping license to platform permissions, creating permissions-info XML (regardless of size and number of processed content items) The figure refers to unrestricted conditions, including non-frontal faces, varying illumination, varying size, occlusions, film grain, etc.). FP rate .05 means that .05X wrongly detected faces were found in X tested images. An algorithm that performs face identification based on a random guess would achieve top-1 precision equal to .004 (ground truth with 250 different classes â&#x20AC;&#x201C; individuals)

Table 65 Technical success criteria for 'Acquisition of co-occurrences'

CUBRIK Revised Requirements Specifications

Page 150

D2.3 Version 1.0

Identity reconciliation Component

Indicator

CROWD prefiltering Entity verification & annotation Entity verification & annotation Entity verification & annotation

Crowd Validation Accuracy

Target level > .5

No. of annotated faces per photograph

< 200

Manual face annotation precision

> .9

Component rendering time (ms)

< 1000

Remarks Percentage of images correctly validated by the crowd

Table 66 Technical success criteria for Identity reconciliation Indexation of textual information Component

Indicator

Entity annotation & Extraction

F-measure

Target level > .85

Remarks

Table 67 Technical success criteria for Indexation of textual information Social graph construction Component

Indicator

Social graph creation

SG Generation time after receiving data from Entitypedia

Social graph creation Social graph creation

Max. number of generated graph nodes Number of entities that can be handled simultaneously Entitypedia updates can be handled in realtime

Social graph creation

Target level < O(n)

Remarks Response Time to a query = Entitypedia response time + SG Generation time n is the number of requested nodes by query

Unlimited > 100,000 1 x RT

Table 68 Technical success criteria for social graph cosntruction

CUBRIK Revised Requirements Specifications

Page 151

D2.3 Version 1.0

Visualization of the social graph Component

Indicator

Graph visualization Graph visualization Graph visualization Graph visualization Graph visualization

Max. number of processed images Max. number of of shown nodes at once Max. number of shown links per node Initial graph rendering time Dynamic graph rerendering time

Target level > 100,000 > 500

Remarks

> 100 < 15,000 ms < 2000 ms

Table 69 Technical success criteria for visualization of the social graph Query for entities Component

Indicator

Graph visualization

Max. number of processed images

Target level > 100,000

Remarks

Table 70 Technical success criteria for Query for entities Expert crowd research inquiry Component

Indicator

CROWD Research Inquiry

No. of false positives in verified information for images and entities

Target level < 10%

Remarks

Table 71 Technical success criteria for expert crowd research inquiry Social graph network analysis toolkit Component

Indicator

Social graph network analysis toolkit Graph visualization Graph visualization Graph visualization Graph visualization Graph visualization

Target level = 100%

Remarks

> 100,000 > 500 > 100 < 15,000 ms < 2000 ms

Table 72 Technical success criteria for social graph network analysis toolkit

CUBRIK Revised Requirements Specifications

Page 152

D2.3 Version 1.0

Context expander Component Expansion through video Expansion through video Expansion through images Expansion through textual documents Expansion through textual documents

Indicator Segment matching precision

Target level > .5

Segment matching recall

> .7

Precision at recall .1 Precistion at recall .2 F-measure

> 0.85 > 0.7 > .85

Response time

< .5 s.

Remarks

Table 73 Technical success criteria for context expander

14.2.2 Fashion In the tables below we summarize the technical success criteria for SME Innovation. Image crawling from social networks Component Image extraction from social network

Indicator No. of domains parsed

Target level > 10

Image extraction from social network

No. of images downloaded per week.

> 100000

Descriptors extractor

Computation time for dominant colors for each image

< 100 msec.

Descriptors extractor

Computation time for texture descriptor for a 100x100 No. of stored URLâ&#x20AC;&#x2122;s to images No. of clothing categories supported Time required to collect implicit feedback on the most interesting key frames in a video from the crowd

< 100 msec.

Clothing item identification Clothing item identification Relevance feedback

>= 100,000 >= 6

Each category that we add in the VApp adds a significant amount of data that we collect and process.

< 1 week

Table 74 Technical success for image crawling from social networks

CUBRIK Revised Requirements Specifications

Page 153

D2.3 Version 1.0

Trend analysis for category Component

Indicator

Trend analysis

Time granularity

Target level < 1 week

Trend analysis

Maximal trend analysis period

>3 months

Trend analysis

Response time

<1s

Trend analysis

The computation time of the analysis

<= 1 day

Descriptor extraction

PSNR

> 25 dB for each image

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image

< 100 msec

Descriptor extraction

< 100 msec

Table 75 Technical success criteria for Trend analysis for category Trend analysis for sample Component

Indicator

Target level

Explanation

Image crawling from SN Image crawling from SN

No. of images/second

>= 10

The component should be able download at least 10 images per second

No. of domains to track

>10

The component should be able to parse at least 10 photo sharing domains in order to extract shared images from the retrieved image URLs

CUBRIK Revised Requirements Specifications

Page 154

D2.3 Version 1.0

Component

Indicator

Target level

Explanation

Trend analysis

Time granularity

< 1 week

Trend analysis

Maximal trend analysis period

>3 months

Trend analysis

Response time

<1s

Trend analysis

The computation time of the analysis

<= 1 day

Descriptor extraction

PSNR

> 25 dB for each image

The analysis component should be able to get information for small periods of time that are less than one week to provide information that is not possible to be extracted without automated processes using only e.g. fashion magazines or cool hunters Due to the large volumes of data to be processed and possible data bursts due to e.g. social events the processing and storage costs for periods larger that few months rise critically. This is the time needed for the component to return results to the UI. No computations will be performed in this step but precomputed results will be fetched. The trend analysis component should be able to compute fresh content or existing content to give answers to new queries posed by an SME. This criterion should be applicable to any trend analysis granularity from the minimum to the maximum. Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability to the final results.

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image Runtime complexity

< 100 msec

Computed detection score

Precision along key dimension No. of key dimensions implemented Response time

> .75

<= 1 m

In case of a pre-scheduled crowd.

Time for estimation of response time

<= 1 m

If no crowd is pre-scheduled, the system will return an estimate of the response time within one minute. The length of the response time is based on an estimation

Descriptor extraction

Body parts detector and router Body parts detector and router Relevance feedback Relevance feedback Relevance feedback Relevance feedback

CUBRIK Revised Requirements Specifications

< 100 msec

<5s

A detection score below 10 indicates a very unlikely detection, and above 20-30 a very likely detection.

>= 3

Page 155

D2.3 Version 1.0

Component

Indicator

Target level

Image similarity detection Image similarity detection Image similarity detection

No. of annotated images

> 100,000

Server reply time

<3s

No. of users that can use the component simultaneously

> 1000

Explanation of the size of the available crowd.

Table 76 Technical success criteria for trend analysis for sample Popularity assessment Component

Indicator

Target level

Explanation

Descriptor extraction

PSNR

> 25 dB for each image

Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability to the final results.

Descriptor extraction

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image Runtime complexity

< 100 msec

Descriptor extraction Body parts detector and router Upper/lower body parts detector Sketchness Sketchness

Conflict manager

< 100 msec

<5s

Computed detection score

Garment recognition: recall Garment segmentation: average pixel accuracy Answer/query ratio per day

> .5

Conflict manager

A detection score below 10 indicates a very unlikely detection, and above 2030 a very likely detection. @FP rate < .1

> .6

< 0.8

< 2.5

Table 77 Technical success criteria for popularity assessment

CUBRIK Revised Requirements Specifications

Page 156

D2.3 Version 1.0

Fashion matching Component

Indicator

Target level

Relevance feedback

Precision along key dimension No. of key dimensions implemented Response time

> .75

Relevance feedback Relevance feedback

>= 3 <= 1 m

Relevance feedback

Time for estimation of response time

<= 1 m

Image similarity detection Image similarity detection Image similarity detection

No. of annotated images Server reply time

> 100,000 <3 seconds > 1000

Descriptor extraction

Body part detector and router Body part detector and router

No. of users that can use the component simultaneously PSNR

Computation time for dominant colors for each image Computation time for texture descriptor for a 100x100 image Runtime complexity

> 25 dB for each image

<5s

> .5

Conflict manager

Garment recognition: recall Garment segmentation: average pixel accuracy Answer/query ratio

<= 80%

Conflict manager

Answer/question ratio

<= 2.5

CUBRIK Revised Requirements Specifications

Dominant Colors extraction should keep quantization noise low in order to keep as much of the initial information as possible for traceability to the final results.

< 100 msec

Sketchness

In case of a pre-scheduled crowd. If no crowd is pre-scheduled, the system will return an estimate of the response time within one minute. The length of the response time is based on an estimation of the size of the available crowd.

< 100 msec

Computed detection score

Sketchness

Explanation

A detection score below 10 indicates a very unlikely detection, and above 20-30 a very likely detection. @FP rate < .1

> .6 Expected Responding Time: Answer/Query Ratio per day: <80% (Based on Answering search queries with CrowdSearcher) Average Answer to Question ratio: 2.5 replies within 1 day (Based on Answering search

Page 157

D2.3 Version 1.0

Component

Indicator

Target level

Accessibility filtering

Change in impairment related parameters after between 10 and 20 feedback loops

< 15%

Accessibility filtering

Change in values in user profile values after 50 feedback loops Percentage of accessibility issues covered by the top 20 search results

< 5%

Processing for changing between different rankings Time required for initial accessibility related filtering

< 2 s.

Accessibility filtering

> 50%

Explanation queries with CrowdSearcher) An initial estimation of these parameters will be provided at the registration phase. Their convergence to the actual impairment level of the user will be achieved via a relevant feedback procedure.

The 20 first results returned by the submitted query should meet the requirements of at least one the two impairments, so as to be visible by the user. i.e. percentage of the accessibility issues. After relevance feedback

< 1 s.

Table 78 Technical success criteria for fashion matching Personalized query suggestion Component

Indicator

Target level

Explanation

Behaviour analyser

No. of suggestions provided to the user % suggestions used by users

Behaviour analyser

Reponse time

< 2 s.

Behaviour analyser

No. of input queries needed before providing suggestions

Behaviour analyser

> 20%

Table 79 Technical success criteria for personalized query suggestion

CUBRIK Revised Requirements Specifications

Page 158

D2.3 Version 1.0

15.

Conclusion and Outlook

In this deliverable we have provided a detailed specification for two vertical applications and their connections with horizontal demos. To conclude, we provide an outlook on the third year of the project, with respect to the applications (Section 15.1) and with respect to the evaluation (Section 15.2).

15.1 Next steps for the development of V-Apps In this section, look forward to the third year of the project with respect to the development of the V-Apps.

15.1.1 Fashion During the second year of the project, the use cases ‘Image crawling from social networks’ and ‘Trend analysis for category’ were implemented. In the third year of the project, our efforts will be focused on both to the extension and refinement of the use cases that have already been implemented and the development of other use cases that have not been implemented yet. ‘Image crawling from social networks’ will be extended with: - The implicit filter feedback (Likelines H-demo) will be added to the use case. This enables us to include video – more specifically: the analysis of keyframes – in the application, instead of images only. ‘Trend analysis for category’ will be extended with the following points: - The “all categories” trend analysis will be implemented; - The users will be provided with more and the time selection of the analysis and the categories will be implemented in the photo galleries of the most popular photos. Development of the other use cases: - The other use cases described in this document will be implemented during the last year of the project

15.1.2 History of Europe In year three the History of Europe app will advance along three axes: 1. Existing components will be extended with new functionalities and refined by taking into account the results of the evaluation for the first demonstrator. Furthermore the integration of the different components will be extended to demonstrate the full benefit of the crosscutting approach followed by the V-Apps. 2. New components and pipelines will be implemented to fully cover the selected user story. This concerns in particular the text indexation pipeline and the integration of GWAPs and gamification approaches. 3. Following an intense negotiation phase in year two, it is planned to integrate the archive of the Audiovisual Service of the European Commission in the HoE app. Even though all legal and contractual concerns have been eliminated a couple of practical issues remain that will need to be resolved early in year three.

CUBRIK Revised Requirements Specifications

Page 159

D2.3 Version 1.0

15.2 Next steps for the evaluation We have formulated a set of technical success criteria and user-based performance criteria that form the basis for the evaluation in Task 10.4 Evaluation of CUbRIK Apps and Task 10.5 Evaluation of CUbRIK Pipelines. Formulating the success criteria is the first step in defining the methods and instruments for the evaluation of the CUbRIK apps and pipelines. In the definition of the methodology, emphasis will be put on the main premise of the CUbRIK project: the added value of combing automatic processing with crowdsourcing in comparison to automatic processing only. Added value refers to both the added value for users and the added value as demonstrated by the technical success criteria. The system evaluation of the CUbRIK pipelines (Task 10.5 Evaluation of CUbRIK Pipelines) is dependent on the datasets that are available to carry out the evaluation. In defining the system evaluation methodology particular attention will be paid to the datasets, the procedures that have been used to establish ground truth, and how the system evaluation of the pipelines will be evaluated against the datasets. The evaluation methodology, the datasets that are used, and the evaluation results will be documented in D10.3 Application demonstrators and pipelines evaluation report we report on the datasets.

CUBRIK Revised Requirements Specifications

Page 160

D2.3 Version 1.0

16.

References

Goodhue, D.L., Thompson, R.L. (1995), Task-Technology Fit and Individual Performance, MIS Quarterly, 19(2), p. 213-236. Venkatesh, V., Morris, M.G., .Davis, G. B., Davis, F.D. (2003), User acceptance of information technology: Toward a unified view, MIS Quarterly, 27(3), p. 425â&#x20AC;&#x201C;478

CUBRIK Revised Requirements Specifications

Page 161

D2.3 Version 1.0