D5.4 - Incentives and decision making methods: Implementation and assessment by Smart Society Project

! ! !

SmartSociety! ! Hybrid!and!Diversity/Aware!Collective!Adaptive!Systems!! When!People!Meet!Machines!to!Build!a!Smarter!Society!

Grant!Agreement!No.!600854!

! ! ! ! Deliverable!D5.4!Work!package!WP5! !

Incentives!and!decision/making!methods:! Implementation!and!assessment! ! ! ! ! ! Dissemination!level! (Confidentiality)1:!

PU#

Delivery!date!in!Annex!I:!

1/7/2015#

Actual!delivery!date:!

6/8/2015#

Status2:!

Final#

Total!number!of!pages:!

29#

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1!PU:!Public;!RE:!Restricted!to!Group;!PP:!Restricted!to!Programme;!CO:!Consortium!Confidential!as!specified!in! the!Grant!Agreement! 2!F:!Final;!D:!Draft;!RD:!Revised!Draft!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Keywords:!

Deliverable!D!<X.x>!

Incentives,##Collective#Adaptive#Systems#

! !

Page!2!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

Disclaimer! This! document! contains! material,! which! is! the! copyright! of! SmartSociety! Consortium! parties,! and! no! copying!or!distributing,!in!any!form!or!by!any!means,!is!allowed!without!the!prior!written!agreement! of!the!owner!of!the!property!rights.!The!commercial!use!of!any!information!contained!in!this!document! may!require!a!license!from!the!proprietor!of!that!information.! Neither!the!SmartSociety!Consortium!as!a!whole,!nor!a!certain!party!of!the!SmartSociety!Consortium! warrant! that! the! information! contained! in! this! document! is! suitable! for! use,! nor! that! the! use! of! the! information!is!free!from!risk,!and!accepts!no!liability!for!loss!or!damage!suffered!by!any!person!using! this!information.! This!document!reflects!only!the!authors’!view.!The!European!Community!is!not!liable!for!any!use!that! may!be!made!of!the!information!contained!herein.! # ! ! ! ! ! ! ! ! ! ! ! ! !

! ! Full!project!title:!

SmartSociety!/!Hybrid!and!Diversity/Aware!Collective! Adaptive!Systems:!When!People!Meet!Machines!to!Build!a! Smarter!Society!

Project!Acronym:!

SmartSociety!

Grant!Agreement!Number:!

600854! !

Number!and!title!of!work!package:!

WP5!

Document!title:!

Incentives!and!decision!making!methods!:!Implementation! and!assessment!!

WorkMpackage!leader:!

Kobi!Gal!

Deliverable!owner:!

Kobi!Gal!

Quality!Assessor:!!

Mark!Hartswood,!OXF!

©!SmartSociety!Consortium!2013!/!2017!

Page!3!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

List!of!contributors! !

Partner!Acronym! BGU! BGU! ! ! ! ! !

Contributor! Avi!Segal! Ya’akov!(Kobi)!Gal! ! ! ! ! !

Page!4!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Executive!summary!! An!incentivizing!and!intervening!solution!for!CAS!should!address!multiple!dimensions!of!interactions! with!its!users!if!it!is!to!have!an!effective!and!positive!impact!on!the!intervened!party.!These!dimensions! span!a!wide!range!of!activities!needed!when!dealing!with!planning!and!administering!interventions!to! populations!in!CAS.!From!deciding!on!the!target!population!and!getting!the!on/going!information! about!its!behavior,!to!choosing!the!mechanism!of!intervention!and!the!channel!it!is!delivered!on,!to! deciding!on!the!intervention!timing!and!length,!these!are!activities!that!are!part!of!each!incentive! strategy.! !

Deliverable!5.4!of!WP5!focuses!on!the!implementation!and!assessment!of!incentives!and!decision/ making!methods!in!CAS.!Building!on!the!findings!of!D5.3!and!the!above!understanding,!we!turn!to! provide!a!concrete!implementation!of!an!incentive!/!intervention!mechanism,!and!focus!on!its! validation!and!assessment!in!a!large!scale!CAS!system.!Our!goal!is!to!design!a!general!purpose! incentive!server!framework!that!can!be!used!in!multiple!experiments!and!in!future!smart!society! implementations,!rather!than!having!a!tailored!code!base!for!a!one!time!situation.!!To!ensure!the! generality!and!usability!of!our!developed!algorithms!and!solutions!we!continue!to!worked!closely!with! our!partners!in!the!Smart!Society!project!and!beyond.!! !

As!part!of!this!work!we!have!built!an!Incentive!Server!framework!focused!on!serving!multiple! applications!that!use!it!for!analyzing!their!user!behavior!data,!recommending!incentives!for!these! users!and!adapting!its!recommendations!over!time.!As!such,!it!includes!mechanisms!for!defining! incentive!types!and!messages,!for!receiving!and!storing!behavioral!data,!for!applying!different! algorithms!on!this!behavioral!data,!for!deciding!on!conditions!that!requires!interventions!and!for! recommending!interventions!for!the!applications!that!use!it.!!We!then!turned!to!use!this!framework!in! modeling!and!developing!a!solution!for!real!time!disengagement!handling!in!Citizen!Science.!!To!this! end!we!have!developed!a!prediction!model!for!disengagement!in!Citizen!Science!and!have! implemented!several!intervention!strategies!based!on!!Self/determination!theory!(SDT).!!SDT!!is!a! general!theory!of!motivation!that!purports!to!systematically!explicate!the!dynamics!of!human!needs,! motivation,!and!well/being!within!the!immediate!social!context.!! !

At!the!time!of!this!writing,!we!are!awaiting!final!approval!from!the!GalaxyZoo!team!to!start!the! intervention!experiment.!!As!an!initial!step!we!have!validated!our!models!by!offline!experiments!on! the!latest!data!obtained!from!Zooniverse.!! !

The!outcome!of!our!engineering!and!research!work!in!the!period!M20/M26!is!described!in!this! deliverable!and!includes!a!detailed!description!of!the!Incentives!Server!Framework!and!the!current! development!of!a!predictive!modeling!and!intervention!mechanism!for!CAS.! ! !

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Page!5!of!(29)!!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

Table!of!Content! 1.! Introduction!and!motivation!..............................................................................................................................................!7! 2.! The!SmartSociety!Incentive!server!Framework!........................................................................................................!7! 2.1.! Motivation!.........................................................................................................................................................................!7! 2.2.! Architecture!......................................................................................................................................................................!8! 2.1.! Evaluation!.......................................................................................................................................................................!10! 2.1.1.! Ethical!Considerations!.....................................................................................................................................!10! 3.! Predictive!modeling!in!CAS!..............................................................................................................................................!11! 3.1.! Related!work!.................................................................................................................................................................!11! 3.2.! Focusing!on!Citizen!Science!....................................................................................................................................!12! 3.3.! Worker!behavior!in!citizen!science!.....................................................................................................................!12! 3.4.! Feature!engineering!for!predicting!disengagement!.....................................................................................!13! 3.5.! Developing!the!model!................................................................................................................................................!16! 3.6.! Selecting!the!Interventions!.....................................................................................................................................!17! 3.7.! Validity!and!Evaluation!.............................................................................................................................................!19! 3.8.! Limitations!and!Generalizations!...........................................................................................................................!21! 4.! Appendix!A:!Intervention!Server's!REST!API!...........................................................................................................!22! References!........................................................................................................................................................................................!28! !

Page!6!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

1. Introduction!and!motivation! ! Deliverable!5.4!of!WP5!focuses!on!the!implementation!and!assessment!of!incentives!and!decision! making!methods!in!CAS.!Building!on!the!findings!of!D5.3,!we!turned!to!provide!a!concrete! implementation!of!an!incentive!/!intervention!mechanism,!and!focus!on!its!validation!and!assessment! in!a!large!scale!CAS!system.!Our!goal!was!to!design!a!general!purpose!incentive!server!framework!that! can!be!used!in!multiple!experiments!and!in!future!smart!society!implementations,!rather!than!having!a! tailored!code!base!for!a!one!time!situation.!We!also!continued!to!worked!closely!with!our!partners!in! the!Smart!Society!project!to!ensure!the!generality!and!usability!of!our!developed!algorithms!and! solutions! ! The!outcome!of!our!engineering!and!research!work!in!the!period!M20/M26!is!described!in!this! deliverable!and!includes!a!detailed!description!of!the!Incentives!Server!Framework!and!the!current! development!of!a!predictive!modelling!and!intervention!mechanism!for!CAS.! ! Note:!in!this!report!we!will!use!the!words!“incentives”!and!“interventions”!interchangeably.!Here!the! two!terms!refer!both!to!the!action!(intervening)!and!to!the!content!(incentive)!presented!by!this! action.!! ! 2. The!SmartSociety!Incentive!server!Framework!! ! 2.1. Motivation+

As!mentioned!earlier!in!our!reports,!an!incentivizing!and!intervening!framework!for!CAS! should!address!multiple!dimensions!of!interactions!with!its!users!if!it!is!to!have!an!effective! and!positive!impact!on!the!intervened!party.!Table!1!summarizes!these!dimensions!and!points! out!some!of!the!prior!work!looking!at!each!one!of!these!dimensions.!The!related!work!section! below!describes!this!prior!work.!! ! Table!1:!Intervention!Strategy!Dimensions!

Dimension!

Description!

References!

Target!

Target!population!receiving! all,!newcomers! the!intervention! (identified!as!risk! factor),!subgroup! identified!with! predictive!modeling!!

[27],![28],! [1],![9]!

Time!

Intervention!timing!

periodical;!!as!soon!as! risk!factor!is! identified;!one!day! after!disengagement!

[1],![9],![16]!

Duration!

Intervention!(interruption)! duration!

for!x!minutes;!until! intervention!is! acknowledged;!while! user!is!on!webpage!

[18],![19],! [20],![26]!

Media!Type!

Method!of!presentation!

text;!graphics;!audio;! video;!mixed!

[7],![8],! [13],[25]!

©!SmartSociety!Consortium!2013!/!2017!

Examples!

Page!7!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

Channel!

Medium!of!delivery!

email;!web!page;! modal!message,! Facebook!

[22],![23]!

Mechanism!

The!method!used!for!this! intervention!

explanation;! encouragement;!help! system;!task!routing;! competition;! achievements;! progress!in!ladder!of! responsibility;!micro! breaks;!links!to! additional!info;!ask!for! feedback;!gamification! elements!(badges,! levels,!etc…)!

[2],![3],![4],! [5],![6],![10],! [11],![12],! [17],![21],! [24]!

Message!

Content!of!the!intervention! communication!

“...If!GalaxyZoo!didn't! [1],![14],! suit!you,!then!check! [15]! out!all!of!the!other! Zooniverse!citizen! science!projects!at! www.zooniverse.org… ”;!checkmark!image;! fun!facts!or!tasks!that! are!chosen!as!favorites! by!others!

! These!dimensions!span!a!wide!range!of!activities,!all!needed!when!dealing!with!planning!and! administering!interventions!to!populations!in!CAS.!From!deciding!on!the!target!population!and! getting!the!ongoing!information!about!its!behaviour,!to!choosing!the!mechanism!of! intervention!and!the!channel!it!is!delivered!on,!to!deciding!on!the!intervention!timing!and! length,!these!are!activities!that!are!part!of!each!incentive!strategy.!Moreover,!many!of!these! activities!generalize!for!different!incentive!scenarios!and!situations.!For!example,!it!is!safe!to! state!that!the!incentive!server!needs!to!get!information!about!the!population!behaviour!no! matter!what!is!the!specific!incentive!mechanism,!channel!or!message.!!To!address!the!above! and!ensure!we!do!not!create!a!“one!time!solution”,!we’ve!decided!to!develop!a!framework! based!solution!that!will!offer!a!componentization!of!these!activities!and!will!define!the! relationships!between!them.!In!this!way,!we!increase!the!probability!that!our!intervention! components!will!be!reused,!and!create!the!conditions!necessary!for!focusing!on!the!main! challenges!of!intervention!design.! We!now!turn!to!describing!the!architecture!of!the!Intervention!Server!and!the!APIs!it!exposes! to!the!external!world.!It!is!worth!noting!that!this!architecture!and!the!APIs!are!already!in!use!in! our!“predictive!modelling!in!CAS”!solution!described!later!on.!!!Appendix!A!includes!the! detailed!REST!API!of!the!Incentive!Server.! 2.2. Architecture+

The!Incentive!Server!is!built!to!serve!multiple!applications!which!use!it!for!analyzing!their!user! behaviour!data,!recommending!incentives!for!these!users!and!adapting!its!recommendations! over!time.!As!such,!it!includes!mechanisms!for!defining!incentive!types!and!messages,!for! receiving!and!storing!behavioural!data,!for!applying!different!algorithms!on!this!behavioural!

Page!8!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

data,!for!deciding!on!conditions!that!requires!interventions!and!for!recommending! interventions!for!the!applications!that!use!it.!! Figure!1!describes!the!main!components!of!the!WP5!Incentive!Server!Architecture.!Following! is!a!short!description!of!the!server!main!architecture!components:! ! / Data!Store:!The!data!storage!is!the!central!repository!of!the!IS.!It!stores!all!the!the! historical!Information!about!the!users!of!the!different!applications.!Each!application!user! information!is!stored!separately.!The!minimal!data!point!for!an!application!include!a! tuple!of!<timestamp,!userid>!which!denotes!a!user!performed!an!activity!in!this! application.!Such!minimal!information!is!already!useful!for!various!interventions! strategies,!e.g.!for!intervening!in!cases!of!predicted!disengagement.!This!will!be!described! in!more!details!later!in!this!report.!!The!datastore!also!stores!all!the!incentive!types! supported!by!the!system,!the!incentives!recommended!for!each!user!in!each!application! and!the!algorithmic!models!used!by!the!IS!for!its!analysis!and!recommendation.!The! current!implementation!of!the!datastore!is!based!on!an!open!source!mysql!database.!! ! / Algorithms!for!predicting!and!shaping!behavior:!The!IS!uses!algorithms!for!offline! analysis!and!training!as!well!as!for!online!analysis!and!recommendation.!These! algorithms!rely!on!behavioral!models!created!beforehand!and!adapted!in!real!time.!The! Algorithmic!sub/system!of!the!IS!includes!an!offline!component!which!handles!the! training!and!storing!of!models!on!a!periodical!basis!as!well!as!a!component!which!is! responsible!for!computing!attributes!and!features!in!real!time!for!analysis!and! intervention!recommendation.! ! / Stream!Reader:!a!stream!based!API!is!defined!to!enable!external!applications!to!pass!user! behavioural!data!to!the!IS.!The!stream!reader!component!is!listening!on!application! streams,!verifies!the!consistency!of!the!information!received!and!stores!it!in!the!IS! internal!data!store!for!further!analysis.! ! / Predictor:!the!predictor!is!the!main!real!time!analysis!component!of!the!IS.!It!is!tasked! with!analyzing!the!real!time!data!arriving!from!the!applications,!applying!the!various! models!defined!for!the!applications,!and!recommending!the!appropriate!interventions!if! needed.!Different!predictors!may!be!defined!for!different!interventions!scenarios.! ! / Dashboard:!a!detailed!dashboard!is!supplied!for!the!human!administrator!of!the!IS.!This! dashboard!handles!the!presentation!of!events!received!by!the!IS!and!the!recommended! interventions!generated!by!it.!The!dashboard!may!include!different!monitoring!screens! for!different!applications.!!In!the!next!chapter!we!will!showcase!such!a!dashboard!that!is! in!operation!for!the!“Predictive!Modeling!in!CAS”!first!application.!! ! ! The!IS!(incentive!server)!uses!REST!APIs!to!communicate!with!other!components.!These!are! used!both!for!input!and!for!output.!The!APIs!are!described!in!details!in!the!next!section.! Additionally,!the!server!listens!to!data!streams!supplied!by!external!components!(servers,! applications).!These!streams!are!used!by!the!components!to!pass!user!behaviour!data!to!the!IS.! This!data!forms!the!basis!of!information!used!by!the!IS!to!decide!on!interventions!and! incentives.!! !

©!SmartSociety!Consortium!2013!/!2017!

Page!9!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

Figure!1:!Incentive!Server!Architecture!!! 2.1. Evaluation+

The!above!architecture!was!validated!by!implementing!a!full!scale!Incentive!Server!for! predictive!modelling!in!Citizen!Science.!This!ongoing!work!is!described!in!detail!in!the!next! chapter.!! 2.1.1. Ethical!Considerations! The!Intervention!Server!approach!and!architecture!design!raises!some!ethical!questions.! We!now!refer!to!them!in!detail:!! ! ● Data!store:!the!IS!stores!historical!data!for!each!user!and!uses!this!data!for!decision! making!and!intervention!design.!This!raises!questions!about!data!privacy!as!well!as! questions!around!data!flow,!its!fair!processing!etc.!We!are!well!aware!of!these! issues!and!they!will!be!reported!in!a!later!WP1!deliverable.! ! ● When!we!talk!about!“algorithms!for!predicting!and!shaping!behavior”!we!are! raising!ethical!issues!regarding!over!manipulation!and!“unethical!persuasion”.! There!is!an!ongoing!stream!of!work!with!WP1!exploiting!these!issues!including! discussions!about!the!level!of!transparency!needed!from!such!intervention! systems.!! ! ● There!are!wider!governance!issues!in!terms!of!deciding!what!behaviours!should!be! incentivised.!In!Zooniverse!this!may!be!less!controversial!because!it!is!obvious!that! improving!how!the!scientists!are!helped!is!a!goal!that!probably!all!might!agree! Page!10!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

with.!But!with!other!CASs!that!are!helping!people!with!real/world!activities!(e.g.! transport)!deciding!what!behaviours!should!be!incentivised!may!be!much!more! controversial.!This!is!an!issue!we!are!well!aware!of!and!are!looking!at.! 3. Predictive!modeling!in!CAS! We!now!turn!to!describe!our!predictive!modeling!work!performed!during!this!work!period.!In!this! work!we!focused!on!the!main!components!of!an!incentive!and!intervention!solution:!understanding! the!behaviour!of!the!user!base,!developing!predictive!models!for!this!behaviour!and!acting!upon!them! in!real!time.!In!the!following!sections!we!describe!each!of!the!above.! Approval!for!this!study!was!granted!by!the!IRB!board!of!Ben!Gurion!University.! ! 3.1. Related+work+

Researchers!from!multiple!domains!have!conducted!research!that!refers!to!the!different! intervention!dimensions!covered!in!Table!1.!! ! Looking!at!the!target!population,!Segal!et!al.![1]!chose!to!focus!their!intervention!on!the!“quick! to!disengage”!group,!as!this!was!by!far!the!largest!population!in!their!target!community.!They! claimed!that!even!a!small!improvement!in!this!population!performance!will!have!a!substantial! overall!impact!due!to!this!group’s!size.!Mao!et!al.![9]!developed!a!predictor!for!disengagement! based!on!many!attributes!of!their!target!population’s!behaviour.!They!were!able!to!generate! more!than!100!features!to!characterise!their!users!work,!looking!at!user!features,!session! features!and!task!features!separately!and!combined.!Eveleigh!et!al.![27]!identified!a!special! group!of!participants,!the!“dabblers”.!These!users!may!be!non/frequent!users,!and!have! somewhat!sporadic!contributions.!Nonetheless,!their!impact!on!the!overall!work!done!in!the! system!is!substantial!and!thus!they!should!be!carefully!handled.!Preece!et!al.![28]!described!the! phenomena!where!in!many!Web!systems!and!beyond!(e.g.!forums),!only!a!small!fractions!of! users,!usually!less!than!10%,!are!contributors!(aka!“‘writers”)!while!the!other!vast!majority!are! only!“readers”.!!Targeting!interventions!should!take!these!different!user!characteristics!into! consideration.! ! Timing!wise,!Mao!et!al.![9]!experimented!with!offline!prediction!for!different!session!length! and!different!prediction!window!length.!They!showed!that!good!disengagement!prediction!is! possible!for!prediction!window!length!of!a!few!minutes!to!30!minutes,!while!the!prediction!is! improving!when!the!window!increases.!Segal!et!al.!decided!to!intervene!one!week!and!two! weeks!after!disengagement!for!people!that!spent!one!day!and!up!to!10!days!in!the!system,! respectively.!They!showed!that!their!intervention!had!significant!impact!on!the!return!rate!of! their!target!population.!O'Brien!et!al.![16]!conduct!a!thorough!analysis!of!the!engagement! phenomena.!They!show!that!re/engagement!is!strongly!related!to!the!reasons!for! disengagement!which!in!turn!depend!on!multiple!dimensions!including!aesthetics,!affective! appeal,!challenge,!meaningfulness!and!more.!The!timing!and!methods!of!re/engagement!is! dependent!on!these!multiple!dimensions.!! ! ! ! ! ! Interruption!wise,!Horvitz!and!Apacible![18]!introduced!methods!for!learning!models!from! data!that!can!be!used!to!compute!the!expected!cost!of!interruption!for!a!user!in!a!computerized! system.!Their!work!also!investigated!the!potential!of!building!transferable!models!of! interruptibility!and!assessment!tools,!in!pursuit!of!systems!that!could!provide!users!with! valuable!attention/sensitive!services.!Short!ET!a.![19]!used!Collaborative!Filtering!approaches! to!exploit!the!similarities!between!users!in!order!to!apply!data!from!known!users!to!new!users! and!build!a!community!based!interruption!management!algorithm.!Adamczyk!et!al.![20]! developed!models!to!identify!interruption!timing!based!on!user’s!predicted!cognitive!load.! Their!results!showed!that!different!interruption!moments!have!different!impacts!on!user! emotional!state!and!positive!social!attribution,!and!suggest!that!a!system!could!enable!a!user!to! maintain!a!high!level!of!awareness!while!mitigating!the!disruptive!effects!of!interruption.! ! ©!SmartSociety!Consortium!2013!/!2017!

Page!11!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

When!considering!the!media!type!to!be!used!in!interventions,!one!should!mention!the!seminal! work!by!Fletcher!and!Tobias![7]!that!stated!that!people!learn!better!from!words!and!pictures! that!from!words!alone.!Their!research!have!found!superior!retention!and!transfer!of!text!when! combined!with!graphics!or!animation!compared!to!text!that!is!presented!alone.! ! Considering!the!delivery!channel,!Segal!et!al.![1]!showed!that!even!a!simple!intervention!on!a! default/messaging!channel,!email,!and!was!able!to!produce!substantial!impact!on!the!return! and!contribution!rates!of!users.!Marshall!et!al.!further!investigated!and!demonstrated!how! different!formatting!of!email!messages!can!influence!the!effectiveness!and!recall!of! interventions!on!target!populations.! ! There!is!extensive!work!that!considers!different!intervention!mechanisms!in!computerized! systems.!Kamar!et!al.![4]!used!machine!learning!approaches!to!learn!about!workers! competencies!and!task!characteristics!in!Citizen!Science!projects!and!to!decide!when!the!best! time!to!effectively!retire!a!given!task!is.!Mao!et!al.![10]!compared!monetary!incentives!to!non/ monetary!incentives!for!Citizen!Science!work,!and!showed!that!a!volunteering!community!can! perform!as!good!as!or!better!than!a!wage!based!community!for!such!tasks.!Kaufmann!et!al.![12]! experimented!with!different!incentive!mechanisms!for!crowdsourcing!work,!comparing! monetary,!social,!and!“commitment!to!the!cause”!mechanisms,!among!others.!Anderson!et!al.! [21]!developed!a!model!for!the!influence!of!badges!in!the!stack/overflow!platform.!They! showed!how!the!community!behavior!is!changing!once!users!get!closer!to!the!badges!frontier,! and!gave!insights!on!the!optimal!placement!of!badges!in!such!a!system.! Finally,!as!Shawn!et!al.!demonstrated![14],!when!manipulating!the!intervention!message,!even! within!the!same!mechanism,!one!should!expect!changes!in!the!intervention!impact!on!different! population!types.! ! 3.2. Focusing+on+Citizen+Science++

Through!our!continued!relationship!with!the!SOCIAM!project!(http://sociam.org)!we!have! developed!a!close!cooperation!with!the!Zoonivese!Citizen!Science!Project.!Citizen!Science!are! mass/participation!platforms!in!which!computer!systems!can!play!a!role!in!task!allocation!and! data!aggregation,!thus!making!them!a!natural!candidate!for!smart!society!research.!Zooniverse! is!the!largest!citizen!science!platform!that!exists!today,!including!over!a!million!volunteers!and! 25!live!projects!spanning!astrophysics,!zoology,!biology,!medicine,!climate!science,!and!the! humanities.!In!all!of!these!projects!the!volunteers!identify,!classify,!mark,!and!label!data,!which! is!subsequently!aggregated!and!analyzed!in!order!to!reach!scientific!conclusions.!Through!this! cooperation!we!are!granted!access!to!a!sophisticated!CAS!environment!with!a!very!large!user! base.!Working!with!the!Zooniverse!team!we!are!able!to!develop!real!time!analysis!and! intervention!mechanisms!that!are!the!core!of!WP5!focus!and!agenda.!Thus,!it!is!only!natural! that!we!chose!to!develop!our!predictive!modelling!in!cooperation!with!the!Zooniverse!team,!to! be!extended!to!additional!platforms!in!the!following!steps!of!the!project.! ! 3.3. Worker+behavior+in+citizen+science+

Our!previous!report!(M24)!gave!a!comprehensive!review!of!workers!behaviour!in!citizen! science.!We!have!shown!that!the!vast!majority!of!participants!in!the!leading!citizen!science! platform,!Zooniverse,!work!on!a!few!tasks!and!participate!for!just!a!few!days!before!leaving!the! system.!Despite!their!casual!participation,!these!users!contribute!a!substantial!fraction!of!the! overall!effort!going!into!the!projects.! ! To! improve! our! understanding! of! the! workers! behaviour! in! citizen! science,! we! have! then! implemented! a! survey! to! uncover! reasons! for! patterns! of! engagement! and! disengagement! within! Zooniverse.! As! described! in! D5.3,! the! survey! identified! “classification! anxiety”! (overestimating! the! effects! of! individual! mistakes),! competition! from! other! life! demands! and! leisure! activities,! and! boredom! from! specific! projects! as! prominent! causes! of! disengagement! Page!12!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

among! volunteers.! For! many! users! the! cause! of! classification! anxiety! was! revealed! to! be! a! misunderstanding! of! the! collective! nature! of! citizen! science! projects,! in! which! aggregation! of! data!diminishes!the!effects!of!individual!mistakes.! ! Based!on!the!above!we’ve!concluded!that!if!we’ll!be!able!to!convince!the!“casual!participation”! users! in! the! system! to! stay! for! a! little! longer! by! addressing! their! main! concerns,! we! may! substantially!increase!the!total!volunteer!work!done!in!a!citizen!science!project.!! ! In! D5.3! this! led! us! to! design! and! implement! an! offline! intervention! mechanism,! based! on! a! reminder! email! sent! to! casual! participants! that! already! disengaged! from! the! system.! We! showed! that! even! with! this! limited! approach! we! were! able! to! bring! back! volunteers! to! the! project!and!these!volunteers!performed!as!good!as!their!non/disengaging!peers,!contribution! wise.! ! In!our!current!work,!we!turn!to!intervene!in!our!Citizen!Science!CAS!in!real!time.!Our!goal!now! is! to! be! able! to! identify! behaviour! patterns! of! CAS! users! while! they! are! interacting! with! the! system,! and! to! guide! interventions! to! them! based! on! these! behavioural! patterns! and! the! system! goals.! We! will! then! want! to! measure! the! impact! of! our! interventions! and! re/tune! our! policies.!! ! 3.4. Feature+engineering+for+predicting+disengagement+

Our!goal!was!to!delay!user!disengagement!in!the!system!by!identifying,!in!real!time,!the!user’s! intention!to!disengage,!and!by!administering!an!appropriate!personalized!message!on!the!CAS! application!screen!for!this!user.!! ! Our!first!task,!then,!is!to!develop!a!disengagement!predictor!for!our!CAS!population.!At!this! stage!of!the!project,!we’ve!focused!our!analysis!on!the!GalaxyZoo!Zooniverse!project.! Nonetheless,!all!of!the!bellow!is!project!independent!and!can!be!extended!to!other!projects!and! systems.!! ! Similarly! to! [29],! we! consider! sessions! of! a! user! on! a! crowdsourcing! platform! to! be! the! periods!of!time!that!the!user!spends!engaged!with!the!platform.!Users!complete!multiple!tasks! over!the!course!of!a!task/centric!session.!The!progress!of!a!user!can!be!interrupted!for!various! reasons.! Short/lived! demands! for! attention! such! as! bathroom! breaks! or! brief! conversations! divide!a!sequence!of!contiguous!tasks!into!contiguous!sessions!of!uninterrupted!work,!divided! by! short! breaks! where! users! intend! to! return! to! the! task.! Users! can! also! decide! to! stop! working! for! longer! periods! of! time! or! end! their! work! for! a! variety! of! reasons;! these! longer! pauses! in! activity! divide! the! activity! into! aggregate! sessions,! comprised! of! one! or! more! contiguous!sessions.! ! The!prediction!question!we!would!like!to!answer!is!the!following:!! Given! the! current! state! of! a! user’s! aggregate! session,! will! the! user! stop! participating! within!a!given!number!of!minutes!of!time?! ! As!shown!for!example!by![29],!one!can!employ!several!attribute!families!on!outcomes!that! describe!user’s!disengagement!over!time!from!a!session:! ! ● Task/Based!Features.!Users!may!be!affected!simply!by!the!tasks!that!they!observe;!For! example,! in! Galaxy! Zoo! users! tend! to! have! a! longer! session! if! the! first! few! galaxies! they! see! are! interesting,! high/quality! pictures,! rather! than! the! more! common! less! interesting! or! low/quality! galaxy! images! (this! information! was! shared! by! the! Zooniverse! team).! These! features! include! those! based! on! use! of! an! estimate! of! the! running!difficulty!of!the!last!X!tasks,!computed!by!considering!differences!in!votes!on! ©!SmartSociety!Consortium!2013!/!2017!

Page!13!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

objects! by! others.! These! differences! are! summarized! in! votes! on! objects! via! computing!the!entropy!of!a!set!of!answers.! !! ●

User! Features:! these! are! features! that! characterize! users! based! on! their! history! and! habits.!These!features!include!the!following!classes:! ○ Summary! features.! ! These! features! include! the! typical! time! spent! on! tasks,! number!of!past!sessions,!and!average!time!spent!on!sessions.!! ○ Start//end/time!features.!!Features!build!on!periods!of!time!when!users!engage! with!the!system,!including!comparison!of!the!current!time!of!day!to!the!typical! time!of!day!that!the!user!has!started!or!ended!a!session!in!the!past.! ○ Session!!!history! features.!!!!Features!!!describing!!!the!user’s!behavior!in! aggregate!sessions,!including!the!number!of!short!breaks!that!are!taken!and!the! length!of!contiguous!sessions.! ○ Inter/session!features.!!These!features!capture!information!about!the!period!of! time!(gap)!since!the!user’s!last!session!and!how!this!compares!with!past!gaps.! ○ Dwell! time! features.! Features! on! the! amount! of! time! that! the! user! spends! on! tasks,! including! consideration! of! statistics! of! different! running! averages! compared!to!previous!computed!averages!over!the!worker’s!history.! ○ Session! task! features.! ! These! features! include! a! set! of! compound! features! that! compare!the!user’s!task/related!statistics!on!the!current!session!with!the!mean! statistics! observed! on! past! sessions,! including! number! of! tasks! completed! and! amount!of!time!spent.!

●

Session! Features:! these! are! attributes! that! characterize! users’! activities! within! the! current!session.!They!include!statistics!around!the!number!of!tasks!completed!in!the! current!session!versus!completed!in!typical!sessions!for!each!user.!These!also!include! statistics!about!the!dwell!time,!capturing!the!amount!of!time!spent!on!each!task,!and! the!user’s!vote!entropy,!which!represents!the!diversity!of!users’!classifications.!These! statistics! could! well! serve! as! signals! of! a! user’s! attention! to! tasks! at! hand.! For! example,! a! running! average! of! dwell! time! as! compared! to! the! average! dwell! for! sessions!can!measure!whether!the!user!is!starting!to!pay!less!attention!or!struggling! on!a!given!task.!Similarly,!a!user!providing!a!set!of!votes!with!low!vote!entropy!on!a! spectrum!of!randomly!sorted!tasks!may!be!selecting!the!same!classification!for!many! tasks! in! the! absence! of! deep! analysis,! and! thus! paying! less! attention! than! someone! who! is! providing! input! that! is! better! matched! to! the! distribution! of! cases.! All! of! the! features!mentioned!can!be!derived!from!behavior!in!the!current!session!regardless!of! the!user’s!histories!or!habits.!!

! The!above!3!families!of!features!can!generate!over!100!features!for!each!user!activity!event!in! the! system.! This! may! put! a! burden! on! the! real! time! predictor! we! are! building.! Thus! we’ve! selected! from! the! above! the! features! with! the! highest! information! gain! influence! on! the! disengagement!label.!Table!2!specifies!the!features!selected.!These!features!are!calculated!in! real!time!for!each!activity!event!received!by!the!Intervention!Server.! ! !

Page!14!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

Table!2:!Real!Time!Predictive!Model!Features! !

Feature!

Description!!

u_bHavePastSession!

Boolean!indicator!!whether!the!user!has! history!(past!sessions)!

u_sessionCount!

Number!of!user!sessions!up!to!this!point!

u_avgSessionTasks!

Average!tasks!per!session!for!this!user!

u_medianSessionTasks!

Median!tasks!per!session!for!this!user!

u_recentAvgSessionTasks!

Average!tasks!per!session!for!last!10!sessions! of!this!users!

u_sessionTasksvsUserMedian!

Difference!between!this!session’s!tasks!the! the!median!tasks!per!session!for!this!user!

u_sessionTasksvsRecentMedian,!

Difference!between!this!session’s!tasks!the! the!median!tasks!per!session!for!the!last!10! sessions!for!this!user!

u_avgSessionTime!

Average!session!time!for!this!user!

u_sessionTimevsRecentAvg!

Difference!between!this!session!time!and!the! average!session!time!for!this!user!

u_sessionTimevsUserMedian!

Difference!between!this!session!time!and!the! median!session!time!for!this!user!

u_sessionAvgDwellvsUserAvg!

Difference!between!this!session!average!dwell! time!(time!between!tasks)!and!the!average!! dwell!time!for!this!user!

u_sessionAvgDwellvsRecentAvg!

Difference!between!this!session!average!dwell! time!(time!between!tasks)!and!the!average!! dwell!time!over!the!last!10!sessions!for!this! user!

s_minDwell!

The!minimum!dwell!time!for!this!session!for! this!user!

s_avgDwell!

The!average!dwell!time!for!this!session!for! this!user!

s_sessionTasks!

The!number!of!tasks!done!by!this!user!in!this! session!

s_sessionTime!

The!duration!of!this!session!

! !

©!SmartSociety!Consortium!2013!/!2017!

Page!15!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

3.5. Developing+the+model++

To!develop!the!prediction!model,!we’ve!worked!closely!with!our!SOCIAM!!!partners.!The! Zooniverse!SOCIAM!team!supplied!us!with!the!entire!GalaxyZoo!activity!information!from! September!10th!2012!till!June!14th!2015.!!! To!build!the!model,!we’ve!chosen!1!million!activity!records!from!the!GalaxyZoo!dataset.!To! make!sure!our!model!is!adapted!to!the!recent!behaviour!patterns!in!GalaxyZoo,!these!records! were!taken!from!January!1st!2015!and!later.!We’ve!used!50%!of!the!data!for!training!a!binary! classifier!focused!on!predicting!disengagement!within!the!next!5!minutes!for!a!30!minutes! session.!The!session!length!and!the!prediction!window!were!set!together!with!our!SOCIAM! partners!and!based!on!usage!patterns!in!GalaxyZoo.!The!other!50%!of!the!data!was!used!as! follows:!25%!for!validation!and!tuning!of!the!classifier!parameters!and!25%!for!final!testing.! ! The!model!was!developed!with!multiple!classifiers,!including!SVM,!Naive!Bayes!and!several! types! of! decision! trees.! The! overall! data! set! is! biased,! containing! lower! positive! instances! where! users! disengage.! As! a! result,! we! considered! the! measure! of! area! under! the! receiver/ operator! characteristic! curve! (AUC)! to! evaluate! the! relative! performance! of! different! classification! algorithms.! The! AUC! measure! can! be! interpreted! as! the! likelihood! that! a! classifier! will! distinguish! a! randomly! selected! positive! instance! from! a! randomly! selected! negative!instance.!A!random!classifier!that!assigns!each!instance!the!prior!probability!of!the! dataset!has!an!AUC!of!0.5,!and!a!classifier!that!can!always!distinguish!positive!from!negative! instances!has!an!AUC!of!1.0.!The!AUC!is!invariant!to!the!prior!distribution!of!labels!in!different! datasets,!which!can!be!highly!skewed.!! ! Details!description!of!each!of!the!classifiers!tested!is!outside!the!scope!of!this!report.!In!our! experiments,!Gradient!Boosting!Classifiers!and!Bagging!Classifiers!consistently!outperformed! the! alternatives! thus! we’ve! focused! on! these! families! of! classifiers.! These! classifiers! exhibit! AUCs!of!0.73!for!users!with!past!history!in!the!system!and!AUC!of!0.65!for!new!users!in!the! system.! Figure! 2! describes! the! relative! importance! of! the! features! used! for! the! Boosting! Classifier.!! !

! Figure!2:!Feature!Importance:!Boosting!Classifier! ! Page!16!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

! In! parallel! to! developing! the! model,! we’ve! developed! the! control! group! model! for! this! experiment.! The! test! condition! will! send! interventions! ! to! the! users! as! soon! as! the! predictive! model! is! predicting! disengagement! in! “the! next! 5! minutes”.! Thus,! we’ve! defined! the! control! condition! to! consist! of! sending! the! same! intervention! messages! to! users,! but! this! time! on! a! random!basis,!without!a!prediction!component.!To!achieve!this,!once!a!new!session!for!a!user! from!the!control!group!starts,!we!sample!the!“expected!session!length”!for!this!session!from!the! known!distribution!of!sessions!in!the!system.!We!then!decide!for!this!session!when!to!send!the! intervention!in!a!random!and!uniform!manner!and!send!this!intervention!on!time!if!the!session! did! not! end! by! that! time.! ! We! also! balance! the! overall! number! of! interventions! sent! between!! the!random!and!test!conditions.!! 3.6. Selecting+the+Interventions+

Having!an!“intelligent”!predictor!and!a!random!control!group!conditions,!we!would!like!to!decide! what! interventions! to! administer.! ! For! this,! we! now! turn! to! a! core! theory! of! motivation! and! behaviour,!the!Self!Determination!Theory.!! ! Self/determination!theory![30]!is!a!general!theory!of!motivation!that!purports!to!systematically! explicate!the!dynamics!of!human!needs,!motivation,!and!well/being!within!the!immediate!social! context.! The! term! self/determination,! as! defined! by! Deci! and! Ryan,! is! ‘‘a! quality! of! human! functioning!that!involves!the!experience!of!choice.![It!is]!the!capacity!to!choose!and!have!those! choices! ...! be! the! determinants! of! one’s! actions”! (p.! 38).! Self/determination! theory! proffers! that! humans’! have! three! universal! and! basic! needs:! ! autonomy! (a! ! ! sense! of! control! and! agency),! competency! (feeling! competent! with! tasks! and! activities),! and! relatedness! (feeling! included! or! affiliated! with! others).! Individuals! experience! an! elaborated! sense! of! self! and! achieve! a! better! psychological! well/being! through! the! satisfaction! of! the! three! basic! needs.! Conversely,! the! deprivation!of!the!three!basic!needs!produces!highly!fragmented,!reactive,!or!alienated!selves.! Another!central!tenet!of!SDT!is!that!as!opposed!to!other!motivational!theories!(e.g.,!Bandura’s! social! cognitive! theory)! that! treat! human! motivation! as! a! monolithic! construct,! SDT! theorizes! human!motivation!into!three!main!categories:!intrinsic!motivation!(doing!something!because!it! is! enjoyable,! optimally! challenging,! or! aesthetically! pleasing),! extrinsic! motivation! (doing! something! because! it! leads! to! a! separable! outcome)! and! amotivation! (the! state! of! lacking! intention!to!act).!!According!to!SDT,!self/determined!types!of!motivation!(intrinsic!motivation)! may! lead! to! positive! outcomes! while! nonself/determined! types! of! motivation! (amotivation,! external)!may!result!in!negative!outcomes.! We! suggest! that! SDT! is! an! appropriate! framework! for! addressing! motivation! in! the! Citizen! Science!domain!and!in!CAS!in!general.!!First,!!!SDT!may!serve!as!!!a!!!theoretical!framework!that! integrates!issues!in!CAS.!It!addresses!autonomy,!relatedness,!and!competence!as!determinants! of!motivation.!These!three!constructs!correspond!strongly!with!our!finding!in!the!D5.3!survey.! As!mentioned!earlier,!the!survey!identified!“classification!anxiety”!(overestimating!the!effects!of! individual!mistakes),!competition!from!other!life!demands!and!leisure!activities,!and!boredom! from!specific!projects!as!prominent!causes!of!disengagement!among!volunteers.!!These!factors! can! be! seen! as! examples! of! competence! and! relatedness! not! being! fulfilled! (“classification! anxiety”)! and! of! autonomy! being! expressed! (boredom,! competition! of! other! engagements)! by! the!system!users.! Additionally,! past! experimental! research! indicates! that! self/determination! theory! predicts! a! variety! of! engagement! outcomes,! including! performance,! persistence,! and! satisfaction! [30].! Also,!the!intrinsic!motivation!focus!of!SDT!lends!itself!naturally!to!the!nonmonetary!volunteer! environment! of! Citizen! Science,! where! the! strong! external! motivators! of! money! and! alike! are! missing.! ©!SmartSociety!Consortium!2013!/!2017!

Page!17!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

Another! advantage! of! SDT! is! that! it! generates! prescriptions! for! motivational! enhancement! in! addition!to!!describing!individuals’!motivation!process.!Self/determination!theory/based!studies! have! identified! strategies! that! foster! individual! self/determination! and! motivation.! Reeve! ! and! Jang! [31],! for! ! example,! focused! on! the! learning! domain! and! validated! eight! types! of! teachers! autonomy/supportive! behaviors,! such! as! ! allowing! ! choice,! providing! rationale,! and! offering! informational! feedback! ! ! that! ! enhanced! students’! perceived! autonomy,! engagement,! and! performance.! The! ! SDT/based! strategies! ! may! be! ! applicable! to! a! ! variety! of! ! other! settings!! including!online!CAS!environments.! ! Considering!SDT!and!our!survey!findings!leads!us!to!choose!several!interventions!that!will! adhere!to!the!3!motivational!factors!identified!by!SDT.!We!thus!select!the!following!intervention! messages!to!be!tested!during!our!experiment!on!the!real!time!GalaxyZoo!system:! ! 1. Addressing!Autonomy!and!Competence!(Reasoning):!“Please!don’t!stop!just!yet.!You’ve! been!extremely!helpful!so!far.!Your!votes!are!really!helping!us!to!understand!deep! mysteries!about!galaxies.”.!Reasoning!is!given!in!this!intervention!to!regulate!Autonomy! through!identification.!This!was!supported!by!Deci!et!al.![30]!who!found!in!laboratory! settings!that!meaningful!reasoning!is!key!for!internalization!of!autonomy!(i.e!“I!identify! with!this!cause!thus!I!choose!to!perform!it”)! 2. Addressing!Relatedness:!“Thousands!of!people!are!taking!part!in!the!project!every! month.!Visit!talk!at!talk.galaxyzoo,org!to!discuss!images!you!see!with!them.”! 3. Addressing!Competence!(Anxiety):!“We!use!statistical!techniques!to!get!the!most!of!every! answer;!So,!you!don’t!need!to!worry!about!being!“right”.!Just!tell!us!what!you!see.”! !!!!!In!addition,!user!can!express!their!choice!(autonomy)!after!every!message!and!choose!to!opt! out!from!receiving!any!further!messages.!! ! These!messages!will!appear!in!front!of!GalaxyZoo!users!when!they!work!on!the!site.!Figure!3! shows!such!a!messages!in!action.!To!increase!the!probability!it!is!noticed,!the!message!“flies!in”! into!the!screen!and!“flies!out”!after!a!time!that!is!given!to!the!user!to!read!it!(currently!20! seconds).! !

! Figure!3:!Intervention!Message!in!Action! ! Page!18!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

! Table!3!summarizes!the!7!conditions!that!will!be!tested!in!the!real!time!experiment.!! ! Table!3:!Real!Time!Intervention!Conditions!

Cohort!#! Cohort!Size!M!Part!of! Intervention! population! 1!

1/7!

No!intervention!!

1/7!

Random!intervention,!Message!1!

1/7!

Random!intervention,!Message!2!

1/7!

Random!!intervention,!Message!3!

1/7!

Intelligent!intervention,!Message!1!

1/7!

Intelligent!intervention,!Message!2!

1/7!

Intelligent!intervention,!Message!3!

! ! 3.7. Validity+and+Evaluation+

At!the!time!of!this!writing,!we!are!awaiting!final!approval!from!the!GalaxyZoo!team!to!start!the! intervention!experiment.!This!is!expected!shortly.!The!GalaxyZoo!community!“managers”,! which!are!the!leading!contributors!to!the!project!as!well!as!the!managers!of!the!project’s! forums,!have!already!approved!the!experiment!under!the!restrictions!(which!we’ve!adopted)! of!having!an!opt/out!options!and!not!sending!more!than!one!intervention!message!per!session! (a!30!minutes!window).!During!the!experiment!we!will!use!the!dashboard!shown!in!Figure!4! which!enables!us!to!see!the!real/time!behaviour!of!the!IS!and!the!interventions!predicted!and! sent!by!it.!

! Figure!4:!Intervention!Server!Dashboard! ! ! ©!SmartSociety!Consortium!2013!/!2017!

Page!19!of!(29)!!

©!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

Following!this!experiment!we!plan!to!engage!in!additional!experiments!based!on!the!IS! framework.!This!will!include!additional!interventions!in!Citizen!Science!based!on!the! data!and!results!from!the!first!experiment,!interventions!in!other!CAS!environments! and!interventions!for!system!goals!beyond!alleviating!disengagement.!! ! To!validate!our!model!and!predictor,!and!perform!additional!fine!tuning!to!our! parameters!based!on!the!system!goals,!we!have!run!multiple!offline!experiments!on!the! latest!data!obtained!from!GalaxyZoo.!In!these!experiments!we!used!our!developed! predictor!to!identify!disengagement!in!“real!time”!as!the!data!was!supplied!to!the!IS! and!decide!on!sending!an!intervention.!During!this!process!we!varied!the!predictor! cutoff!probability!from!the!default!0.5!value!to!values!ranging!from!0.3!to!0.8.!We!then! compared!the!predicted!disengagement!to!the!real!disengagement!obtained!from!the! data,!and!then!calculated!Precision,!Recall,!F1!and!Accuracy!metrics.!Figure!5!shows!the! main!results!of!these!experiments.!! !

! Figure!5:!Validation!and!Optimization!Experiments! As!expected,!once!can!see!that!the!classifier!thresholds!enabled!us!to!balance!between!our! required!precision!and!recall.!We!can!see!that!the!F1!is!indeed!maximized!at!the!default! classifier!cut!off!point!of!0.5.!For!trying!to!capture!more!“about!to!disengage”!users,!at!the! expense!of!intervening!with!users!who!do!not!plan!to!disengage,!one!may!consider!lowering! the!cutoff!point!below!0.5.!If!precision!is!of!outmost!importance,!the!cutoff!point!may!be! increased.!Nonetheless,!this!will!result!in!sharp!decline!in!the!recall!value.! ! !

Page!20!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

©!SmartSociety!Consortium!2013!/!2017!

3.8. Limitations+and+Generalizations+

! The!intervention!Server!was!defined!and!built!to!enable!a!substantial!amount!of!generalization.! Data!acquisition!is!strongly!decoupled!from!data!processing!through!usage!of!standard!REST! API,!intermediate!storage!in!a!standard!database!and!an!extendible!data!representation!format! (Json!key:value!standard).!The!algorithmic!framework!supports!implementation!of!multiple! predictors!and!intervention!agents,!and!ongoing!learning!is!enabled!!by!the!continuous!real! time!acquisition!of!behavioural!data.!Interventions!are!send!by!both!a!push!and!pull!manners,! again!in!a!standard!extensible!API!based!format!and!additional!interventions!can!be!defined!in! a!declarative!manner.!The!disengagement!alleviation!predictor!is!domain!independent!as!it! uses!only!“action!signals”!without!understanding!the!meaning!of!these!actions.!Thus,!it!is! expected!to!support!any!task!based!CAS!environment.!Intervention!messages!!to!alleviate! disengagement!can!be!easily!changed!and!adapted.!Nonetheless,!our!field!experimentation!is! currently!focused!!on!the!citizen!science!domain!and!on!the!engagement!and!disengagement! phonenomena.!Additionally,!our!current!interventions!are!heuristic!in!nature,!do!not!include!a! reinforced!learning!component!and!currently!!do!not!adapt!in!real!time!in!a!significant!manner.! To!deal!with!these!limitations!we!will!need!to!experiment!beyond!the!citizen!science!domains,! focus!on!challenges!beyond!the!engagement!challenge!and!develop!reinforced!learning!based! intervention!agents.!! ! ! ! !

©!SmartSociety!Consortium!2013!/!2017!

Page!21!of!(29)!!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

! 4. Appendix!A:!Intervention!Server's!REST!API! ! We!now!turn!to!describe!the!different!APIs!supported!by!the!IS.!! ! Home!Page:! URL! /! Method! GET! Parameters! ! Field! Type! ! ! Response! ! Description! Auto!Generated!Home!Page! !

Description! !

! Admin&Page:! URL! Permission! Method! Parameters!

/admin/! Logged!In.! GET! ! Field! Type! ! ! ! Auto!Generated!!Admin!Page!

Description! !

Response! Description! ! Login:! URL! Method! Parameters!

/login/! POST! ! Field! Type! Description! username! String! The!Username.! password! String! The!Password.! POST!/login/!HTTP/1.1! ContentGType:!application/json! Host:!127.0.0.1:8000! Connection:!close! UserGAgent:!Paw/2.1.1!(Macintosh;!OS!X/10.10.2)!GCDHTTPRequest! ContentGLength:!35! !!!!{! !!!!!!!!! !!!!!!!!!!!!"username":"dor",! !!!!!!!!!!!!"password":"123"! !!!!}! {! !'Token':'ec1db856e230a61cc12ad5040a554c2a312fcee9'! }! Login!Action.!Used!by!external!applications!to!obtain!access!to!the! intervention!server!(IS).!Must!be!performed!before!any!other!operation! !

Usage!

Response!

Description! Page!22!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

access!to!the!IS.! ! Get&All&Incentives:! URL! Method! Permission! Parameters!

/api/incentive/! GET! Logged!In.! ! Field! Type! ! ! {! !!!!"count":!2,! !!!!"next":!null,! !!!!"previous":!null,! !!!!"results":![! !!!!!!!!{! !!!!!!!!!!!!"owner":!"dor",! !!!!!!!!!!!!"schemeName":!"SendEmail",! !!!!!!!!!!!!"schemeID":!123123,! !!!!!!!!!!!!"text":!"Hello!World!",! !!!!!!!!!!!!"typeID":!1212,! !!!!!!!!!!!!"typeName":!"sending",! !!!!!!!!!!!!"status":!true,! !!!!!!!!!!!!"ordinal":!21312,! !!!!!!!!!!!!"tags":![],! !!!!!!!!!!!!"modeID":!21312,! !!!!!!!!!!!!"groupIncentive":!false,! !!!!!!!!!!!!"condition":!"if!the!user!is!older!than!21"!! !!!!!!!!},! !!!!!!!!{! !!!!!!!!!!!!"owner":!"dor",! !!!!!!!!!!!!"schemeName":!"DoNoting",! !!!!!!!!!!!!"schemeID":!14232,! !!!!!!!!!!!!"text":!"Hello!World!",! !!!!!!!!!!!!"typeID":!1212,! !!!!!!!!!!!!"typeName":!"sending",! !!!!!!!!!!!!"status":!true,! !!!!!!!!!!!!"ordinal":!21312,! !!!!!!!!!!!!"tags":![],! !!!!!!!!!!!!"modeID":!21312,! !!!!!!!!!!!!"groupIncentive":!false,! !!!!!!!!!!!!"condition":!"if!the!user!is!older!than!21"!! !!!!!!!!}! !!!!]! }! Get!All!Incentives!types!from!the!IS!

Description! !

Response!

Description! ! Add&Incentive:! URL! Method! Permission!

/api/incentive/! POST! Logged!In.!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Page!23!of!(29)!!

Parameters!

Deliverable!D!<X.x>!

! Description!

Type!

Field!

String!

schemeName!

Int!

schemeID!

String!

text!

Int!

typeID!

String!

typeName!

Boolean!

status!

Int!

ordinal!

Tag[]!

tags!

Int!

modeID!

Boolean!

groupIncentive!

String!

condition!

The!Scheme!Name.!

The!Scheme!ID.!

The!Text!of!the!Incentive.! The!Type!ID.! The!Type!Name.! The!Status.! The!ordinal!number.!

List!of!Tags!for!this!Incentive! (Array!of!Tags).!

The!Mode/ID.! If!this!a!group!Incentive.!

The!Condition!for!this! incentive.! !

Response!

Page!24!of!(29)!! !

HTTP!!201!CREATED! ContentGType:!application/json!! Vary:!Accept! Allow:!GET,!POST,!HEAD,!OPTIONS! !!!!!!!!{! !!!!!!!!!!!!"owner":!"dor",! !!!!!!!!!!!!"schemeName":!"SendEmail",! !!!!!!!!!!!!"schemeID":!123123,! !!!!!!!!!!!!"text":!"Hello!World!",! !!!!!!!!!!!!"typeID":!1212,! !!!!!!!!!!!!"typeName":!"sending",! !!!!!!!!!!!!"status":!true,! !!!!!!!!!!!!"ordinal":!21312,! !!!!!!!!!!!!"tags":![],! http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

!!!!!!!!!!!!"modeID":!21312,! !!!!!!!!!!!!"groupIncentive":!false,! !!!!!!!!!!!!"condition":!"if!the!user!is!older!than!21"!! !!!!!!!!}! ! }! Add!an!incentive!type!to!the!IS!

Description! ! Get&Incentive&for&User:! URL! Permission! Method! Parameters!

/getIncUser/! Logged!In.! GET! ! Field! Type! Description! userID! String! The!User!Id.! {! !"schemeID":"123"! }! Get!The!Best!Incentive!for!this!User.!Pull!operation!by!external!application! to!obtain!incentive!for!user! !

Response!

Description! ! !

View&live&interventions:! URL! Method! Parameters!

/dash/pages/dash.html! GET! ! Field! Type! Description! ! ! ! ! Dash!Page:!once!loaded!presents!in!real!time!the!events!received!by!the!IS! and!the!interventions!generated!by!it.! !

Response! Description! !

Get&interventions&by&date:! URL! Method! Parameters!

/ask_by_date/! GET! ! Field! Type! Description! ! ! ! json:! {! !!!!!!!!!!!!!!!!"id":!id,! !!!!!!!!!!!!!!!!"user_id":!str(user_id),! !!!!!!!!!!!!!!!!"created_at":!str(created_at),! !!!!!!!!!!!!!!!!"intervention_id":!str(intervention_id),! !!!!!!!!!!!!!!!!"preconfigured_id":!str(preconfigured_id),! !!!!!!!!!!!!!!!!"cohort_id":!str(cohort_id),! !!!!!!!!!!!!!!!!"algo_info":!str(algo_info),! !

Response!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Page!25!of!(29)!!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

!!!!!!!!!!!!!!!!"country_name":str(country_name)! }! get!intervention!list!from!the!given!date.!

Description! !

get&interventions&by&ID:! URL! Method! Parameters!

/ask_by_id/! GET! ! Field! Type! Description! record_id! int! ID!of!first!wanted!ID! json!or!json!list:! {! !!!!!!!!!!!!!!!!"id":!id,! !!!!!!!!!!!!!!!!"user_id":!str(user_id),! !!!!!!!!!!!!!!!!"created_at":!str(created_at),! !!!!!!!!!!!!!!!!"intervention_id":!str(intervention_id),! !!!!!!!!!!!!!!!!"preconfigured_id":!str(preconfigured_id),! !!!!!!!!!!!!!!!!"cohort_id":!str(cohort_id),! !!!!!!!!!!!!!!!!"algo_info":!str(algo_info),! !!!!!!!!!!!!!!!!"country_name":str(country_name)! }! Intervention!or!intervention!list!from!the!given!id.! !

Response!

Description! ! ! Posting!an!Incentive!to!an!External!Server! In!addition!to!the!pull!mode,!the!IS!supports!sending!incentives!to!an!application!by!calling!this! application!REST!API.!For!this,!the!IS!expects!to!have!the!following!REST!API!supported!by!the!external! application:! ! URL! Method! Parameters!

"http://<External!Application!Server>/users/"!+!user_id!+!"/interventions"! POST! ! Field! Type! Description! project! string! project!name! intervention_type! string! Type! text_message! string! ! cohort_id! int! ! time_duration! int! ! presentation_duration! int! ! intervention_channel! string! ! take_action! string! ! preconfigured_id! int! ! experiment_name! string! ! {! """""cohort_id":"1,! """""created_at":""2015407407T10:38:37.703+00:00",! """""details":"null,! !

Response!

"""""experiment_name":""Zooniverse4MSR4BGU"GalaxyZoo"Experiment"1",! """""intervention_channel":""web"message",! """""intervention_type":""prompt"user"about"talk",! """""preconfigured_id":"1,!

Page!26!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

ÂŠ!SmartSociety!Consortium!2013!/!2017! """""presentation_duration":"30,! """""project":""galaxy_zoo",! """""state":""active",! """""take_action":""after_next_classification",! """""text_message":""please"return",! """""time_duration":"120,! """""updated_at":""2015407407T10:38:37.703+00:00",! """""user_id":""17303",! """""id":""559bac2d3031650001140000"! }!

Description!

! Post!Incentive!to!!External!Application!by!calling!this!REST!API!!

! ! Listening!to!Behavioural!Data!! The!IS!is!supporting!stream!delivery!of!behavioural!data!from!external!applications.!!For!this,!it! expects!the!external!application!to!respond!to!the!curl!query!below!with!behavioural!data!in!json! format.!Details!are!given:!! ! URL! curl!GH!"Accept:!application/<param>+json"!<URL>! Response! ! Field! Type! Description! user_id! string! mandatory! created_at! string! mandatory! action! string! optional! other!fields! string! optional! Description! Listen!to!behaviour!event!streams!! !

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Page!27!of!(29)!!

ÂŠ!SmartSociety!Consortium!2013!/!2017!

Deliverable!D!<X.x>!

!References! 1. 2. 3. 4.

5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.

17. 18. 19. 20.

21. 22.

23.

A.!Segal,!R.!Simpson,!K.!Gal,!V.!Homsy,!M.!Hartswood,!K.!Page,!M.!Jirotka:!Motivating!the!Zooniverse:! Improving!Productivity!in!Citizen!Science!through!Controlled!Intervention!(in!submission).! Kamar,!Ece,!Ashish!Kapoor,!and!Eric!Horvitz.!"Lifelong!learning!for!acquiring!the!wisdom!of!the!crowd."! Proceedings#of#the#TwentyGThird#international#joint#conference#on#Artificial#Intelligence.!AAAI!Press,!2013.! Lin,!Christopher!H.,!Ece!Kamar,!and!Eric!Horvitz.!"Signals!in!the!Silence:!Models!of!Implicit!Feedback!in!a! Recommendation!System!for!Crowdsourcing."!(2014).! Kamar,!Ece,!Severin!Hacker,!and!Eric!Horvitz.!"Combining!human!and!machine!intelligence!in!large/scale! crowdsourcing."!Proceedings#of#the#11th#International#Conference#on#Autonomous#Agents#and#Multiagent# SystemsGVolume#1.!International!Foundation!for!Autonomous!Agents!and!Multiagent!Systems,!2012.! Nov,!Oded,!Ofer!Arazy,!and!David!Anderson.!"Dusting!for!science:!motivation!and!participation!of!digital! citizen!science!volunteers."!Proceedings#of#the#2011#iConference.!ACM,!2011.! Prestopnik,!Nathan,!Kevin!Crowston,!and!Jun!Wang.!"Exploring!Data!Quality!in!Games!With!a!Purpose."! (2014).! Fletcher,!J.!D.,!and!Sigmund!Tobias.!"The!multimedia!principle."!The#Cambridge#handbook#of#multimedia# learning!117!(2005):!133.! Clark,!Ruth!C.,!and!Richard!E.!Mayer.!EGlearning#and#the#science#of#instruction:#Proven#guidelines#for# consumers#and#designers#of#multimedia#learning.!John!Wiley!&!Sons,!2011.! Mao,!Andrew,!Ece!Kamar,!and!Eric!Horvitz.!"Why!Stop!Now?!Predicting!Worker!Engagement!in!Online! Crowdsourcing."!First#AAAI#Conference#on#Human#Computation#and#Crowdsourcing.!2013.! Mao,!Andrew,!et!al.!"Volunteering!Versus!Work!for!Pay:!Incentives!and!Tradeoffs!in!Crowdsourcing."! First#AAAI#Conference#on#Human#Computation#and#Crowdsourcing.!2013.! Bragg,!Jonathan,!Andrey!Kolobov,!and!Daniel!S.!Weld.!"Parallel!Task!Routing!for!Crowdsourcing."!Second# AAAI#Conference#on#Human#Computation#and#Crowdsourcing.!2014.! Kaufmann,!Nicolas,!Thimo!Schulze,!and!Daniel!Veit.!"More!than!fun!and!money.!worker!motivation!in! crowdsourcingâ&#x20AC;&#x201C;a!study!on!mechanical!turk."!(2011).! Lasecki,!Walter!Stephen,!Ece!Kamar,!and!Dan!Bohus.!"Conversations!in!the!Crowd:!Collecting!Data!for! Task/Oriented!Dialog!Learning."!First#AAAI#Conference#on#Human#Computation#and#Crowdsourcing.!2013.! Shaw,!Aaron!D.,!John!J.!Horton,!and!Daniel!L.!Chen.!"Designing!incentives!for!inexpert!human!raters."! Proceedings#of#the#ACM#2011#conference#on#Computer#supported#cooperative#work.!ACM,!2011.! Evans,!Laurel,!et!al.!"Self/interest!and!pro/environmental!behaviour."!Nature#Climate#Change!3.2!(2013):! 122/125.! O'Brien,!Heather!L.,!and!Elaine!G.!Toms.!"What!is!user!engagement?!A!conceptual!framework!for!defining! user!engagement!with!technology."!Journal#of#the#American#Society#for#Information#Science#and# Technology!59.6!(2008):!938/955.! Haaranen,!Lassi,!et!al.!"How!(not)!to!introduce!badges!to!online!exercises."Proceedings#of#the#45th#ACM# technical#symposium#on#Computer#science#education.!ACM,!2014.! Horvitz,!Eric,!and!Johnson!Apacible.!"Learning!and!reasoning!about!interruption."!Proceedings#of#the#5th# international#conference#on#Multimodal#interfaces.!ACM,!2003.! Shrot,!Tammar,!et!al.!"CRISP:!an!interruption!management!algorithm!based!on!collaborative!filtering."! Proceedings#of#the#32nd#annual#ACM#conference#on#Human#factors#in#computing#systems.!ACM,!2014.! Adamczyk,!Piotr!D.,!and!Brian!P.!Bailey.!"If!not!now,!when?:!the!effects!of!interruption!at!different! moments!within!task!execution."!Proceedings#of#the#SIGCHI#conference#on#Human#factors#in#computing# systems.!ACM,!2004.! Anderson,!Ashton,!et!al.!"Steering!user!behavior!with!badges."!Proceedings#of#the#22nd#international# conference#on#World#Wide#Web.!International!World!Wide!Web!Conferences!Steering!Committee,!2013.! Papadimitriou,!Panagiotis,!et!al.!"Display!advertising!impact:!Search!lift!and!social!influence."! Proceedings#of#the#17th#ACM#SIGKDD#international#conference#on#Knowledge#discovery#and#data#mining.! ACM,!2011.! Marshall,!Bryan,!et!al.!"An!Exploratory!Study!of!the!Impact!of!Formatting!on!Email!Effectiveness!and! Recall."!Communications#of#the#IIMA!9.4!(2014):!1.!

Page!28!of!(29)!! !

http://www.smart/society/project.eu/!!

Deliverable!D!<5.4>!

24. Dow,!Steven,!et!al.!"Shepherding!the!crowd!yields!better!work."!Proceedings#of#the#ACM#2012#conference# on#Computer#Supported#Cooperative#Work.!ACM,!2012.! 25. Jackson,!Corey,!et!al.!"Motivations!for!Sustained!Participation!in!Citizen!Science:!Case!Studies!on!the!Role! of!Talk."!17th#ACM#Conference#on#Computer#Supported#Cooperative#Work#&#Social#Computing.!2014.! 26. Kamar,!Ece,!and!Barbara!Grosz.!"Applying!MDP!approaches!for!estimating!outcome!of!interaction!in! collaborative!human/computer!settings."!(2007).! 27. Eveleigh,!Alexandra,!et!al.!"Designing!for!dabblers!and!deterring!drop/outs!in!citizen!science."! Proceedings#of#the#32nd#annual#ACM#conference#on#Human#factors#in#computing#systems.!ACM,!2014.! 28. Preece,!Jennifer,!and!Ben!Shneiderman.!"The!reader/to/leader!framework:!Motivating!technology/ mediated!social!participation."!AIS#Transactions#on#HumanGComputer#Interaction!1.1!(2009):!13/32.! 29. Mao,!Andrew,!Ece!Kamar,!Horvitz,!Eric.!“Why!Stop!Now!?!Predicting!Worker!Engagement!in!Online! Crowdsourcing”.!Proceedings#of#the#first#AAAI#Conference#of#Human#Computation#and#Crowdsourcing##! 30. Deci,#E.#L.,#&#Ryan,#R.#M.#(1985).#Intrinsic#motivation#and#selfGdetermination#in#human#behavior.#New#York:# Plenum.## 31. Reeve,#J.,#&#Jang,#H.#(2006).#What#teachers#say#and#do#to#support#students’#autonomy#during#a#learning# activity.#Journal#of#Educational#Psychology,#98(1),#209–218.##

! ! ! !

Page!29!of!(29)!!