The Blue Book

BLUE BOOK

A set of cybersecurity roadmaps and challenges for researchers and policymakers

EDITORS

EvangelosMarkatos WP4Leader FoundationforResearchand Technology-Hellas

KaiRannenberg CyberSec4EuropeManager Goethe-UniversityFrankfurt

AUTHORS

EliasAthanasopoulos UniversityofCyprus

PanagiotisBountakas UniversityofPiraeus SunilChaudhary NTNU SaidDaoudagh CNR AntreasDionysiou UniversityofCyprus ChristosDouligeris UPRC AfonsoFerreira IRIT

SimoneFischer-Hübner KarlstadsUniversitet VasileiosGkioulos NTNU

LeonardoHornIwaya KarlstadsUniversitet MeikoJensen KarlstadsUniversitet WouterJoosen KULeuven

MarkoKompara UniversityofMaribor PanayiotisKotzanikolaou UniversityofPiraeus StephanKrenn AIT AntonioLioy PolitecnicodiTorino AlbertoLluchLafuente TechnicalUniversityofDenmark HarryManifavas FORTH EdaMarchetti CNR

ConstantinosPatsakis UniversityofPiraeus JoaoRobertoPeres KOMP&FaculdadeGetulio LaurensSion KULeuven SilviaSisinni PolitecnicodiTorino ChristosXenakis UniversityofPiraeus

Oum-El-KheirAktouf, GrenobleINP, France

SpirosAntonatos, AegisTechnologies,Singapore Antanas ˇ Cenys, VilniusGediminasTechnicalUniversity NabinChowdhury, NTNU

ChristoforosDadoyan, IonianUniversity ClaudiaDiaz, KULeuven

NikolajGoranin, VilniusGediminasTechnicalUniversity

DimitrisGritzalis, AthensUniversityof EconomicsandBusiness

StevenFurnell, UniversityofNottingham

MaritHansen, UnabhängigesLandeszentrumfürDatenschutzSchleswig-Holstein (ULDSH)

MarkoH˝olbl, UniversityofMaribor ThorstenHolz, CISPAHelmholtzCenter forInformationSecurity

Jaap-HenkHoepman, RadboudUniversity

AlexandrosKapravelos, NorthCarolina StateUniversity

MariekeHuisman, UniversityofTwente, Netherlands

MarcJuarez, UniversityofEdinburgh

NicolasKourtellis, TelefonicaI+D

GiuseppeLami, CNR,Italy

AlexiosLekidis,PublicPowerCorporation

NicolasMayer, LuxembourgInstituteof ScienceandTechnology

WeizhiMeng, TechnicalUniversityofDenmark

MarinoMiculan, UniversitàdiUdine

PanagiotisPapadopoulos, iProovLimited

JasonPolakis, UIC

LorenzoPupillo, CEPSandLUISS JoaoResende, NOVAuniversityofLisbon KonradRieck, TechnischeUniversität Braunschweig

VittorioRosato, ENEAandUniversity CampusBiomedicoofRome

AntonioF.Skarmeta, UniversityofMurcia

ThomasSchaberreiter, CS-AWARECorporation

StefanSchiffner, UniversityofMünster RobertoSettola, UniversitàCampusBioMedicodiRoma

MauriceH.terBeek, CNR,Italy

DenisTrˇcek, UniversityofLjubljana AndreaVandin, SSSUP,Italy LucaViganò, KingsCollegeLondon KimWuyts, imec-DistriNet,KULeuven ApostolisZarras, UniversityofPiraeus

AlejandroCabreraAldaya, Networkand InformationSecurity(NISEC)Group,TampereUniversity ArttuPaju, NetworkandInformationSecurity(NISEC)Group,TampereUniversity JuhaNurmi, NetworkandInformationSecurity(NISEC)Group,TampereUniversity MuhammadOwaisJaved, Networkand InformationSecurity(NISEC)Group,TampereUniversity

NicolaTuveri, NetworkandInformationSecurity(NISEC)Group,TampereUniversity

AlbertoCarelli, LINKSFoundation,Italy

AndreaVesco, LINKSFoundation,Italy

IoHT InternetofHealthcareThings

IoMT InternetofMedicalThings

IoP InternetofPeople IoT InternetofThings

IoTT InternetofTransportationThings

Preface

Afterthecompletionofitsthirdyearofoperationin2022,theCyberSec4Europe1 pilotproject(https://cybersec4europe.eu/)producedthis”BlueBook”(and delivereditasDeliverableD4.7)toserveasaHorizonResearchRoadmapin theareaofcybersecurity.Tomakethisbookareality,theprojectputtogether a”TaskForce”ofyoungandseniorresearchersintheareaofcybersecurity. TheTaskForceproposedaninitialsetoftopicsandreferredbacktoitsconstituency,whichiscomposedoftopcybersecurityresearchers,askingthem whattheimportantresearchproblemsshouldbeinrelationtothesetopics. Theresultofthisconsultationwasadescriptionofeachtopicthatcontained thefollowingaspects:

• Whatisthetopic?Describethetopicandhowitinteractswithcyber security.

• Whoisgoingtobeaffected bycyberattacksinthisarea?ordinary people?organisations?thegovernment?who?

• Whatisexpectedtohappen ifwearesubjectedtosuchcyberattacks? financialloss?lossofproductivity?lossoflife?what?

• Whatistheworstthing thatcanhappenifthingsgoreallywrong? massivelossoflife?awar?financiallossesintherangeofbillionsof euros?what?

• Whatarethemainresearchgaps? Whatdoweneedtodofroma researchpointofviewinordertodealwiththisproblem?Whatarethe importantresearchquestionsthatneedtobeaddressed?

• Exampleproblems.ProvidespecificresearchproblemsthatcanbeaddressedinasinglePhDthesisorinasmallnumberoftheses.

1CyberSec4EuropeisfundedbytheEuropeanUnionundertheH2020ProgrammeGrant AgreementNo.830929.Thispublicationreflectsonlytheauthors’view.TheCommissionis notresponsibleforanyusethatmaybemadeoftheinformationitcontains.

Afterthisconsultation,thetaskforcealsoaskedtheexpertswhatthe GrandChallengesincybersecurity shouldbe.Thesewhouldbetopicsthat wouldneedhundredsofpeopleandseveralyearstosolve.However,ifsolved, theywouldfundamentallychangetheproblemofcybersecurity.

Wehopethatthisbookwillprovideusefuldirectiontoresearchers,will givegoodadvicetopolicymakersandwillproveusefultoallwhoreadit.

HowtoReadthisBook

Policymakers maywanttofocusonChapter 1 (page 1)whichprovidesa shortExecutiveSummaryofthebook,andonChapter 16 (page 111) whichdescribesGrandChallengeResearchProblemsinthearea,which canbesolvedonlywiththecollaborationofseveralresearchorganisationsandthesupportofleadingfundingagencies.

Youngresearchers whoareinterestedindoingaPh.D.insystemssecurity shouldreadatleastthefinalsectionofeachchapter,whichdescribes problemsthatareappropriatetobesolvedwithinthecontextofaPh.D. thesis.

Experiencedresearchers maywanttoreadallchaptersbutespeciallyChapter 16 (page 111),whichdescribesGrandChallengeResearchProblems inthearea.

10Metaverses 59 10.1Introduction .............................. 59 10.2WhoIsGoingtoBeAffected? .................... 60 10.3WhatIsExpectedtoHappen? .................... 61 10.4WhatIstheWorstThatCanHappen? 63 10.5ResearchGaps 64 10.5.1Buildingtrustworthymetaverses .............. 65 10.5.2Metaversesandthephysicalworld ............ 65 10.5.3Compliancebydesign .................... 65 10.5.4Interactivityandimmersivetechnologies ......... 65 10.5.5Metaversesdesign ...................... 66 10.5.6Interoperabilitybetweenmetaverseplatforms 66 10.5.7MetaversesandEnvironmental,Social,andGovernance (ESG)issues 66 10.6Exampleproblems 67 11Malware 69 11.1Introduction .............................. 69 11.2WhoIsGoingtoBeAffected? .................... 70 11.3WhatIsExpectedtoHappen? .................... 71 11.4WhatIstheWorstThatCanHappen? 71 11.5ResearchGaps 72 11.5.1Provablysecuresystems ................... 72 11.5.2Malwaredetection ...................... 73 11.5.3Machinelearninginmalwaredetectionandclassification 73 11.5.4Extendtheplatformscope ................. 74 11.5.5Commandandcontrolservers ............... 74 11.5.6Post-infectionmanagement 75 11.6Exampleproblems 75 12SoftwareLifeCycle 77 12.1Introduction .............................. 77 12.2WhoIsGoingtoBeAffected? .................... 77 12.3WhatIsExpectedtoHappen? .................... 78 12.4WhatIstheWorstThatCanHappen? 78 12.5ResearchGaps 78 12.5.1VerifiableandAuditableSoftware ............. 79 12.5.2ContinuousSoftwareAssessment ............. 79 12.5.3Secure-by-designAgileSoftwareDevelopment ..... 79 12.5.4LightweightFormalMethods ................ 79 12.5.5DecentralisedSoftwareGovernance ............ 80 12.5.6TrustworthyAI-poweredSoftwareLifeCycle ...... 80

1

1.1

project,takingintoaccountinputfromtheproject’sassociatesandexternal experts,haveformulatedanumberofresearchdirectionsthatwillbeimportantforthefuture.Thesedirectionsinclude:

1.2 GrandChallenges

Althoughshort-termprojects1 mayhaveanimmediateimpactonthemarket, suchimpactisusuallyincrementalandmaynotbelong-lastingasitfocuses onanimmediateproblemthatmaynotbesorelevant,say,fivetotenyears downtheroad.Tomakefundamentalbreakthroughsintheareaofcyber security,wehaveproposedseverallong-term“GrandChallenge”problems. Toselectasmallnumberof“GrandChallenges”,themembersoftheTask Force,alongwiththemembersofthebroaderconstituencyproposedseveral such“GrandChallenges”,fromwhichthefollowingwereselected:

1Whenwesayshort-termprojectswemeanprojectsthatlasttwotothreeyearsandhavea fundingoftwotothreemillioneuros.

Introduction

Thepenetrationofcyberspaceintooureverydayliveshasreachedunprecedentedlevels.Although30yearsagotheInternetwasacuriositymostly usedamongacademics,todaymorethan92%ofthehouseholdsintheEuropeanUnionhaveaccesstotheInternet[2].TheEuropeansusetheInternet forseveralaspectsofeverydaylives:morethan50%useitforsocialmedia, around50%useitforInternetbanking,around66%useittofindinformation aboutgoodsandservices,and55%useittoseekhealthinformation[81].The COVID-19pandemicjustincreasedtheuseoftheInternet,asevenmoreeverydayactivitiesmovedonline.Forexample,duringthepandemic,schooling, shopping,andsocialisingcouldonlybedoneonlineforextendedperiodsof time.Althoughthepandemicisathingofthepast,thepenetrationofsome oftheseInternetactivitiesisheretostay.

Althoughmovingactivitiesonlinehascertainadvantages,itmayalsocreatethreatsforpeople.Indeed,asmoreandmoreactivitiesmovefromthe physicalworldtothedigitalworld,thisjustincreasesthe attacksurface.That is,cyberattackershavemoreopportunitiestoattack.Thisissimpletounderstand:ifpeopledotheirbankingonline,thieveswilltrytostealmoney online.Similarly,ifpeopledotheirtelephonecallsusingsomeonlinevideo conferencingsystem,eavesdropperswilltrytolistentotheseconversations onlineviaawidevarietyofoptions:theymayoffersuchasystemforuse forfree;theymaycompromiseoneofthosesystems;theymaybugthesoftwarewithavirusoftheirown;theymay“purchase”suchabuginorderto compromisethesystem.Heretheskyisthelimit.Themostimportantpoint isthatpeoplehavemovedtheirconversationstoonlineplatforms.Oncethis movehasbeenmade,attackerswillthinkofanumberofdifferentwaysto eavesdropontheseconversations.Thesameappliestoallotheractivitiesof oureverydaylives:oncewemoveanactivitytocyberspace,cyberattackers haveawealthofnewopportunitiestoattack.

Havingrealisedthisincreasingthreatintheareaofcybersecurity,thepartnersoftheCyberSec4Europeprojectputtogetheralistofcybersecurityareas thatweshouldfocusonoverthenextfewyears.Theyhaveexplainedthe securitythreatsintheseareasandtheyhaveelaboratedonwhatkindof

cybersecurityresearchneedstobedone.Theareastheyhavestudiedare: anonymity,authentication,criticalinfrastructures,effectivethreatmodelling, IoTsecurity,machinelearning,malware,metaverses,privacybydesign,securityawarenessandtraining,softwarelifecycle,testingandcertification,and trustedexecution.

3 TheErosionofAnonymity

3.1 Introduction

Overthepastfewyears wehaveincreasinglybeen usingcyberspaceformost ofoureverydayactivities: shopping,working,watchingmovies,listeningtomusic,chattingwithfriends,entertaining,etc.Therecent COVID-19pandemicintensifiedthiseffectandforced ustodomostofouractivitiesonline:schooling,shoppingforgroceries,socialising,keepingintouch,almosteverythingwasdoneonline.Insomecases, thingsbecamesoextremethatdoingsomeoftheseactivitiesofflinewas completelyillegal.Indeed,duringthoselock-downperiods,face-to-facevisitstofriendswereillegalinsomecountriesandincurredheavyfines.Thus, duringsuchperiodstheonlywaytovisitfriendswasthroughsomeon-line videoconferencingtool.

Althoughsuchonlineactivitieswereconvenient(orevenabsolutelynecessaryduringthepandemic),theyusuallyrequiredstrongauthenticationand identificationforallpartiesinvolved.Forexample,onlineshoppingwasnot possiblewithanonymouscash,butrequiredtheuseofdebit/creditcardsand possiblyonlinebankaccounts.Deliveryofthepurchasedproductsrequired thedisclosureofthedeliveryaddress,thepresentationofsomeidentifying information,possiblythedisclosureofamobilephonenumber,etc.1 Thesituationwasnobetterforotherformsofinteraction,suchaskeepingintouch

1Althoughthisinformationisrequiredforsuchonlinetransactionsindependentofthepandemic,beforethepandemicpeoplehadachoice:Theycouldoptoutofsuchtransactions.During thepandemicthechoicewasnotthereanymore.

withfamily.Indeed,asaphysical“visit”tofamilywasalmostimpossible, theonlywayofinteractionwasthroughvideoconferencing,whichusually impliedtheinstallationofsomevideoconferencingsoftwarethatneededthe user’sname,theiraddress,andpossiblyacreditcardforpaymentpurposes. Andtomakemattersworse,thissoftwarehadtheabilitytotrackwhois talkingtowhom,andwhattheysay.

ThedisclosureofallthesepersonaldataisinsharpcontrasttothepreCOVIDerawherepeoplecouldcarryoutallthesekindsofinteractionswithouttheneedtodiscloseanykindofpersonalinformation.Thisdisclosure ofpersonalinformationusuallyleadstoalossofanonymity:peoplecannot visittheirparentswithoutinformingseveraldifferentcompaniesonline.The samelossofanonymityhappensinotherareasofourlives.Forexample, inthepastpeoplecouldpurchaseacanofsodafromtheirminimarket,pay cash,andstayrelativelyanonymous.Today,inordertopurchaseacanof sodaonlinetheyneedtodisclosetheirname,theiraddress,theircreditcard details,whiletheymayultimatelybetrackedbydozensofcookies,trackers andadvertisers,whichusetheirdataforallsortsofmarketingpurposes.

Onemightbetemptedtosay:“Itisnotnecessarytocarryouttheseinteractionsonline:wecanalwaysgobacktophysicalinteractions.”Althoughitis nicetohavesuchoptimisticpointsofview,weareafraidthatsoontheremay beno“back”togo“backto”.Onlineinteractionskeepincreasingandthere isnoindicationthattheyaregoingtosignificantlydecrease:onlineshopping isontherise,theuseofsmartphonescontinuestoincrease,andpeopleseem tospendevermoretimeonline.Asaresult,itseemsthatonlineinteractions areheretostayandwejustneedtodealwiththetrackingandtheerosionof anonymitythatcomeswiththem.

3.2

WhoIsGoingtoBeAffected?

Asitismoredifficulttostayanonymousonline(comparedtotheoffline world),mostlaw-abidingcitizenswhousetheInternetwithoutanyspecial anonymisationsoftwarearepotentiallygoingtobeaffectedbythiserosionof anonymity.Itseems,however,thatyoungerpeoplewillbeaffectedthemost, astheycanbeexpectedtospendalongerpercentageoftheirlivesonline. Inaddition,peoplewhohavesomerolethatisvisibletothepublic(such asactors,politicians,etc.)willalsobedisproportionatelyaffected,astheir (private)liveswillbeheavilyscrutinised.Unliketheftofphysicalproperty, erosionofanonymityismuchlikedatatheft:oncethedataaregonethereis usuallynowayofgettingthemback.Itisnotlikestolensilverware,which theownerwillgetbackiftheycatchthethief.Stolendatamaybecopied andgoneforever:thereisno“back”togobackto.Inadditiontopeople,

theircontactswillbeaffectedaswell.Exposingthepersonalinformationof asinglepersonnotonlyharmsthepersonherself,butmaypotentiallyharm anyonewhointeractsonlinewithher:herfriends,relatives,etc.

Inaddition,peoplewhoneedanonymityfortheirphysicalsafetywillbe severelyimpacted.Forexample,peopleinnon-democraticcountriesmayface immediatedanger.Evenpeopleindemocraticcountries,suchaswhistleblowersandjournalists,maybeseverelyimpactediftheycannotoperate anonymously.

Finally,organisationswillalsobesignificantlyaffected.Indeed,informationthatusedtobeconfidentialwithinabusiness(suchasnumberofcustomers,numberofsales,peaktimes,etc.)couldnowbefound(oratleast inferredwithhighaccuracy)bytrackersandadvertisersthatareinvolvedin theinteraction.Onemightthinkthatlargeorganisationswouldbeableto scrutinisetheirwebsitesanderadicateanytrackingdonebythirdparties. Thisisprobablytrue.Itisnotclear,however,whethersmallcompanieswill havetheexpertiseand/orthecapabilitytodosomethinglikethat.

3.3 WhatIsExpectedtoHappen?

Inaworldwhereanonymityisnoteasytoachieve,peoplewilljustnot beabletoactanonymously.Allaspectsoftheiractivitywillberecorded somewhereonlinebysomeonetheyprobablydonotknow:whattimethey wakeup,whattimetheygotowork,whatitemstheypurchase,whatbooks theyread,whatnotestheytake,whatnewstheyareinterestedin,wherethey eat,wheretheyspendthenight,whotheyspendthenightwith—everything isgoingtoberecordedonline.2 Peoplewillhavelittle(ifanyatall)private lifeanymore.Intheabsenceofastronglegalsystemthatheavilypenalises unauthorisedaccesstoinformation,weareafraidthatthisinformationmay eventuallyreachthewrongpeople.Indeed,althoughinitiallyinformation maybesharedwithatrustedentity(suchasourISPorouremailprovider), information,muchlikeanyotherdigitalcommoditymayeventuallybesold, acquired,orevenstolen.Theworstthingofallisthatwedonotreallyknow ifthiswillhappen,orevenifithasalreadyhappened.

Somepeoplemightsay“Ihavenothingtohide”,sotheymaythinkthatit isreasonabletodisclosealloftheiractivitiesonline.However,themainpoint hereisthatonceinformationisdisclosedonlineitmayeventuallyfinditsway tothewrongpeopleormayfallintothewronghands.Ifitfallsintothewrong hands,informationmaycausemajordamagetopeople.Imagine,forexample, organisedcrimesyndicates.Theywouldlovetoknowthewhereaboutsof

2EventhetimeofthedaywhenIamtypingthesecharactersandthetimeofthedaythereader readsthistextismaybebeingrecordedsomewhereonline.

people:whoisalone,whoisonvacation,whichhouseisempty,whichelderly peopleboughtjewellery,etc.Recentstudiessuggestthat78%ofburglarsuse socialmediatofindtheirtargets[3].Theseburglarsusesocialmediatofind picturesofhomes,orevenpicturesofhousekeys[33],toseewhetherpotential targetsareonvacation,tofindtheirdailyroutines,andtoseewhetherthey havecheckedinatarestaurant.Allthisinformationcanbeusedinorder tofindthemostpromisingtargetsandwhenisthebesttimetorobthem. Onemightbetemptedtothink“Oh!Idonotpostsuchinformationonline, thusIamsafe.”Weareafraidthatthisisfarfromtrue.Indeed,severalof theappsinoursmartphones(andespeciallythosethathaveaccesstoour GPScoordinates)knowwhereweare.Theyknowifweareonvacation, theyknowwhichrestaurantwearein,theyknowwhenweleavehome,they knowwhenwereturn,etc.Thefactthatwedonotpostsuchinformationin socialmediadoesnotmeanthatthisinformationisnotrecordedonlineby severaldifferentactorswhohaveaccesstoit.And,aswehavesaid,ifsome informationiscollectedonline,itmaylaterbeshared,sold,orevenstolen.

Itseemsthatmostpeoplearenotawareofthesedangers.Asaresult, theydonotseekanonymityandtheyexposethemselvestomaliciousactors outthere:burglars,robbers,orevenkillers!Forexample,recentresearch on350homicidessuggeststhatbeforemurderingtheirvictimskillersstalk theirvictimsinsocialmedia[198].Theseexamplessuggestthatthislackof anonymousinteraction,inwhichseveralpeopleengage,mayleadtoserious damage:theft,lossofproperty,andevenlossoflife!

3.4 WhatIstheWorstThatCanHappen?

Weareafraidthattheimpactonsocietywillbe muchgreaterthanwhathasbeendescribedsofar. Ifanonymityiscompletelylost,itwillbelikeliving inaworldwhereeachandeveryactivityofoursis beingmonitoredallthetime.Thiswillbelikeliving ina“BigBrother-like”dystopiansociety,whereeach andeveryactionwillbemonitoredandrecorded. Andtheworstpartofallisthatwedonotreally knowwhoisrecordingitandwhohasaccesstothis information.Isitanadvertiserwhowantstoknow whatcolourofshoeswelike?Isitacrimegangthat wouldliketoknowwhichelderlypeoplerecentlyboughtjewellery?Isitthe governmentofahostilecountrythatwouldliketoknowthedailyroutineof thepeopleinourcountryandpossiblybugthemwhentheyvisitonvacation? Wedonotreallyknow.

Weareafraidthatthiscompletelossofanonymitywillnotonlytransform thelivesofindividuals,butwilltransformentiresocieties.Peoplemaybecomeextremelyconservativeandmaybecomeafraidofeachandeveryaction theytake.Insuchanenvironmentpeoplemayrefrainfromexercisingtheir rightsoutoffearthatdoingsomayhaveconsequences;thiswouldseverely damagedemocracyitself.The1984-likedystopiansocietiesthatwemanaged toavoidwillcomeagaintohauntusthroughourownfaultsandourown negligence.

3.5 ResearchGaps

Toaddresstheproblemweneedacombinationoflegalandtechnicalactivitiesinthisarea.

3.5.1

ProvideStrongAnonymousCommunicationatLargeScale

Todaythereareveryfewopportunitiesforanonymouscommunication.The onionrouter(Tor)isoneofthebest-knownones[62].However,lessthan 1‰ofInternetusersuseit.Weneedtoprovideeasy-to-usesystemsthatgive strongprotectionandcanresistpowerfuladversariesunderavarietyofthreat models.

3.5.2

Provideonlinethesamelevelofanonymityyouexpectoffline

Today,anonymityhasbeenimplementedinonlyasmallportionofonline interactions,mostlyinanonymouswebbrowsing.Indeed,theTornetwork mentionedabovecomeswithabrowserthatmakesinstallationconfiguration mucheasierforusers.Thisanonymityshouldbeextendedtoallkindsof interactions,includinganonymousshopping,anonymousentertainment,etc. Theruleofthumbhereshouldbe:ifitcanbedoneanonymouslyoffline,we shouldtrytodoitanonymouslyonlineaswell.

3.5.3 Measure/Monitorthescaleoftheproblem-AchieveTransparency

Itisnotcleartomostpeoplewhatthescaleofthisproblemis:whatisthe amountofpersonalinformationthatisbeingshared.Thewebtrackerskeep inventingnewwaystotrackusersonlineandtodeprivethemoftheabilityto operateanonymously[181].Itisbasicallya“gameofcat-and-mouse”,where trackersinventnewwaysoftrackingandresearcherstrytodetecttheseways oftracking,possiblyviareverseengineering.Weneedtobetterunderstand thescaleandmechanismsoftrackingandlossofanonymity.Weneedto developmechanismsthatcontinuouslymonitorthiserosionofanonymityat alldifferentlevelsinallpossibledifferentways.Thesemechanismsshouldbe abletooperatefrequentlywithoutthecooperationofwebcontentproviders..

3.5.4

NovelDataAnonymizationandDe-Anonymizationapproaches

Weneedtodevelopnoveldataanonymizationmechanismsthatwillallow sharingofdataatalargerscale.Althoughsomedataanonymizationapproachesalreadyexist(see[226],and[68]),thereisstillalongwaytogo beforeanonymousdatacanbesharedonalargescale.Weneedtostudy attackstoexistingdataanonymizationapproachesthataimtode-anonymize thedata,anddevelopdefencesthatwillresultinbetteranonymizationapproaches.

3.5.5 ResistCensorship

SeveralcountriesallovertheglobecensorcommunicationsontheInternet. InsuchsettingsusershavelimitedaccesstotheInternetor,insomecases, noaccessatall.Weneedtodeveloprobustandpracticalsystemsthatbypass censorshipandenablepeopletosafely(andanonymously)publishandaccess information.

3.5.6

Developrobustanti-fingerprintingmethods

Tobreaktheanonymityoftheirusers,severalwebsitesusefingerprinting methods.Suchmethodstrytoidentifyvariousaspectsoftheuser’sbrowser (e.g.browsertype,fontssupported),ortheuser’scomputer(suchaslocallanguage,operatingsystemversion,screensize,etc.)inordertouniquelyidentifyusersastheybrowsetheweb.Althougheachofthesefeaturesalone(such asscreensize)isnotenoughtouniquelyidentifyauser,thecombinationof allofthemisusuallysufficient.Weneedtodevelopstronganti-fingerprinting approachesthatallowlittle(orno)informationtobecollectedabouttheusers astheyroamaroundtheInternet.

3.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

Identityleaks. Monitorhowwebsitesuseallkindsofmechanisms(suchas cookies,URLarguments,URLheaderfields,etc.)totransferpersonal datafromonewebsitetoanother.Developdefencesagainstsuchmechanisms.

MakeAnonymizingNetworksmoreresistanttoattacks. Studypossibleattacksthatmaycompromiseanonymityinanonymizingnetworks.Explorethemagnitudeoftheseattacksandproposepossiblesolutions. InitiallyfocusonwebsitefingerprintingattacksonTor.

Operatewithanonymouspersonas. Developfakepersonasthatallowusers tousethewebwithoutrevealingtheirtrueidentity.Developasystem thatwillclearlyevaluatethetrade-offbetweenusabilityandprivacyin providingfakeinformationindifferentsettings.Explorethesituations wherepersonasprovideaddedutility.

UnderstandingofPrivacy. Improveusers’understandingoftheirprivacyrelateddecisions,suchasthecookieconsentformsthattheyagreeto. Develop(semi-)automatedtoolsthatimprovethisunderstandingand quantifythechoicesmadebytheusers.

DataProvenance. Developsystemsthatenableuserstodetecttheprovenance ofdataandthusdiscoverstolen/leakeddata.Addresstheproblem fordifferentkindsofdataincludingtimeseries,images,videos,multidimensionalsignals,etc.

4 MachineLearning

4.1 Introduction

MachineLearning(ML)hasbecomethe technologypoweringawide-rangeofapplicationsandservices.Theperformance andthegeneralisabilityofMLmodelsmade themagoodcandidatefortacklingaseriesofreal-lifeproblemsthatexhibithigh complexity.TakeforexampletherecentadvancesofGenerativeAdversarialNetworks (GANs)thatmanagetosynthesisehighlyrealistichumanfaceswithasmallnumberofreal-worldsamples[127].Generallyspeaking,ML-basedsystemsmanagedtoachievehighsuccessrateson problemswheretheclassicrule-basedapproachesdidnotperformwell.

Nowadays,MLhasbeendeployedinmanysectorsofoureverydaylives. Forexample,duringouronlineshoppingonEbayorAmazon,anML-based personalisedrecommendersystem,runninginthebackground,proposesproductsaccordingtodifferentparametersrelatedtotheuser,e.g.thehistoryof previouspurchasesandthetimespentlookingataspecificproduct.Inaddition,theautomotiveindustryhasincorporatedMLtechnologiesintotheir carstomakethemdrivethemselveswithoutanyhumansupervisionwhatsoever.Furthermore,ML-basedNaturalLanguageProcessing(NLP)techniques havebeendevelopedforimprovingthesafetyofonlinediscussionenvironments,e.g.todetecttoxic,sarcastic,harassingandabusivecontent[169].In general,MLtechnologieshavebenefitedvarioussectors,someofthembeing thefollowing:medicaldiagnosis[131],detectionofcreditcardfraud[146], stockmarketanalysis[41],bioinformatics[63],speechrecognition[99],object detection[40],androbotlocomotion[129].

TograspthepotentialofMLalgorithms,itisenoughtosaythatmany techgiants,suchasGoogleandAmazon,offerMachineLearningasaService (MLaaS)platforms,wheretheuserscanuploadtheirowndatatotraintheir ownMLmodelsandsolveaspecificclassification/predictiontask.Thus,the

users’data–whichinmanycasescontainsensitiveinformation,suchasmedicalrecords,photosandotherpersonaldescriptors–isusedasthetraining databytheMLaaSplatforms.Additionally,someMLaaSoperatorsmaygive dataownerstheoptiontosellaccesstotheirtrainedMLmodelstothegeneral public.

DespitethemassivesuccessofMLintacklingnumerousdifficultproblems,severalsecurityandprivacyvulnerabilitieshavebeenshowntocoexist withthesemodels[142, 182].Forexample,thinkofthecasewhereanNLP modelmisclassifiesamovie’sreviewas"excellent"insteadof"bad".This(misclassification)errorresultsinahigherscoreforthatparticularmovie.Thus, usersthatconsultaspecificsiteformovieratingswillbeluredtowatchthat moviebecauseofitshighrating.Afterwatchingthatmovieuserswillrealise thatitwasnotasgoodastheratingsitesuggestedand,asaconsequence, avoidusingthesamesiteagain.Onamoreseriousnote,thinkofthecase whereanimagerecognitionmodelisdeployedonanautonomousdriving vehicleforidentifyingroadsigns.Ifanattackerdeliberatelyperturbstheinput(video)totheimagerecognitionmodel,thenthemodelmightwrongly recognisea“stop”signasa“minimumspeedlimit”signandaccelerateinsteadofstoppingthecar.Asyoucaneasilyimagine,suchattackscanhave seriousconsequences,evencausingfatalities.Inconclusion,sinceMLhas dominatedacrossmanysectors,weneedtocomeupwithsolutionsforensuringitssecureoperation.

4.2 WhoIsGoingtoBeAffected?

SincethewidespreadadoptionofMLmodelsintoavarietyofservicesand applications,anyonewhohasaccesstoamoderndevice(e.g.asmartphone, apersonalcomputer,avehicle,orevenahomeappliance)canbeaffected.In general,anyindividualwhopossessesanelectronicdevicecanbeaffected. Nonetheless,youngstersareexpectedtobeaffectedtoalargerdegreecomparedtoolderindividuals,sincetheyoftenutilisenewertechnologiesand applicationsthatareoftenpoweredbyML[124].

AlargeportionofML-basedapplicationsareoftentrainedonpersonal (sensitive)data.Leaksofsuchdatamayleadtoseriousconsequencesforthe affectedindividuals.ThinkofthecasewhereanMLmodelistrainedtoassociateapatient’sinformationwithaspecificdiseaseclass.Ifanadversary knowsthatapatient’sdatawasincludedinthemodel’strainingdataset,they candrawconclusionsaboutthevictim’shealthstatus(knownasmembershipinferenceattacks[211]).Inasimilarfashion,ifanadversarymanages tosuccessfullygenerateinputsresemblingtheoriginalonesusedfortrainingthetargetmodel,thenthismightenablethede-anonymisationofusers

andexposepersonalorsensitiveinformation(knownasmodelinversionattacks[85]).Finally,adversarialimagegenerationattacks,whereanadversary introducesslightmodificationstoanexistingimageinordertoconfuseor deceiveanimagerecognitionMLmodelintoperformingamisclassification, havebeenproposedintheliterature[96].

Finally,companiesthatprovideML-basedsolutionsmayalsobeimpacted, inadditiontoindividuals.Inparticular,disclosingthattheMLservicesofferedbyacompanyarevulnerabletotheaforementionedattackscanseriously harmthatcompany’sfinancesandreputation.

4.3 WhatIsExpectedtoHappen?

Thegeneralisationability(performance)ofML-basedapplicationsheavilydependsonthequantityofavailabletrainingdata.But,asthetraining datavolumegrows,sodoesthechancethatsensitivedatawillbepresent. Thus,itisrealistictoassumethattheattentionofpotentialadversariesand maliciousgroupsisgoingtobefocusedonattackingsystemsthatutiliseML components.

ThenumberofdatabreachincidentsthatexploitMLcomponentswill increaseinthenearfuture.ThisisbecauseMLmodels,runninginthebackgroundandcollectingsensitivepersonaldata,willbedeployedwithinmore andmoreapplicationsandservices.Thus,potentialadversarieswillhave accesstoawiderrangeofexploitabletargets.

MLmodelscanbedeployedinsectorswherewrongdecisionmakingimpliesseriousconsequences(e.g.inhealthcare).Thus,legislations/regulations willbedrawnupinordertoexplicitlystatetheliableentitiesincasesomethinggoeswrongornotasexpected.Inaddition,thesecurityandprivacy standardsthatmustbemetbydeployedMLmodelswillbereleased.These standardswillensurethatdeployedMLmodelsarerobustagainstspecific (known)threats.Finally,guidelinesforbestpracticeswillbeformedinorder tohelpnonML-expertdeveloperswhowishtoincorporateMLtechnologies intheirapplications.

4.4 WhatIstheWorstThatCanHappen?

Asalreadymentioned,itisexpectedthatthenumberofdatabreachincidentscausedbytheexploitationofMLcomponentswillincrease.Inordertopreventpotentialexploitsthatcouldhaveseriousrepercussions,relevantauthorities,suchastheEuropeanUnion(EU),shouldkeeptakingbold steps(e.g.seePupilloetal.[144]andENISApressreleasesonAI/MLsecurity[73, 74, 76])tostrengthenthesecurityandprivacyofML-basedsys-

temsandservices.Onlywithsuchconcreteregulations/policiesinplacewill thecommunityexperiencethefullpotentialofMLtechnologies.

Moreover,forcompaniesthatofferML-basedsolutions,potentialattacksontheirsystemsmayimply millionsofdollarsinfinancialdamageandlossofreputation.Inaddition,attackssuchasthosedescribed inSec.4.2mightmakealargeportionofML-basedsystemsunusable.

Last,butnotleast,thedegreeto whichpeopletrustML-basedsystemswillbegreatlydecreasedifappropriate securityandprivacymeasuresarenotconsidered.Thisisimportant,because thetraction(usage)ofsuchsystemswillbedecreasedaswell.Peoplewill behesitanttoprovidetheirvaluabledatafortrainingMLmodels.Thus, advancesinML,andartificialintelligence(AI)ingeneral,willdeclinesignificantly.Infact,peoplewillbecomesosuspiciousoftechnologythattheywill behesitanttouseit.Muchlikethe5Gcase,wemayevenseeuprisingsand protestmovementsagainstMLtechnologies.

4.5 ResearchGaps

InordertoimprovethesecureoperationofML-basedsystemsseveral actionscanbetaken.

4.5.1

Exploringthesecurityandprivacyrobustnessofstate-of-the-artML modelsunderdifferentadversarialscenarios

Sofar,thescientificcommunityhasidentifiedanumberofsecurity/privacy vulnerabilitiesthatcoexistwithstate-of-the-artMLmodels.Nonetheless,exposingthosemodelsindifferentadversarialscenariosmightrevealadditional vulnerabilitiesfromwhichtheymaysuffer.Discoveringthoseweaknesses willsignificantlyaidthecommunityindevelopinggenerallyapplicabledefencesordesigningimprovedarchitecturesintermsofprovidingspecificsecurity/privacyguarantees.Inthatsense,MLauditingframeworkscanbe developedthatwillbesolelyresponsibleforevaluatingtherobustnessofML modelsagainstspecificsecurity/privacythreats.

4.5.2

DesigningarchitecturesandtrainingalgorithmsforincreasingML models’generalisationandrobustnessagainstsecurity/privacyattacks

AnumberoftheattributesofMLmodelsmightberelatedtotheirvulnerabilitytospecificsecurity/privacyattacks.Forexample,membershipinference attacks(MIAs)havebeenshowntobemoreeffectiveonoverfittedmodels (i.e.modelsthatdemonstratelowgeneralisation)ratherthanwell-generalised ones[211].Inaddition,thearchitectureofthemodelitselfhasbeenshown toplayanimportantroleinitsvulnerabilityagainstMIAs.Inthatsense,researchershavedemonstratedthatanaiveBayesmodelismuchmoreresilient toMIAscomparedtoadecisiontreeand,therefore,maybethepreferred modeltypeforaparticularMLservice[234].Thus,MLmodelarchitectures thatofferincreasedrobustnessagainstsecurityandprivacyattacksshouldbe developed.

Inasimilarfashion,thetrainingofMLmodelsshouldbeoptimisedtowardsofferingincreasedsecurityandprivacyguarantees.Forexample,DifferentialPrivacy(DP)[68]offersprobabilisticguaranteesabouttheprivacy ofindividualrecordsinadatabase.DPretainstheglobalstatisticaldistributionofadataset,anditscontributiontoanMLmodel’sweights,while atthesametimereducingtheinfluenceofeachtraininginstance.Similarly tok-anonymity[139]anddiversification[9],DPcanbeusedtomitigatethe riskagainstvariousprivacyattacks,suchasmembershipinferenceandreidentification.TheapplicationofDP,however,imposesatrade-offbetween securityandutility(usefulness).Inotherwords,thestrongerthesecurity guaranteesthatDPoffers,thelargerthenegativeimpactonthemodel’sperformance.Thus,noveltrainingalgorithmsandtechniquesthatmaximisethe security/privacyguarantees,whilealsosacrificingaslittleperformanceas possible,shouldbedeveloped.

4.5.3

OnthetransparencyandinterpretabilityofdeepMLmodels

MLmodelsareoftenviewedasblackboxesthatcanmakeadecisionbased onanypossibleinputvariant.ThecomplexnatureofMLmodelsmakes theirinnerworkingsdifficulttocomprehend.However,whatisdifficultto understandisalsodifficulttoaudit.Andwhatisdifficulttoauditisalso difficulttotrust.Generallyspeaking,thelevelofmodeltransparencydepends ontheknowledgerequiredtounderstandtheinternalmechanicsoftheML algorithm.

TherearequiteafewMLalgorithmsthatdirectlyorindirectlyproduce humancomprehensibleoutput,suchasalinearmodeloradecisiontree. Supposethatwecantracethechainofreasoningofeachdecisionthatan algorithmmakes.Canweclaimthealgorithmistransparent?Theanswer

isunfortunatelyno.Thechainofreasoningonlytellsus“how”adecision wasmadeforagiveninputbutnot“why”.Forexample,knowing“how” isnotsufficienttojustifythatthedecisionismadeconsistently,accurately, reliably,andvalidly.Thus,foralearningmodeltobetrulytransparentwe needtoknowboth“how”and“why”.Duetothehighcomplexityofdeep MLmodels,whichoftenincorporatehundredsoffully(orpartially)interconnectedlayers,apromisingapproachforincreasingtheirtransparencyand interpretabilityistoprovidejustificationsandinsightsforthedecisionsthat canbegaugedexternally.

4.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

ExploringsecurityandprivacyattacksonMLmodels. AnimportantdirectionforenhancingthesecurityandprivacyofMLalgorithmsistoreveal additionalhiddenvulnerabilities.Apartfromthealreadyestablishedsecurity/privacyattacks,suchasmodelinversion,membershipinference, modelextractionandadversarialsamplegeneration,additionaleffortis requiredtodetermineotherpossiblethreats.Inaddition,weneedconceptsandtechniquestomeasurethevulnerability/robustnessofML.

Proposinggenerallyapplicabledefencestrategies. Anotherinterestingdirectionisthedevelopmentofgenerallyapplicabledefences,morespecifically,defencesthatcanbeappliedtoexistingtrainedMLmodelswithouttheneedforretraining,whichisatime-consumingprocessand wouldrequirelargecomputationalresources,oranymodificationsto theirarchitecture/trainingalgorithm,whichwouldrequiresignificant manualinterventionfromexpertsinthefieldofAIandML.

Applyingvulnerablelearningmodelsinasecureway. Onemightsaythat perfectlysecureMLisprobablyanillusion.Thus,insteadoffocusingon increasingtheirrobustness,analternativedirectionisfocusingonhow toapplytheminsuchasecurewaythatexploitingthoseMLmodels becomessignificantlyharder.

Developinghuman-friendlyinterpretabilitytechniques. Thisangleinvolves thedevelopmentofsystemsandservicesthatareabletoprovidehumanfriendlyexplanationsforthedecisionsofcurrentstate-of-the-artML models.Whenwereferto“human-friendlyexplanations”,wemean justificationsthatarepreferablysimpleenoughforpeoplewhoarenot expertsinthefieldsofAIandMLtounderstand.

5 Authentication-BeyondPasswords

5.1 Introduction

Nowadays,theincreasinguseof cyberspacerequireseachpersonto haveseveralaccountsinorderto accessthesystemsandwebapplicationsnecessaryforeverydayactivities.Oneoftheoldestprotectionmechanismsofsystemsand webapplicationsistheauthenticationmethod,wheretheuserisasked toprovehis/heridentitytogainaccess.Themostcommonmethod ofauthenticationbyasystemor anapplicationisviatheso-called username-passwordmethod.Inthismethod,theuserhastoprovidetheusernameandthepasswordthatwerechosenduringtheaccountcreationprocess (registration).Despitethefactthatusername-passwordisoneoftheoldest authenticationmethods,itisstillusedbyalmosteverysystemandapplication(bothonlineandoffline).Forinstance,adoctorinahospitaldeploysthe username-passwordmethodtoaccessheraccountinboththehospitalandan onlineshop.Duringthepastfewyears,thenumberofaccountseachuser maintainshasgreatlyincreased;consequently,usersfinditdifficulttomemorizeandmanageallthesepasswords.ArecentstudybyNordPassshowed thatanaveragepersonhas100differentpasswordstoremember,leadingtoa problemcalledpasswordoverload[199].Moreover,theusername-password paradigmissubjecttovariouscyber-attacks,suchasrecoveringapassword fromitsleakedhashthroughbruteforce(passwordcracking),recoveringa passwordwhentransmittedthroughanuntrustedchannel(eavesdropping), trickingauserintoenteringhis/herpasswordonanuntrustedorcompromisedendpoint(phishingwebsites,ATMskimmers),orallowingtheuseof defaultpasswordsthatcanbeusedbyadversaries[28][174][8][83].

Henceinorderforapasswordtobeconsideredstrong,assuggestedby Microsoft,itshouldcontainatleast12characters,becomplex(i.e.contain alphanumericcharacters,numbers,symbols,andnon-dictionarywords),be differentfromotherpasswordstheuserusedinthepast,andbedifficult forotherstoguess[152].Alltheseconditionsalongwiththehighnumber ofdifferentaccountshaveaffecteduserswhofinditdifficulttomemorise (StrengthofMemorizedSecrets[173])andmanageallthesepasswords.To solvethispasswordoverloadproblem,usershavecomeupwithsolutions thatdirectlyaffectthesecurityoftheiraccountsandtheprivacyoftheirdata; theyeithersimplifytheirpasswordstobeeasytoremember,reusethesame passwordondifferentservices,orstoretheirpasswordsina“secure”place, forexampleonpaperorusingapasswordmanager.Butevenifthepassword isstrongandtheuserhandlesitappropriately,theserviceprovidersalso havetokeeptheirendofthedealandstoretheirusers’passwordssecurely. NISTprovidessuggestionsonhowtoproperlystorepasswordsondatabases (MemorizedSecretVerifiers[173]),thoughmanypopularopensourceserver softwaredonotofferadequatesecuritybydefault[170]andanumberofdata breachesexposedimproperlystoredpasswords[113].

Severalmethodshavebeenintroducedtoenhancetherobustnessofthe authenticationprocess,especiallyoncriticalsystemsandapplications;with thebestknownbeingtwo-factorauthentication(2FA),alsorecommendedby ENISAtoimprovepasswordsecurity[77][5].Duringa2FAmethod,the userhastoprovehis/heridentitybasedontwofactorsratherthanone.For instance,toaccessawebbankingaccount,apartfromprovidingtheusername andpassword,theuserisalsoaskedtoprovideaone-timepassword(OTP) thatisreceivedviaaShortMessageService(SMS)inordertobeauthenticated. Althoughthismethodimprovesthesecurityoftheauthenticationprocess,it lacksuser-friendliness[148],whichisanimportantfactorintheauthentication procedure,andcanalsobeexploitedthroughSIMswapattacks(wherethe adversarymanagestoclonetheSIMcardofthevictim,allowinghimtosteal theSMS)orbytrickingtheuserintorevealingtheOTPcodethroughafake call,websiteoremail(phishing).

5.2

WhoIsGoingtoBeAffected?

Anyonewhousesacomputerorsmartphoneisgoingtobeaffectedbythe weaksecurityofpassword-basedauthenticatedmethods.However,people withmoreaccountsaremorelikelytobeaffected,sincetheattacksurfaceis widerinotherwords,attackershavegreaterchancestocompromiseanaccount.Forexample,ifBobhasoneaccount(e.g.anemailaccount)andAlice hasthreeaccounts(e.g.email,onlineshopandstreamingaccounts),then

anattackercantargetBobononeapplication,whileAlicecanbetargeted onthreedifferentapplications.Apartfromindividuals,companies/organisationsmightalsobeaffected,sinceifanemployee’spassword-onlyprotected accountiscompromised,corporatedatacouldbestolenormalicioussoftware maybeplanted,resultinginjeopardisingthereputationofthecompany/organisation,whichwillleadtomoneyloss.Lastbutnotleast,governments andcriticalinfrastructureswillbeaffectedthemost,becauseifanattacker weretogainunauthorisedaccess,theirmaliciousactionsmightalsohavea seriouseffectonEuropeancitizens.Forexample,thecompromiseofapower gridwillsignificantlyaffectthepublic.

5.3 WhatIsExpectedtoHappen?

Inacasewhereasystem’sauthenticationiscompromised,notonlywill theuser’sdatabeatrisk,butalsotheattackerwillhaveaccesstothesystem toperformvariousmaliciousactions,suchasstealingpersonalinformation ordocuments,installingsometypeofmalware,orperforminganAdvanced PersistentThreat(APT)attack.Thus,theconsequenceswillvarydependingonthecriticalityofthesystemandtheattacker’sactions.Inmostcases, compromiseoftheauthenticationprocessleadstoadatabreachandmoney loss.However,whentheauthenticationprocessofacriticalinfrastructureis atstake,theconsequencesmightbemuchmoreseverethanthelossofmoney. Theworstthingisthatwecannotknowbeforehandthemaliciousactionsthat anattackerwillperform.

5.4 WhatIstheWorstThatCanHappen?

Mostofourdigitalservicesrelyuponsecureauthenticationoftheusers, andthuswehavetomakesurethatweuseadequatelysecureauthentication methods.Assumingthatwewillcontinuetobaseallofourauthentication methodsonpasswords,eventuallyeverysystemwillbecompromisedatleast once.Everycompanywillbeaffectedbyincidentsanddatabreaches,resultinginmillionsofeuroslost.Massiveamountsofpeople’sleakeddata (e.g.email,photos,residence,socialnumber,telephone,creditcardnumbers, financestatus,medicalrecords,etc.)willbeavailableonlinetothehighest bidder,thusaffectingprivacysignificantly.

Criticalinfrastructureswillalsobeaffecteddeeply.Suchinfrastructures (likethepowergrid,watersystems,hospitals,telephonecommunications) stillconnectedtotheinternetwillpossiblyposeathreatduetothehighrisk ofbeingcompromised.Cyber-attackswilltargetsuchsystems,creatingahigh riskofespionage,cyberterrorismorevenlossoflives.

5.5 ResearchGaps

Toensurewekeepauthenticationmodulesadequatelysecure,actionsneed tobetakeninthisarea.

5.5.1 Improvepasswordlessauthenticationmethods

Althoughpasswordlessauthenticationmethods—suchasthefingerprint unlockofourphones—arealreadyavailable(e.g.FIDO[6]andWebAuthn[110]),thereareanumberofpointsthatcanbeimproved.Tostartwith, weshouldmakesurepasswordlessauthenticationisaccessiblebyeveryone (thoughourpersonaldevicessuchasoursmartphonesorpersonalcomputers)inauser-friendlyyetsecureway,byincreasingtheadoptionofpasswordlessauthenticationmethods(e.g.increasethenumberofapplicationand websitethatsupportpasswordlesslogin)andimprovingtheinteroperability betweenauthenticatordevicesandservicesrequiringauthentication(e.g.use thefingerprintsensoronyoursmartphonetoauthenticateonyourlaptop).

Sinceseveralpopularpasswordlessauthenticationmethodsrelyonbiometrics (e.g.irisscan,fingerprintscan,facescan),lookingintoensuringthesecurity andtrustworthinessofbiometricauthenticationmethods(e.g.byreducing thefalsepositiveswhereanunauthorisedentitymaybefalselybeidentified asanauthorisedone)whilealsorespectinguser’sprivacy(e.g.securelystoringbiometrics’relateddatalocallyonlyforusetoauthenticatetheuser)isof highimportance,whilealsolookingintohowtheycanbeusedalongwith fuzzycryptography(wherebiometricdatacanbeusedasaninputtocryptographicfunctions).Furthermore,theusageofpasswordlessauthentication inadvanceauthenticationscenarios(e.g.multipartypasswordlessauthentication,wheretheauthentication/authorisationisperformedbymorethanone entities)shouldbeinvestigatedinordertomeetspecialisedneedsthatexisting methodsdonotcover(e.g.allowingtheauthorisationofatransactionorthe signingofadocumentby2ormorepeople).Weshouldalsolookintonovel passwordlessauthenticationapproachesforbothonline(e.g.loggingintoan onlinewebsite)andofflineusage(e.g.loggingintoyourlaptop).Lastbut notleast,thereneedstobeinvestigationintoimprovingtheauthenticationof usersbyleveragingexistingtechnologies(e.g.SingleSignOn)andnewdigital identityschemes(e.g.Self-sovereignidentity,DecentralizedIdentifiers[219], VerifiableCredentials[220])incombinationwithpasswordlessauthentication (whenneeded),aswellassecurerecoveryorfallbackmechanicsforusewhen themainauthenticationmechanicisnotavailable(e.g.incaseyouloseyour smartphoneoryourUSBsecuritykey).

5.5.2 Measure/monitortheuseofinsecureauthenticationmethods

ItisofhighimportancetomonitorthesecuritystateofauthenticationmethodsinEurope,bymeasuringboththeadoptionofpasswordlessauthenticationandtheuseofinsecurepassword-basedauthenticationmethods.With betterinsightintotheproblem,measurescouldbetakentoreducethesecurityrisk.Forinstance,wecanintroducenewregulationsorimproveexistingones,targetingcriticalsystemsaffectedbytheproblem,aswellasset minimumsecurityrequirements(e.g.appropriatecertification,securityassessmentsandauditing)toensureanappropriatelevelofsecuritytoprotect Europeancitizensandoursocietyasawhole.Researchcouldalsofocuson theeconomicsideoftheissueandinvestigatewhetherbetterandnewerauthenticationmechanicsareaffordablebyallkindsoforganisationsorwhether suchtechnologicalsolutionsdonotfitthebill.

5.5.3 Understandinguser’spsychologyrelatedtoauthentication

Animportantresearchgapisrelatedtothehumanpsychologyandauthentication.Furtherresearchintotheuser’spsychologyduringauthentication shouldprovidemoreinformationrelatedtodeceptionattacks(socialengineeringrelatedattacks)aswellasprovidingvaluableinformationabouttheuser’s perceptionregardingtheusabilityofanauthenticationmechanism.

5.5.4

EnhancingbiometricauthenticationmethodsusingAImethods

Biometricauthenticationmethods,suchasfingerprint,face,andvoicerecognition,areheavilyutilisedinsmartphonestologinuserswithoutpasswords. Yetthosemechanismscomewiththeirownlimitations.Tonameafew,dirty handswillaffectfingerprintrecognition,weatherconditionsfacerecognition, andloudenvironmentsvoicerecognition.Thus,furtherresearchisrequired toalleviatethoserestrictions.OneapproachcouldbetoemployAImethods, suchasmachinelearning(ML)anddeeplearning(DL),intheauthentication processtomakethebestoftheincomingdataincaseswheretheconditions arenotoptimal.

5.5.5 Continuousauthentication

Theuser’suniquecharacteristicscanbedeployedforauthenticationwithout needinghis/herinteraction.Forinstance,inthecaseofamobilephone, eachpersonholdshis/herphonedifferently,typesdifferently,swipesfrom differentangles,etc.Utilisingallthisdataregardingeachperson’sbehaviour andleveragingAIcanresultincontinuousauthenticationwithouttheuseof passwordsorbiometrics.Researchonthistopicshouldfocusonincreasing theaccuracyofthebehaviouralauthenticationmechanisms,atthesametime

reducingthefalsepositivesandfalsenegatives,whilealsolookingintohow topreserveuser’sprivacyanduser’scontrol.

5.5.6 Trainingpeopleinauthenticationrelatedtopics

Thereareseveral2FAmethodsthatcanbeusedtodayincombinationwith passwordstoprovideadequatesecuritytosystems,butmostusersoptnot tousethem.Theresearchcommunitywillhavetolookintothereasons whymanyusersdonotenablepasswordlessormulti-factorauthentication (MFA)anddevelopefficientusertrainingtotackletheissue.Furthermore, althoughvariousrecommendationsonhowtohandlepasswordsexist,both usersandsoftwareengineersstillfailtofollowthemresultinginhandling theminsecurely(e.g.userscontinuetosharepasswords,engineerscontinue tostorepasswordsinsecurely).Conductingrelatedtraining(orincreasing theirefficiency)willensurethateveryonehasaccesstoandknowshowtouse correctlyandeasilystrongauthenticationmechanics,minimisingtheriskof theiraccountsbeingcompromised.

5.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

Userfriendliness. Researchshouldbeconductedonhowtheuserfriendlinessofpasswordlessauthenticationmethodscouldbeimproved.Apart frommakingthemethodseasiertouseforthegeneralpublic,neweasy tousemethodologiestotransferorbackupcredentialsusedbyauthenticatordevicesshouldbetested.

Transitionfrompassword-basedtopasswordless. Inmanycasesthetransitiontonewerpasswordlessauthenticationmethodsisnottrivialasmany systemsdonotsupportthemoutofthebox.Furthermore,usersnotfamiliarwithpasswordlesstechnologymayfacedifficultiesinpreparing theirenvironmenttousethenewauthenticationmethods.Furtherresearchinthetopicmaylookintohowtointroducepasswordlessauthenticationinauser-friendlywayandasasecuritylayerwrappinglegacy system.

Resistancetoattacks. Tosecureourfutureweshouldalsolookintohowattackscanbemitigatedandhowmeasurescouldbeintegratedintoour passwordlessauthenticationmethods.Inmanycasessuchproblems mayariseasaresultofinsecureconfigurationorfaultyimplementation,whileinothercasestheyareamongthedisadvantagesoftheselectedmethod(e.g.somepasswordlessauthenticationmethodsarenot phishing-resistant).

WeakauthenticationonIoTdevices.

TheintroductionofIoTdevicestoour livesandtheirinterconnectionandexposuretotheinternetcreateda newattacksurfaceforattackers,namelyattackersapartfromtargeting userauthentication,attackerscannowtargetthedeviceauthentication process.Novelpasswordlessauthenticationmethodsshouldbeintroducedforsuchsmallsmartdevices(e.g.remotelyaccessibleIPcameras) thatusuallyfeaturelimitedresources.

SecurityAwarenessandTraining

6.1 Introduction

Organisationalcybersecurityiswidelyacknowledgedtorelyonthreepillars: namely,technologies,processesandpeople.Additionally,transformingraw dataintoeligibleinformation,andinformationintoactionableintelligence,is anincreasinglysignificantcomponentofmaintainingsituationalawarenessof cybersecurity.

Peopleareoftenperceivedastheweakestlinkinthecybersecuritychain [32][164].Thoughthisnegativecharacterisationofhumannatureisdebatable [123],itisundeniablethatthehumanisamajorcontributingfactortothe majorityofcybersecuritybreaches[128].Cybercriminalsfrequentlyemploy techniques,suchassocialengineering,thatexploitinnatehumanweaknesses tocarryoutattacksandtoimprovetheirchancesofsuccess.

Cybersecuritycompetencedevelopmentfocusesonenablingpeopletoestablishtechnicalandoperationalbarrierstocybersecuritythreats,andtoconductthemselvesappropriately,throughthevigilantprocessingofactionable intelligence.Itisaniterativeprocessofcontinuousandincrementalimprovement[249]targetedtowardtransformingthehumanfactorfromapotential attackvectortoamultiplieroforganisationalpreparednesstoprotectagainst, detect,respondtoandrecoverfromcyber-attacks.Cybersecuritycompetence developmentisbasedonacontinuumthatexpandsformaleducationthrough addedvalueactivities,suchasi)hands-onexperience,ii)awarenessprogrammesandiii)trainingprogrammes,witheachofthesemultipliersserving particularfunctionsinmaintainingorganisationalcyberhygiene.

Leveraginghumanfactorsincybersecuritygoesbeyondtraditionaltrainingandawarenessmethods.Itcallsformodernapproachesthatdrawonunderstandingshumanbehaviourandimplementingtoolsthatprovidetargeted cybertrainingandawareness.Hands-onexperience(alsoknownaslearning bydoing)isanextremelyeffectiveapproachtoteachingandlearningcybersecurity[213].Itengagesthelearnersandimprovesknowledgecomprehension andretention,aswellasthepossibilityoftranslatingacquiredknowledgeinto action[90].Manysuccessfulstrategiesareusedforthispurpose,including exercisingcyber-attacksdetectionanddefenceskillsinacyberrangeenvi-

ronment[42],participatingincybersecuritycompetitions[162],participating inflagshipcybersecurityexercises[55],andlearningthroughgameplay(e.g., seriousgames)[207].However,integratingcybersecurityawarenessandtrainingonlyreduces,noteliminates,thepossibilityofhumanneglectanderrors, implyingthatsmarttechnicalinterventionstocheckandregulateemployees’ mistakesremainvitalforanorganisation’soverallcybersecurityposture[143].

6.2

WhoIsGoingtoBeAffected?

Asmentionedearlier,cybersecurityiswidelyacknowledgedtorelyonthree pillars:namely,technologies,processesandpeople.Humanscanbenegligent arepronetoerrors,andcanrepresent,eitherintentionallyorunintentionally, aweaklink[164].Therefore,technologiesandprocessesaimtoreducethe overallburdenorresponsibilitybyautomatinganddemarcatingprocedures, asweseethroughtheongoingdigitaltransformation[161].However,itis peoplewhodevelop,operationaliseandmaintaintechnologiesandprocesses. Thus,whiletechnologiesandprocessesconstituteessentialtoolsforcybersecurityhardening,thehumanfactorplaysthemostcriticalroleinensuring cyberhygiene.Regardlessofhowmanyexpensiveandsophisticatedtechnologicalsecuritysolutionshavebeendeployed,theycannotbeconsidered secureaslongashumanfactorsdonotworkandbehaveinasecuremanner. Moreover,technologicalsecuritysolutionsrequirehumaninputforproper andeffectivefunctioning:forexample,firewallsmustbeactivated,software mustbeupdated,andsecuritywarningsmustbeacknowledgedandacted upon.

Lackofemphasisonsecurityawarenessandtraininghaspersonal,organisational,andevennationalramifications,whileimprovedvigilance,or lackthereof,permeatesandspillsoverbetweenthepersonalandprofessional spheres.Weseetheripplingeffectsoflowawarenessandknowledgeacross nearlyallcybersecuritytopicsandsectors[179],fromprivacyimplicationsto criticalinfrastructuresecurity[43].Humanbehaviour,moreoftenthannot,is thesoftunderbellyofsecuritydesignsandarchitectures,presentingtopotentialattackersapathofleastresistance,ifnotaclearentrypoint,withalimited technicalthreshold.Therefore,thechallengeisnottodeterminewhowillbe affectedbylimitedcybersecurityawarenessandtraining,buttoidentifywho maynot.

Itmustalsobenotedthattheoverallimpactofdigitaltransformation highlydependsontheacceptanceofthenewlydevelopeddigitaltechnologies,referringtoboththosethataredevelopedwithacybersecurityfocus andthosethatarenot.Cybersecurityawarenessandtrainingcanfacilitate stakeholderacceptanceandadoptionofinnovativedigitaltechnologies,asit

enhancesunderstandingoftherelatedcybersecurityrisksanddevelopsactive barriersagainstthem.

6.3 WhatIsExpectedtoHappen?

Automatedandautonomoussystemshavebeendevelopedacrossseveralsectorsincludingcybersecurity[161]toassisthumansoreventoremovethem fromtheloop.Nevertheless,thisprocessisstillinitsinfancy,andevenin developingthosesystems,peoplearetheprincipalcontributors[221].Additionally,cybersecuritybestpracticeshavebeendevelopedacrossallthephases ofsecuresystemsengineering,fromplanningallthewaytodisposal.However,theseprocesses,whetherreferringtosystems,policiesorprocesses,oftenincludeinputsthatarebiasedbyqualitativeexpertknowledge[101],or requirecompromisestomeetrequirementsandconstraints.Limitedcybersecurityawarenessandtrainingrepresenttherootcausesofvulnerabilities introducedwithinthedeployedsystems,technologies,processesandpolicies. Thisoccursacrossallthestagesoftheirlifecycle,arisingfromseveralfactors, suchasdesignflaws,integrationmistakes,oroperationalnegligence.Theexactimpactandconsequencescanonlybeestimatedonacase-by-casebasis. However,itiscriticaltoacknowledgethatpromotingtargetedcybersecurity awarenessandtraininginaniterativeprocessofcontinuousdevelopmentis essentialforensuringcyberhygiene,preparednessandresilience.

6.4 WhatIstheWorstThatCanHappen?

Theextentofthepotentialimpactandconsequencesduetolimitedcybersecurityawarenessandtrainingcanonlybeestimatedonacase-by-casebasis.The majorconsiderationinthatrespecthastodowiththefactthatalackofrelevantcompetenceshasaknock-oneffectonthecybersecurityandresilienceof technologies,systems,processesandpolicies.Therefore,althoughtheimpact andconsequencesdependonthespecificsofanincident(e.g.sector,scope, objectives,attackercapabilities),limitationsincybersecuritycompetencesplay acriticalroleintheprobabilityofanincidentoccurring,andwillhaveanimpactontheeffectivenessoftheresponseandrecoveryactionstaken.Thus, itisnaturaltoconsidercybersecurityawarenessandtrainingasapositiveor negativemultiplieracrosstheoverallcyberhygiene.

Thebenefitsofcybersecurityawarenessandtrainingextendbeyondthe detectionandmitigationofcybersecurityissues[151].Tobegin,withskilled employeeswhoarefamiliarwithcybersecurityprinciplesandunderstand theirroleinkeepingthebusinesssecure,downtimeofcriticalbusinesssystemsduetosecuritybreachesorincidentscouldbeavoided.Thiswillsave organisationsfromthecostlyandtime-consumingprocessofrepairingand reinstatingnormalbusinessoperations.Next,employeeswhoarefamiliar

withcomplianceregulationsandhaveaclearunderstandingofhowtohandle sensitivedataandinformationcanhelptominimiseregulatorycomplianceinfractionsandtheirnegativereputationalandfinancialimpactonbusinesses. Finally,organisationsthatimplementproactivecybersecuritymeasuresand havedemonstratedcyberresilienceboostcustomerconfidence.

Letuslookattheimpactofdatabreachesinorganisationstohaveabetterideaoftheissuesthatcouldariseasaresultofalackofcybersecurity awarenessandtraining.Weconsidereddatabreachesastheexamplesimply because82%ofdatabreachesinvolvedahumanfactor[240].Thesebreaches occurredbecausepeoplefellvictimtosocialattacks,andeitherdeliberately (misuse)orinadvertently(errors)actedorfailedtoactwhennecessary.More importantly,theycouldhavebeenavoidedtoagreaterextentifthepeople involvedwereproperlyawareandtrainedinrelationtotheirsecurityoperationsandresponsibilities.Nowletusassesswhatcouldpotentiallyhappenif thereisadatabreachinsomeorganisation.Whenitcomestohospitaldata,a breachcouldjeopardiseandharmthepatient’shealthandsafety,i.e.endangerhumanlife.Inthecaseoffinancialservicedata,itsbreachcouldresult inahugefinancialloss.Andinthecaseofgovernmentdata,abreachcould compromisenationalsecurity.Lastbutnottheleast,irrespectiveoftheorganisationtype,adatabreachwouldcausealossofcustomers’andpartners’ trust,diminishedmarketreputation,lossinbusiness,andpenaltieslevied, whichmightleadtobankruptcy.

6.5 ResearchGaps

6.5.1

Cybersecurityawarenessandtrainingneedsacrosslevelsandfields ofstudy

Theongoingdigitalisationofproducts,services,supplyandvaluechains highlightstheneedfortheincreasedtechnicalliteracyofdigitalnatives.Therefore,inadditiontodedicatedstudyprogrammesforthedevelopmentofdedicatedprofessionalcompetences(e.g.computerscience,networkengineering), relevantmodulesareintegratedacrossmoststudyprogrammes,andlevels andfieldsofstudy[246].However,topicsrelatedtocybersecurityarescarcely introducedoutsideprogrammesthatareparticularlytargetedtowardsdevelopingcybersecurityprofessionals.

Accordingly,itisessentialtoidentifythecybersecurityskillsandcompetencesthatareneeded,aswellassuitabledeliverymethods,startingfromprimaryeducationallthewaytohighereducationandspecialisedfieldsofstudy. Thisrequiresexaminingtheuniversalcybersecurity-relatedcomponentsthat aretargetedatenhancingthecybersecurityawarenessofthebroaderpublic,

aswellasspecialisedtopicsthatarespecifictodistinctoccupations.Furthermore,itrequiresassessingdeliverymechanismsthatareadjustedand optimisedwithrespecttotheattributesoftherelevanttargetgroups.

6.5.2

Cybersecurityawarenessandtrainingneedsmultidisciplinaryapproachinvestigations

Itwasandisappropriateatthistimetoask,“Whyarecybersecurityawarenessandtrainingfailingtoyieldtheexpectedoutcomes?”[22]Thequestion hasbeenthesubjectofnumerousinvestigations,butnoclearanswerhasbeen foundyet.Thismaybearesultofthenarroworlimitedperspectivefrom whichweviewtheissue.

Cybersecurityawarenessandtrainingmostlyrevolvearoundcomprehendingandtransforminghumanthoughtandbehaviour,whichareundoubtedly complextopics.Therefore,aslongascybersecurityresearchersandprofessionalsattempttospecifyandcontrolhumanthinkingandbehaviourthrough asmallsetofdrivers,whichmostpsychologistsandsocialscientistswould considermisleading,thelikelihoodofsuccessfulcybersecurityawarenessand trainingwillprobablyremainlow[22].Thisalsoimpliesthataddressingthe issuewouldrequireamorecomprehensiveandholisticapproachthatutilises knowledgeandexpertisefrommultipledisciplines,includingengineering, pedagogy,behaviouraleconomics,marketing,andsocial,cognitiveandorganisationalpsychology,amongothers.

6.5.3

Computer-basedcybersecurityawarenessandtrainingneedtheimplementationofAIandMLalgorithmsfortheirautomationpurposes

Therearehardlyanydisciplinesthatarenotutilisingthecapabilitiesofartificialintelligence(AI)andML,oratleastattemptingtodoso.Cybersecurity awarenessandtrainingcannotbeanexception.Infact,therearenumerous waysthatAIandMLcouldbeusefultoraisethestandardandimpactof cybersecurityawarenessandtraining[207].

ByutilisingAIandMLalgorithms,manyactivitiesofcybersecurityawarenessandtrainingcouldbeautomated.Automationwouldhelptoachieve on-demandcybersecurityawarenessandtraining.Additionally,theycould facilitatethedesignanddeliveryofamorecustomised,personalisedandoptimisedawarenessandtrainingexperiencetotheaudience.Forexample,AI andML-assistedcomputer-basedtestscouldbedevelopedandusedtoidentifyvulnerablegroups.Furthermore,basedonthetestresults,andoncemore withtheapplicationofAIandMLalgorithms,morecustomised,personalised andoptimisedawarenessandtrainingresourcescouldbepreparedfortheaudience.

6.6 ExampleProblems

Tangibleexampleproblemsmightinclude:

IoTcybersecurityawarenessandtrainingmodules TheuseofInternetofThings (IoT)technologyisexpandingdailyinallspheresofbusinessandsociety,fromconsumer-focusedgoodsandservicestoindustrialIoT.This hasalsointroducedunprecedentedsafety,securityandprivacyrisks [23].ThemajorityofIoTsecuritydeploymentstakeplaceatthebusinessunitlevel,whereITdoesparticipate,albeitinsufficiently.Thisalso impliesthatanumberofkeystakeholdersinIoTsecurityareunfamiliar withtheITsecuritysideofthings.Furtherexacerbatingthesituation, IoT-relatedrisksareoftennotwellarticulated,resultinginlowawarenessamongusersandemployees.Thus,IoTsecuritycannotberobustif thepeopleinvolveddonothaveagoodunderstanding,andthisrequires themtohavetherelevantawarenessandtraining[134].

AwarenessofadversarialAIattacks ContrarytotheuseofAI/MLmethods tostrengthencybersecurity,threatactorsareleveragingAI/MLmethods formaliciouspurposes,forexample,toincreasethenumberofattack surfacesandbolstertheirattackingcapabilities[154].

AdversarialAImethodsareusedtocraftmisleadingdataorbehaviours withtheintentionofmanipulatinganddisruptingcriticalAIsystems. ThereisgrowingevidencethatadversarialAImethodshavebeenimplementedinreal-worldattacks.Inspiteofthis,theefforttodefend AIsystemsfromadversarialAIattacksisgenerallyanafterthought.It isunfortunatethatmanycompaniesstillremainunawareofadversarial AIattacksandthefailureofAIsystemstheattackscancause.Therefore,itisurgenttoraisecompanies’awarenessofadversarialAIattacks andmotivatethemtobealertandpreparedtodefendtheirAIsystems, especiallythoseusedincrucialsectors,againsttheattacks.

Cybersecurityawarenessandtrainingmodulesformobileusers Themobile phonehasgainedwidespreadacceptanceasacommonplacetoolforaccessingtheInternetanddoingsensitivejobs.Thesecouldbethecauses ofthedailyriseincyberattacksandcrimesaimedatmobilephone users[200][31].However,suitablecybersecurityawarenessandtraining formobilephoneusersarestillrare.Thereisacommonassumption thatmobilephoneuseissimilartousingadesktoporlaptop,which isonlypartiallycorrect.Indeedtheyshareacommonalityascomputingdevices;however,atthesametimetheyalsohavemanydifferences. Forexample,mobilephonespossessahigherriskfortheftorloss,authenticationusedtolockamobilephoneisoftenweakasaresultof

thehighfrequencyofloginstomobilephones,andthesmallerscreen sizeofmobilephonesoftenmakesitdifficulttonoticesecuritywarnings.Additionally,mobilephoneusersarefarmorediversethanthose oflaptopsordesktops.Peopleofvariousbackgrounds,fromurbanto rural,educatedtouneducated,white-collartoblue-collar,andsoon, usemobilephones.Therehavenotbeenmanyinvestigationsintowhy andhowthesediverseindividualsuseamobilephone,andwhattheir expectationsfromcybersecurityawarenessandtrainingmightbe.

Cybersecurityawarenessandtrainingevaluationfocusingonbehaviouralchange. Evaluationsofcybersecurityawarenessandtrainingarefrequentlyrestrictedtogaugingsecurityknowledgeandself-reportedattitudeshifts.

Indeed,improvementinknowledgeandattitudeisimportant,butthe evaluationshouldactuallymeasurethechangeincybersecuritybehaviour; afterall,behaviourchangeiswhattheawarenessandtrainingprogrammesareultimatelyaimingtoachieve[39].Studiesexaminingactual cybersecuritybehaviourareuncommon(moststudiesareoftenlimited toassessingintention),andthosethatdosoareoftenincomprehensible andincomplete.Regrettably,whilenumerouscomponentsofcybersecurityawarenessandtrainingarebeingdiscussed,thereisstillnoproper andreliablemethodtomeasurecybersecuritybehaviouralchange.

7 TrustedExecution

7.1 Introduction

Inthelasttwodecades,almosteveryaspect ofpeople’sdailylivesandallareasofhumanactivityhavebeenpervadedandrevolutionisedbydigitaltechnology.Sectorsvitaltosocietyandnations,suchastheeconomy,industry,culture,healthcare,socialand governmentactivities,nowadaysusemassiveamountsofsoftwaretodelivertheirservices,benefitingfromindisputableadvantagesintermsoftime,costandefficiency.

However,ITsystemsarevulnerabletoahugenumberofcyber-attacks,that areconstantlygrowinginbothnumberandseverity,thustrustedsoftware executionisthegoalthatindustryandacademiaarepursuingtoprotectIT systemsandtheirsensitivedatafromcybercrimeattacks.

Traditionally,hardwareisolationmechanismshavebeenintroducedtoprovidevariousprotectionmechanisms:virtualaddressspacesandmemorycontrolunitsprotectuserapplicationsfromeachother,privilegedinstructions protectsystemsoftwarefromuserapplications,andhardwarevirtualisation createsisolatedexecutionenvironmentsprotectedfromeachother.However, userapplicationsremainunprotectedbytheprivilegedsoftwareoftheoperatingsystemandhypervisor,consistingofmillionsoflinesofcodethathosta veryhighnumberofbugs[53, 88],exploitablebyattackerstogainprivileged accesstotheplatform[187].

Thisscenarioisfurthercomplicatedbytheadventofcloudcomputing, nowadaysincreasinglyusedbycompaniesduetoitsindisputableeconomic advantages.Inthiscase,theuserapplicationshavetotrustthehonestyofthe infrastructureprovider,theemployeeswithprivilegedaccountsorphysical accesstothecloudnodes,andtheothertenantsrunningtheirworkloadson thesameplatform.

TrustedExecutionEnvironments (TEEs)wereintroducedtoallowsecuritysensitiveuserapplications,orthemostcriticalportionsofthem,totrustonly

thehardwaresupportfortheTEEplusasoftwarelayerthatrunsinisolationandconstitutesthe TrustedComputingBase (TCB)fortheapplication.The smallertheTCBandthebetteritssecurity,becausethisreducestheattack surfaceandthelikleynumberofvulnerabilities.TEEsalsoprotectapplicationsfromphysicalattackers,forexamplethosethatcouldreadsensitivedata loadedinclearintotheRAMoftheplatform.Thisprotectionisachievedby meansofcryptographiclayersthatshielddatawhiletheyareprocessed.

Inthe2000stheTrustedComputingGroup(TCG)proposedthe Trusted PlatformModule (TPM)asasecureco-processortoperformparticularservices definedbytheTCG,mainlyaimedattheverificationoftheplatform’sintegritystatusandtheprotectionofprivatekeysfromunauthorisedaccess. However,theTPMisnotintendedtoexecutearbitraryapplicationsinitsisolatedenvironment,norcanitbeinstalledonanytypeofdevice.Tomeet theneedtoprotectarbitraryusercodeanddata,theindustryworldbeganto worktowardsthecreationofTEEssolutionsbasedonspecialsecuremodes ofthemainprocessor,thefirstofwhichwas TrustZone [17],proposedin2002 byARM,followedin2014byIntelwith SoftwareGuardExtensions (SGX)[119], andin2016byAMDwith SecureEncryptedVirtualization (SEV)[125].Atthe sametime,theacademyalsolookedforsuitablesoftwaresolutionstocreate TEEs,amongwhichwefind AEGIS [225],proposedin2003, Bastion [37]in 2010, Sanctum [54]in2016,and Keystone [138]in2020.

DespitetheimprovementsintroducedbyTEEsolutionstopursuetrusted softwareexecutionthroughsmallerTCBandstrongisolation,achievingsecuritydependsnotonlyontheTEEtechnologyadoptedbutalsoonthetrustworthinessoftheapplicationcodethatrunsinsideit.IdentifyingvulnerabilitiespresentinthecoderunningintheTEE,aswellasdetectingitscompromiseatrun-time,constitutechallengesthatcurrentstate-of-the-artTEEs donotaddressbutneedtobeconsideredbythescientificcommunityinnext years[166].

7.2

WhoIsGoingtoBeAffected?

AstheITsystemsarebecomingmorepervasive,distributed,andvitalinthe currentworld,thereisnosectorofoursocietythatcanlivewithouttrustin theexecutionofitssoftwarecomponentsandprotectionofthesensitivedata.

Ofcourse,thereisarelativescaleofimportance.Ifindividualsarenot offeredtrustedexecutionontheirpersonalsystems,thentherisksarelimited totheassetsofthatspecificindividual.Onanotherhand,iftheITsystemof acommercialcompanyoragovernmentbodydoesnotsupporttrustedexecution,thenthestakesaremuchhigher,dependingontheapplicationareaof theaffectedsystem.Inparticular,suchlargesystemsarethepreferredtargets forransomwareattacks(i.e.,amalwarethatencryptsdataandaskaransom

todecryptthem)andAPTinjection(AdvancedPersistentThreat,i.e.,apermanentmaliciousapplicationthatremainshiddentocontinuouslyexfiltrate informationorwaitingacriticaltimetoperformadestructiveattack).Recoveringfromransomwaremaytakeaverylongtime,fromdaystomonths(note thatpayingtheransomisnoguaranteetohaveallthedataback).APTare evenmoreinsidiousastheycangoundetectedforyears.

7.3 WhatIsExpectedtoHappen?

Ifsoftwarecomponentsareexecutedwithoutproperprotection,thentheresultsgeneratedcannotberelieduponforanypurpose.Henceanykindof damagecanbeexpected.

Ifusedinanindustrialcontrolsystem(ICS)thenproductioncanbeblocked orproductsmaybemanufacturedinthewrongway,eventuallyleadingtodefectsordamageinothersystemsusingtheseproductsascomponents.

Iftheattackedsoftwareelementisanapplicationhandling(directlyor indirectly)money(suchasanInternetbankingapporacompanypayment system)thenfinanciallosscanbeexpected.

Trustedexecutionisparticularlyimportantforcyber-physicalsystemsinteractingwithhumans.Forexample,thisisthecaseofrailwayorairtraffic controlsystemsorautonomousvehicles.Injectionofmalwareormodification oftheconfigurationofthesesystemsmayleadtophysicalharmtopersons, uptodeath.

Anotherpossiblescenarioconcernsthetheftofsensitiveuserdata,such asdigitalidentity,bankcredentials,orcommercialplans.Ifthisinformation isnotproperlyprotectedandusedwithintrustedexecutionenvironments,it isvulnerabletotheftbyanattacker,whocanuseittoimpersonateanother persontoobtainmoneyillegallythroughunauthorisedbankingtransactions, commitscams,discreditorputapersoninabadlightbycarryingoutillegitimateactionsonhisbehalf.

Inthefieldofcommercialespionage,companiescansufferconsiderable economicandimagedamageifattackersmanagetostealcustomerdataor confidentialinformation,relatedtoproductionprocessesornewprojectsupon whichthefuturedevelopmentofthecompanydepends,thusbringingillicit advantagetocompetingcompanies.

7.4

WhatIstheWorstThatCanHappen?

Theworstpossibleconsequencesdependontheapplicationcontrolledbythe systemtargetedbytheattacker.Therefore,itisobviousthatthemorecritical thesystemandtheworsttheeffectoftheattack.

Thestudy"CostofaDataBreachReport2022"[116]showsthatransomwareanddestructiveattacksrepresented28%ofbreachesagainstthe

criticalinfrastructuresexamined,highlightingthatattackersaimtointerrupt financialservicesandtodamageindustrial,transportationandhealth-care organisations.Thecriticalityoftheseinfrastructuresrequirestheadoption ofcutting-edgesecuritytechniques,suchasthecreationoftrustedexecution environmentsandthetimelydetectionofanytamperingwiththecodeand configurationofthesystems.

Forexample,intheeventthatagroupofattackerssucceedsinblockinga nation’selectricitygrid,millionsoffamilieswouldbeleftinthedark,companies’productionwouldbeblocked,communicationswouldbecut,banks wouldbeoffline,hospitalswouldnotbeabletoguaranteehealthcare,air andtraintrafficwouldstop.OnesuchattackhappenedinDecember2015in Ukraine,whenthreeutilitycompanieswereattackedsimultaneouslybythe BlackEnergymalware,leavinghundredsofthousandsofhomeswithoutelectricityforsixhours.AnotherattackofconsiderablegravityoccurredinIran in2010,whentheIraniannuclearprogramwasblockedduetosabotageof theNatanzenrichmentplantbymeansoftheStuxnetvirus,whichcausedthe destructionofthecentrifugesoftheplantwhilepreventingthedetectionof themalfunctioningofthesystemitself.Runningthecriticalapplicationthat supervisestheoperationofthecentrifugeswithinaTEEwouldhaveprotected itfromavirusthatinfectstheRichOS.

Thislastattackisaclearexampleofwhatistheworstscenario:theinjected maliciousapplicationdoesnotlimititselftoblockthenormalbehaviourofthe systembutcompletelysubvertsittoperformwrongoperationsthatwould directlydamagethesystemitselforpersonsthatuseit.

7.5 ResearchGaps

Overthepasttwodecades,alotofworkhasbeendonetobuildexecutionenvironmentsabletoguaranteeconfidentialityandintegritytoexecutionandto allowexternalentitiestoassessthetrustworthinesslevelofsystems.Nonetheless,theTEEsthemselvesposenewchallengesthatneedtobeaddressedby thescientificcommunity.

7.5.1

AttackvectorsagainstTEEsecurityguarantees

ATEEcanbeexposedtotypicalsoftwarevulnerabilities,withtheadditionof architecturalvulnerabilitiesnativetoaparticularTEEsolution.ATEEshould haveasmallTCBwithanarrowinterfacetominimisetheattacksurface.Over theyears,severalsoftwareandstructuralvulnerabilitieshavebeenfoundin specificTEEimplementations.However,moreexperiencedteamsaredevelopingsmallerandmoresecureTEEs,thankstothescrupulousadoptionof securesoftwaredevelopmentbestpracticesandrigorousvalidationofthe TEEdesignandcode.Thishascausedattackerstoshifttheirfocustomore

sophisticatedattacksattheedgebetweenhardwareandsoftware[197].An importantresearchareaforthenextyearswillconcernthestudyofmicroarchitecturalside-channelattacks,thatis,attacksthatexploitinformation leakagefromthehardwareinfrastructuretorevealsensitiveinformation,such asprivatekeys.

7.5.2 ProtectionmechanismsagainstcompromisedTEEapplications

TEEsrepresentavalidtechnologicalsolutiontoexecutesecuritysensitive workloadsinaprotectedenvironment.However,iftheapplicationcodedeployedwithinthemcontainsvulnerabilities,theycanbeexploitedbyanattackertocompromisethesecurityoftheentireTEE.Thisproblemisbecoming moreandmoreconcrete,anditssolutionmoreurgent,becausedevelopersbegintouseTEEstoruncomplexapplicationscontainingalargecodebase,thus increasingthelikelihoodthatexploitablebugsarepresentwithintheTEE.It hasalsobeenobservedthatthesecurityfeaturesoftheTEEsthemselvescan helpattackerstoinstallhigherlevelstealthyrootkitsthatareextremelydifficulttodetectthroughcurrentdefensemechanisms[166].Forexample,antivirustoolsrunningontheoperatingsystemcannotdetectmaliciouscode nestedinaTEEbecause,bydesign,theOScannotaccesstheTEEmemory, whichisoftenalsoencrypted.

Forwhathasbeensaid,thesecurityofaTEEcannotbegivenforgranted becauseit’sacomplexmatter,notguaranteedjustbyaperfectarchitectural designandimplementation.Thereforethecreationofsolutionsabletodetect bugsintheapplicationcodedevelopedforaTEEandmonitoritstrustworthinessatrun-timerepresentsanimportantresearcharea.

7.5.3 TEEsandcloudcomputing:interoperabilityandmanagementchallenges

SomeofthemajorcloudinfrastructureprovidershaveincludedTEEsintheir serviceoffering,sinceTEEsareabletoimprovethesecurityandprivacyguaranteesofcloudcomputing.However,twoconceptuallydifferentTEEsmodels canbeadoptedforcloudcomputing[94]:thevirtualmachine-basedmodel encryptstheentiresystemmemoryofavirtualmachine;theprocess-based modelselectivelyencryptsamemoryzoneofthedeployedapplication,delegatingtothedeveloperthedecisiontochoosewhichsectionofanapplication’scodetoprotect.Concreteimplementationsofthesemodelshavebeen developedfordifferentplatforms–SGXandthenew TrustedDomainExtension (TDX)forIntelplatforms,SEVandtheforthcoming SecureNestedPaging (SNP) forAMDplatforms,TrustZoneandRealmsforARM–andCPUarchitectures (x86,RISC-V,ARM).Thegreatvarietyofproposalsfieldedbyresearchand industrycausesinteroperabilityproblemsinmovingaservicefromonear-

chitecturetoanother,andcompatibilityproblemsindeployingapplications writtenfortraditionalsystemswithinTEEs.Animportantresearchareais thestudyanddevelopmentofframeworksthatofferalevelofabstraction capableofmakingtheheterogeneityofTEEsolutionstransparenttotheapplicationdeveloper,whilemaintainingthesamesecurityguaranteesoffered bytheunderlyingTEE.

AnotheraspectthatisgainingmoreandmoreimportanceisthedevelopmentofsolutionsthatallowtocombineTEEtechnologieswithcontainer technologies,inordertopromotetheuseofTEEsincloud-nativescenarios andfacilitatethedeploymentofTEEs-basedapplicationsinsidecontainers,at thesametimeofferingthesameuserexperienceasordinarycontainersanda smoothintegrationwiththeKubernetesecosystem.

7.5.4 TEEscryptographicprimitivesinthepost-quantumera

Inrecentyearswehavewitnessedremarkableadvancesinthefieldofquantumcomputers,whichallowustopushcomputationalcapabilitiesfarbeyondclassicalones.Thishasinevitablycausedimportantconsequencesin thefieldofcryptography,asquantumcomputersallowtheexecutionofalgorithmsthatofferquantumspeedtothesolutionofthemathematicalproblemsonwhichclassicalcryptosystemsarebased.Thisthreatwashighlighted withNIST’scall,in2016,topresentnewcryptographicalgorithmsresistant toquantumcomputerattacks.InJuly2022,NISTselectedthefirstfouralgorithmsthatwillbecomepartofNIST’spost-quantumcryptographicstandard[168]:CRYSTALS-Kyberforgeneralencryption,CRYSTALS-Dilithium, FALCONandSPHINCS+fordigitalsignatures.

TEEsbasetheirsecurityoncryptographicprimitivesimplementedinthe hardwarerootoftrustoftheplatform,currentlybasedonclassicalcryptosystems.Animportantresearchareaforthenextfewyearswillbethedesignand implementationofhardwarerootoftrustrelyingonpost-quantumcryptography,inordertowithstandquantumcomputationandquantumside-channel attacks.

7.6 Exampleproblems

Tangibleexampleproblemsmightincludethefollowingones:

DetectingacompromisedTEEapplication. AnapplicationrunninginaTEE couldbecompromisedbyanattackerduetothepresenceofvulnerabilitiesinitscode.Thestrongsecurityandisolationguaranteesoffered byTEEscanbeexploitedbyattackerstoimplementandinstallhardto-detectadvancedrootkitsinaplatform[166].Aimoftheresearch istodevelopsolutionsabletodetectcompromisedTEEapplicationsat run-time.

Technology-agnosticTEEsolutionsincloudcomputing. In2019,agroupof companies,includingIntel,Microsoft,GoogleandARM,foundedthe ConfidentialComputingConsortium (CCC)withtheaimofpromoting theadoptionofTEEsolutionsintheCloud.CCCsponsorsseveral open-sourceprojectsthatoffersolutionstothecompatibilityandinteroperabilityproblemsthatTEEtechnologiespose,suchasEnarx[70], Gramine[98],andOcclum[172].Theobjectiveofthisresearchistoanalysetheeffectivenessofcurrenttechnology-agnosticconfidentialcomputingsolutions,evaluatetheirperformance,studytheirpossiblesecurityshortcomings,applythemtothecloudcomputingdomain.

TEEapplicationsincloud-nativescenarios. Today,manyserviceproviders offertechnicalsolutionstofacilitatethedevelopmentandexecutionof TEEapplicationsinthecloud-Google’sAsylo[97]andAzure’sOpenEnclave[21]aretwoimportantexamplesofthem.However,whilethey simplifythedevelopmentofTEE-basedapplications,theystillrequire thedevelopertoacquirenewprogrammingskillsanddevelopthecode usingthecorrespondingSDKs.Furthermore,eventhoughthegoalof theseframeworksistosupportheterogeneousTEEsbyusingthesame API,theystillrelyprimarilyonIntelSGXtechnology.Theaimofthe researchistodesignanddevelopsolutionsthatallowuserstoruntheir servicesinside“TEE-basedcontainers”,withoutrequiringmodifications totheapplicationcode,whilesupportingheterogeneousTEEback-ends andprovidingeasyintegrationwiththeKubernetesorchestrator.

Trustedexecutioninlow-endIoTdevices. Nowadays,thesecurityofIoTdevicesisessentialastheyareincreasinglyusedinmultiplefields(e.g.vehicles,industry,smartcities,healthcare).However,IoTsystemspresent specialsecuritychallengesduetotheirheterogeneity,consideringnot onlytheembeddeddevicesbutalsothenetworks,themanagement anddataanalysisservices,andthestorage.Furthermore,whilehigh/middle-enddevicescanbenefitfromthesecurityguaranteesofferedby TEEs,low-enddevicestypicallydonothavehardwaresecuritymechanismstoprotectsecurity-sensitiveapplications.Aresearchareainthe IoTfieldconcernsthedesignofTEEarchitecturesthatmeetthechallengesposedbylow-costandlow-powerdevices,toensurethetrustworthinessofawiderrangeofIoTapplicationsandthedatatheyproduce. ThisshouldgoalongwiththedevelopmentofsolutionsforasecureremoteandautomatedmanagementoftheIoTdevices,ofteninstalledin uncontrolledenvironments.

TEE’ssecurityfunctionsintegrationinthenetwork. Goaloftheresearchis theintegrationofTEEstechnologieswithinthecommonnetworkop-

erationalmechanismsandtheenhancementoftheirsecuritythanksto TEE’shardwareandsoftwareguarantees.Forexample,animportant aspectconcernsthecreationofmutuallytrustedchannelsbetweenTEEbasedapplications,extendingtheTLSprotocolwithmechanismsthat allowtheverificationoftheintegrityandauthenticityoftheend-points ofthecommunicationchannel,portableforheterogeneousTEEs.

Quantum-resistantrootsoftrustforTEEs. OpenTitan[177]isanopen-source frameworkthatsupportsthedesignandintegrationofvendor-and platform-agnosticsiliconrootsoftrusttointegrateintoservers,storagedevices,peripheralsorothertypesofplatforms.Thegoalofthis researchistorealiseanOpenTitanextensioncapableofusingpostquantumcryptographyinsilicondesignandfirmwareimplementation ofarootoftrust,inordertosupportquantum-resistantTEEs.

Runtimedetectionofmanipulationofsystemconditions

Alteringthecorrect configurationinwhichachiphastooperatecanleadtounexpected softwarebehaviourorchangesintheexecutionflowofthecode;this istypicallyaccomplishedbyphysicallymodifyingthepowerofthedevice,theclock,theelectromagneticfieldorthephysicalinterfaces[197]. Theaimofthisresearchisthecreationofruntimemechanismscapable ofdynamicallysendingalertswhenachangeinsystemconditionsis detected.

8

PrivacybyDesign

8.1 Introduction

Inaworldthatisincreasingly digital,vastamountsofpersonal dataarecollectedandprocessed, oftenubiquitouslyandintransparently,andusedbygovernments and/orcommercialisedamongseveralserviceproviders,databrokers,andadvertisers.Thiscommoditisationofpersonaldatahas furthererodedindividuals’rights toprivacy.Formanydecades, researchershavelookedintothis growingOrwelliantrendofprofilingandsurveillance,attemptingto findabalancebetweentheadvances intechnologyandtheprotectionof privacy.Aimingattheverycoreof thesystems’design,AnnCavoukian coinedthetermPrivacybyDesign (PbD)backinthe’90s,proposinga seriesofsevenfoundationalprinciples,instillingprivacyassuranceasanorganisation’sdefaultmodeofoperation[35](seeFigure 8.1).Behindthese principlesisalsotheobservationthatprivacyisbestachievedwhenaddressed attheearlieststagesoftechnologydevelopment,i.e.intheconceptualdesign phase.

Figure8.1:Cavoukian’s7FoundationalPbDPrinciples[35]

Althoughacclaimedbymanyresearchersandpolicymakers,PbDisoftencriticisedasbeingtoovagueandhardtotranslateintoconcretesoftware engineeringpractices[239].Infact,today,thereisstillasignificantgapbetweenresearchandpractice,e.g.,translatinghigh-levelPbDprinciplesinto concreteengineeringpracticesthatsoftwarepractitionerscaneffectivelyand efficientlyadopt.Aimingtoclosethisgap,theemergingdisciplineofPrivacy

Engineeringhasbeenformedthatfocusesondesigning,implementing,adapting,andevaluatingtheories,methods,techniques,andtoolstosystematically captureandaddressprivacyissuesinthedevelopmentofsocio-technicalsystems[102].Therefore,furtherdevelopingthisareaofPrivacyEngineeringis asignificantchallengeforresearchersaswellasfororganisationsthatwant tointegrateandoperationalisePbD.Fororganisations,thischallengeisalsoa matterofregulatorycompliancenowthatthenotionsofdata-protection-bydesignanddata-protection-by-defaulthavebeenincorporatedaspartofthe EuropeanGeneralDataProtectionRegulation(GDPR),inforcesince2018.

8.2 WhoIsGoingtoBeAffected?

Asmentioned,theGDPRhasraisedthebarforprivacy,includingPbDas partofArticle25“Dataprotectionbydesignandbydefault”forprotecting thefundamentalrightsofindividualsinEurope.Thisregulationaffectsall organisationsthatcollectandprocesspersonaldataofEUcitizensandresidents,meaningthatitcanapplyevenifanorganisationisbasedoutsideEU. However,thelegislationleavesitopentowhichexacttechnicalandorganisationalprotectivemeasuresaretobetakentofulfiltherequirementsofPbD. This,ofcourse,createsfurtherchallengestoorganisations,butmorespecifically,tosoftwarearchitectsanddevelopers,whoareultimatelyresponsible fordesigningthesystems.

Besidesthat,largetechnologyorganisationshavestartedhiringprivacy engineersandestablishingprivacyredteamoperations,whichhelptoembedprivacyinthesystem’sdesignandproactivelytestprocessesandsystemstoidentifyprivacyrisks.However,notmanyorganisationshavethe resourcestohireprivacyengineers,letalonetomaintainanentireprivacy engineeringdepartment.Fromwhatwesee,thisisespeciallythecasefor smallandmedium-sizedenterpriseswhichcomprisethemajorityoforganisationsnowadays.EventhoughthepracticaleffectsofGDPRarebeneficial toindividualsandsocietyasawhole,theyposesignificantchallengesfororganisations,andinturn,totheresearchcommunitythataimstomakePbDa realityinanever-changingtechnologylandscape.

8.3

WhatIsExpectedtoHappen?

Inourdailyliveswearesurroundedbytechnology,withawiderangeof data-intensivesoftwaresystemsbeingusedforpersonalandprofessionalactivities.FailingtoaccomplishPbDintoday’sworkingsystemscanseverely jeopardiseindividualsandthedemocraticsocietyasawhole[205].Thelack ofprivacyhasnegativeimpactstoindividualsrangingfromembarrassment andreputationdamagetovariousformsofdiscriminationthatadverselyaffectindividuals’rightsandfreedomsandphysicalandmentalhealth.Ona

societallevel,privacyisalsoconsideredasanessentialcomponentforafunctioningdemocraticsociety[26, 69].Ifpeoplecannotfullyexercisetheirrights andfreedoms,suchasfreedomofassociation(e.g.,politicalandreligious) andfreedomofopinionandexpression(includingholdingbackone’sviews), therearenegativeimpactsontheindividualsdemocraticparticipation,also harmingtheirhumandignityandpersonalautonomy.

Forsuchreasons,itisexpectedthatorganisationswillresponsiblycreate andadaptsoftwaresystemsfollowingPbDprinciples,adheringtoprivacy rightsasenshrinedintoday’sregulations.Asaresult,peoplewouldbeable totakeprivacyforgranted,withthefullexpectationthatanydatathatiscollectedandprocessedhasbeenlawfullyacquired,thatthespecificpurposesfor processingaretransparentlycommunicatedandadheredto,andthatwheneverpossibleusersareabletoexercisevariousrightsovertheirdata,e.g., access,correction,deletion,objectprocessing,etc.

8.4 WhatIstheWorstThatCanHappen?

TheGDPRissometimesreferredasaregulationthathas“realteeth”,applyingmassivefinestoorganisationsthatviolateprivacyrights.Non-compliance withtheGDPRcanleadtofinesofupto20millioneurosorupto4%ofan organisation’sworldwideannualturnover,whicheverishigher.Studiesare showinganincreasingnumberoffinesbasedontheGDPRsinceitspublicationin2018[190,250],withthelargestfinesofarof746millioneurosimposed byLuxembourg’sDataProtectionAuthorityagainstthetechgiantAmazonin July2021[141].

However,arguably,thecollectivesocietalcostsofprivacyviolationscan bemuchhigherthananylegalfines.Asmentioned,thedeteriorationofindividualprivacyrightsincursintheweakeningofdemocracyinitself.Ifleft unchecked,organisationscanexploitadvancedtechnologiessuchasartificial intelligencetocarefullycraftandtargetadvertisements,generatingascenario ofsocialmanipulation[147].EvidencesforsuchexploitativeuseofAI-based profilingofusershavebeenseeninsocialmediamanipulation,spreading fakenewsandmisinformation,andtargetingvoterswiththeintenttotiltthe resultsofelections–theCambridgeAnalyticascandalprobablyasthemost widelyknowncase,butthereisalsoevidenceofinterferenceintheBrexit referendumandelectionsinBrazil,Sweden,andIndia[15].

8.5 ResearchGaps

8.5.1

PrivacyGoalsvs.OtherGoals

Solvingtrade-offsthatneedtobemadebetweenprivacyprotectionandother goalsconstitutesamajorchallenge,asillustratedin[104].Also,ourinter-

viewsconductedwithintherequirementelicitationphaseofCyberSec4Europe [82]conveyedthatforthedomainofprivacy-enhancingidentitymanagement systems,researchandpracticalchallengesforadequatelyaddressingtradeoffsthatneedtobemadebetweenprivacyprotection,usabilityandtrustneed tobeaddressed.Preferencesforprivacytrade-offpreferencesalsodiffersculturally,whichalsoneedstobeconsideredforachievingusableprivacyand identitymanagementsolutionsbydesign[121].

8.5.2 BuildingtheTheoryofOrganisationalPrivacyCultureandClimate

Asadvocatedby[24],organisationscanbeseenaslivinghumanentities,and asahumangroupstructure,theyhaveaculture.Thisgroupcultureisareflectionoftheconsciousnessofitsleaders.Therefore,thevaluesandbehaviours oftheleaderswilllargelyinfluencethecultureoftheentireorganisation.If acultureis“toxic”inoneormoreofitsfacets,itisimportanttolookclosely atthevaluesandbehavioursdisplayedbyleadersandtopmanagement.And thisincludesthefacetofprivacyandhowitisperceivedinsidetheorganisation.

Manyresearchershaveaddressedthetopicsoforganisationalprivacyculture[57]andclimate[16, 103],showingthatthesecomponentsstronglyinfluencetheemployees’perceptions,attitudes,andbehavioursconcerningprivacy.SuchresearchemphasisesthatleadersmustcreateaconduciveenvironmenttointegratePbDsuccessfullyintotheorganisationalprocesses.However,sincethisareaofOrganisationalPrivacyCultureandClimate(OPCC)is initsembryonicstage[122],therestillneedstobemoreprimaryresearchto solidlybuildthetheoryaroundthetopic,aswellastodefinewaystomeasure andembedprivacyinorganisationsreliably.

8.5.3

CounteringDeviceFingerprinting

Devicefingerprintingattacks,whichcanrecalladevicebycoincidentaldata thatthedeviceleaveswhilecommunicatinginanetwork,becomeserious threatforlocationprivacy.Networkdevicesbecomeincreasinglyheterogeneous,whichenablesadiversityoffingerprintsthatcanbeexploitedforattacks.Furtherresearchisneededforunderstandingandmeasuringtheaccuracyoffingerprintingattacks,e.g.,bymeasuringhowmuchentropyiscontainedinaspecificfingerprintingsourceforprovidingguidanceonachieving dataminimisationinaPbDprocess.

8.5.4

DataSubjectRightsEngineering

AccordingtoArt.15-21GDPR,Europeancitizenswhosepersonaldataisprocessedatanyorganisationgloballyhaveasetofrightstowardsthesedata processingorganisations.Forinstance,therightofaccessallowsthemto

beinformedaboutthenatureandpurposeofprocessing,aswellasabout thesetofdatastoredandprocessed.Tosomedegree,thisevenspreadsto sub-processorsthatareinvolvedinthedataprocessingaswell.Therightto erasure,oftenalsodubbedtherighttobeforgotten,allowsfordemanding deletionofall(orpartofthe)personaldatastoredatanorganisation–unlessotherexplicitreasoningrestrictsthis(e.g.,concerningpersonalrecords atlawenforcementagencies).Therighttorectificationenablestheindividualsconcernedtochangetheirdata,e.g.,tocorrectfalseinformationinadata record.

However,whenitcomestoenforcementofthesedatasubjectrightsgranted bytheGDPR,alotofopenissuesarise.Howcanthetransparencydemanded bytherightofaccessberealisedinamulti-organisational,distributedworkflow?HowcanrestrictionstoprocessingaccordingtoArt.18GDPRbe implementedintosuchaworkflow?Howcanarequestforerasureorrectificationbepropagatedthroughoutaprocessingchain,andwhichpartofthe processingconstitutethesameworkflowwithrespecttothespecificpurpose ofprocessing?Whendotwoprocessingactivitiesbelongtothesameworkflow,andwhendotheyinstantiateaseparatedataprocessinginstance,with separateneedsforuserconsentanddatasubjectrightsenforcement?

8.6

ExampleProblems

Whenaddressingtheseandotheropenresearchchallengesinthedomain ofPrivacybyDesignorPrivacyEngineering,thefollowingspecificproblem domainsneedtobeaddressedmoreclosely.

IdentifyingfactorsanddefiningconstructsintheOPCCarea. Organisations cangreatlybenefitfrompracticalinstruments,suchasquestionnaires [57],thatcouldhelpthemtomeasureorassessorganisationalaspects suchas“privacyculture”and“privacyclimate”.Todoso,researchers stillneedtounderstandandidentifythekeyfactorsthatformOPCC constructs,andtestinstrumentsintermsofvalidityandreliability.

Algorithmicfairnessvs.dataminimisation. Trade-offsbetweendataminimisationandfairnessformachinelearningmodelswasrecentlydiscussed [38]andisstilltoalargeextendanunsolvedissue.

Meteringrisksindevicefingerprinting. Furtherresearchisneededondefencemechanismsfordevicefingerprintingrisksthatcanbeavoided (e.g.basedonsoftware-definedbehavioursuchasAPIs)andonremainingrisksthatwillbehardtodefend(e.g.,fingerprintingattacksbased onphysicaldeviceproperties,suchasdriftofphysicalclocks).

Measuringthelevelofprivacyprotectionoffered. Itisoftenunclearwhat levelofprotectionisactuallyprovidedbyacertainprivacy-enhancing

technologyofprivacy-awaredesigndecision.Alongwiththisuncertaintycomesthequestionwhetheragivensetofprivacy-enhancing measureswassufficienttobeconsideredstateoftheartinthesense ofGDPR,orwhatotherlevelsofprotectionwouldhavebeenadequate. Hence,theselectionofPrivacy-EnhancingTechnologies(PETs)toapplyinagivencontextandscenario,alongwiththedeterminationof theprotectionachieved,isanopenresearchquestion.Earlymethodologiesexist(likethePrivacyDesignStrategies[111],LINDDUN[60],or theStandardDataProtectionModel[51]),butthesearenotsufficientin detailingthemeteringofthelevelofprotectionprovided.

Threatmodellingas“bydesign”enabler. Privacyneedstobecomeintegrated intoallstepsofsoftwaredevelopmentbydesign.AccordingtoGDPR’s risk-basedapproach,itiscrucialtofirstdeterminetheprivacyproblemsthatcanariseinordertoproperlyresolvethem.Threatmodelling isawell-knownapproachinthesecuritydomain1 andhasbeengainingmomentumintheprivacycommunityaswell.Privacythreatmodellingallowstosystematicallyidentifyandmitigateprivacyissuesat thearchitecturallevel.Byidentifyingtheseproblemsearly,theycanbe tackledatthesystem’scoreinamoreefficientway.Thethreatmodel shouldinformdecisionsinsubsequentdesign,development,testing, andpost-deploymentphases2 (e.g.determinethekeyverificationtargetsforsoftwaretesting).Riskassessmentshouldguideprioritisation. Threatmodellingautomationisthenextsteptofacilitateagrowing adoption.Developmentsinrun-timeandadaptivethreatmodellingwill alsostrengthentheincorporationinContinuousIntegrationandContinuousDelivery(CI/CD)settings.

Datacustodiansanddelegateddatasubjectrights. Utilisingdatasubjectrights againstadatacontrollerrequiresaspecifictypeofinteractionaccordingtotherulesoutlinedintheGDPR.DataControllershavetoprovide communicationmeansforsuchrequestsaccordingly,andmayhavea largeincentivetoautomateoratleaststructuresuchrequestsasfaras possible,e.g.tosaveworkforce.Atthesametime,theburdenofutilisingone’sdatasubjectsrightsovertimecaneasilybecomecumbersome, e.g.ifrequestsforerasureneedtobedonerepetitivelyduetodata collectionprocessesnotproperlycontrolledbythedatacontroller,orif rightofaccessrequestsmustbepreprocessedtobecomeunderstandable tohumanreaders.Insuchcases,theinstantiationofadedicateddata custodianthatenforcesdatasubjectrightsonbehalfofthedatasubject

1seee.g.OWASP’stop10: https://owasp.org/Top10/ 2seealso www.threatmodelingmanifesto.org

maybecomeessential.AsforeseenintheEuropeanDataGovernance Act,dataintermediariesthatenforcedatasubjectrightsmustfulfilspecialrequirements,andmaybetemptedtoautomatetheiroperationsas faraspossibleaswell.Here,openresearchchallengescanbeidentified intheseaspects,suchasdatasubjectrightsengineering,transparency bydesign,rightofaccessasaservice,ordatacustodiantrustdelegation models.

9

CriticalInfrastructures

9.1 Introduction

Althoughtheprotectionofcriticalinfrastructures(CIs)hasreceivedtheattentionoftheresearchcommunityformorethanadecade,securingCIsfrom emergingcyberandhybrid(cyber-physical)attacksisstillanopenchallenge. VariousdefinitionsofCIscanbefoundinthescientificliterature,internationalstandardsandregulatorydocuments.Insimpleterms,andinlinewith therelevantEuropeanCouncilDirective[64],CIsarelarge-scalesystemsor systems-of-systems,thatareessentialfortheproperoperationofvitalsocietal functionsandforpeople’swell-being.

Takeforexamplethe healthcare sector:thissector iscomprised,amongother things,ofhospitals,healthcarecentres,pharmaceutical labs,bloodsupplyfacilities, emergencyservicesandresearchfacilities.Thedisruptionordestructionofsuch facilities,especiallyifextensiveorforasignificantduration,mayhaveasevereimpactonpublichealth.Asanotherexample,considerthe transport sector:inthiscaseairports,ports,railwayinfrastructuresandroadtrafficcontrolsystemsplayasignificantrole inpeople’smobility,aswellasintheproperoperationofthesupplychain. OtherexamplesofCIsectorsinclude informationandcommunicationtechnology infrastructures,suchastelecommunicationnetworksandcloudinfrastructures; energyinstallations includingelectrical,gas,oilornuclearpowerproduction,storage,transmissionanddistributionnetworks; waterfacilities,including dams,waterstorage,managementandnetworks; finance,suchasbankingfacilitiesandinter-bankingcommunications; foodmanagement,includingfood production,foodsafetysystems,wholesalesupplychain,andmanymore.

Onemightarguethat,“sinceCIshavebeenarearoundforseveraldecades (orevencenturies),theymustalreadybematureenoughandadequatelyprotected".Unfortunatelythisisfarfrombeingtrue,forseveralreasons.Thefirst reasonistheincreasedaccessibilityofmodernCIsandtheirincreasedcouplingwithinformationandcommunicationsystems.Afewdecadesago,CIs usedtobeclosedsystems.Nowadays,InternetconnectivityoffersCIadministratorsmoreefficient,real-timeandremotemanagement,withoutrequiring physicalproximitytotheinfrastructure.Ontopofthat,CIshavealsobecome moreaccessibletoendusersandcloselyconnectedwithInternet-of-things (IoT)systems.Forexample,whilesomeyearsagomeasurementconsumption intheelectricgridrequiredphysicalaccesstotheend-usermeteringsystems, nowadayssmartmeteringsystemsallownotonlyremotemeasurement,but alsoremotecontrol.WhilechangesliketheonesdescribedabovehaveincreasedtheefficiencyofCIoperations,atthesametimetheyhaveincreased theirattacksurfaceandhaveenabledtheirexposuretoremotecyber-attacks.

Lastbutnotleast,theincreasedconnectivityofCIshasalsoincreasedthe dependencies betweenthoseinfrastructures.Differenttypesofinfrastructure dependenciesexist.Forexample,anenergyproviderwhoreceivescommunicationservicesfromatelecommunicationoperatorhasa cyberdependency.On theotherhand,thetelecomoperatorwillrequireelectricalpowertosupport itsnetworkoperations.Anydependencyonaphysicalresource,suchason theenergysupplyasdescribedabove,isa physicaldependency.Othertypes ofdependenciesinclude geographical (whentwoCIsdependoneachother becauseoftheirphysicallocation)and logical (whensomekindofdependency otherthanthoseabovecanbeidentified).

9.2 WhoIsGoingtoBeAffected?

Anyonewhoactsasa“consumer"oftheservicesprovidedbyaCIwillbe affectedifaCIiscompromised,includingpeople,companiesandotherorganisations.Unfortunately,thedependenciesbetweenCIprovidersincrease thesignificanceofpotentialattacks,aswellastheextentoftheorganisationsandpeoplethatwilleventuallybeaffected.Considerforexamplea cyber-attackagainstanelectricaldistributionnetwork,whichsupportsmany othernearbyCIs(geographicalandphysicaldependencies),suchastelecom providers,traffic-lightsystems,governmentservices,datafacilities,hospitals, datacentresorairports.Thedisruptionorthedegradationoftheelectrical supplywillconcurrentlyaffecttosomeextent,alltheCIsthatdependonthe specificelectricaldistributionnetworkunderattack.Suchtypesofconcurrent dependenciesofmultipleinfrastructuresonasingleCImayresultinwhatis knownas common-causefailures.Common-causefailureswillobviouslyaffect

multipleorganisations,inboththepublicandintheprivatesector,aswellas manypeoplewho“consume"theaffectedservices.

Anothertypeofdependenciesthatmayconcurrentlyaffectaconsiderable numberofpeople,companiesandorganisationsarethosedependenciesthat cascadefromoneCItoanother.Oneofthemostfamousandwell-studied casesistheCaliforniablack-out[196],wherethefailureofapowerstation causedmultiple cascadingfailures,duetoaseriesofCIdependencies.For example,theenergyreductioncausedadecreaseintheamountofpetroleum thatwaschannelledtotheairportfacilities,thereforecausingsevereproblems totheairportservicesandultimatelytotheflightschedulesoftheairport operators.Atthesametime,thelossofelectricalpowerledtothedegradation ofthesteaminjectionunitsthatwereusedtopoweroilrecoveryunits.The latterledtoafeedbackeffect,sincetheproducedoilwasalsousedasafuelby theelectricalpoweroperatorthatwasinitiallyaffected!Finallytheelectrical powerreductionalsoaffectedthewaterpumpsthatwereusedincropfields.

9.3

WhatIsExpectedtoHappen?

AttacksagainstCIsmayleadtoallkindsofconsequences,suchaslossof life,financialloss,publicdisorderordisruptionofbusinessoperations[222]. Attacksagainsthospitalsmayaffectpatienttreatment.Forexample,aransomwareattackinaGermanhospitalcausedadelayonapatient’semergency treatment,whoeventuallylostherlife1.Althoughtherelevantpoliceinvestigationconcludedthat“thedelaywasofnorelevancetothefinaloutcome"2 , italsowarnedthatit’samatteroftimebeforehackinghospitalsleadsto tragicresults.Attacksagainstenergyinfrastructurescandirectlyleadtoloss ofproductivity,andtosevereeconomicloss,especiallyifcascadingeffectson otherinfrastructuresarealsoconsidered.Examplesofcyberattacksofthis kind,allegedlycausedbynation-stateadversaries,includetheattacksagainst Ukraine’selectricalgridin2015andin20163.Telecommunicationinfrastructuresarealsoveryattractiveattacktargets,sincetheyandenergyarethetwo sectorswiththehighestlevelofincomingdependenciesfromotherinfrastructures(almostanyCIdependsonenergyandtelecommunicationservices). Attacksagainstroadtrafficmanagementinfrastructuresordirectattackson

1Theuntoldstoryofacyberattack,ahospitalandadyingwoman: https://www.wired.co. uk/article/ransomware-hospital-death-germany

2RansomwareattackinGermanhospital.Areportontheinvestigationfindingsandwarningscanbefoundin: https://www.technologyreview.com/2020/11/12/1012015/ransomwaredid-not-kill-a-german-hospital-patient/

3HackerstriggeryetanotherpoweroutageinUkraine: https://arstechnica.com/ information-technology/2017/01/the-new-normal-yet-another-hacker-caused-poweroutage-hits-ukraine/

vehiclesofanykind(cars,shipsorplanes)mayleadtodisruptionoftrafficor eventolethalaccidents.

9.4 WhatIstheWorstThatCanHappen?

AsCIsarevitalforpeople’swell-beingandtheirdisruptioncouldleadto severesocietal,financialandsafetyconsequences,theyareveryattractivetargetsformaliciousattackers.

ThepreparationofcyberattacksagainstCIsrequireshighmotivation,usuallyhighresourcesandskillsandsomekindofcapabilityforinitialaccessby theadversaries.Unfortunately,varioussuchadversariesexistinthecurrent threatlandscape.Forexample,nationstateadversariesmaybesufficiently motivatedandmayhavetherequiredtime,resourcesandskillstodeploy AdvancedPersistentThreats (APT)againsttargetedCIs,withtheintentionof causingseveredamagetotheCIsofanenemystate.Thismaybeusedas anasymmetricattackoraspartofahybridcyber-physicalwar.Terrorists mayalsobemotivatedtocauseseveredisorderandlossofpublicconfidence. Finally,cybercriminalsmaybemotivatedtoattackCIs,aimingtoahigheconomicgain,e.g.throughransomwareandblackmailingattacks.

Theincreasednetworkconnectivityandtheinter-connectivityofCIsis increasingtheattacksurface,asitmayprovideadversarieswithseveralinitial pointsofentry.Inaddition,thelackofsecuritytrainingandawarenessmay alsobeexploitedbyadversariestogaininitialaccess.Forexample,spear phishingcampaignsaimedattargeteduserscanbeapreparatoryactionfor anAPT.

9.5 ResearchGaps

ObviouslytheprotectionofCIsfromattackssuchasthosedescribedabove isnotatrivialtask.CIprotectionisamulti-disciplinaryprocess.Froma technicalpointofview,itrequiresabetterunderstandingofthethreats,vulnerabilitiesandexposures,aswellasanefficientandeffectiveprotection. Fromasocialandbusinessperspective,itrequiresabetterunderstandingof thedependenciesbetweenCIsandtheimpactrelatedtothespecificattacks, aswellasincreasedtrainingandawarenessofthepeopleinvolved.

9.5.1

APTsareexpectedtobecomemorepowerful,evenmoresophisticatedand morefrequent.Thesameistrueforothertypesofman-madehybrid(cyberphysical)attacksorevenfornaturalandclimate-changedisasters.Becauseof theircomplexity[209],CIsarenotcurrentlyequippedwithadvancedmodellingtoolsthatwillallowthemtoadequatelypreparethemselvesforsuch

non-trivialthreatsandtoeffectivelymanageanylikelyattack[242].Thereisa needfornovelapproachestosupportthemodelling,analysisandsimulation ofsuchthreats,e.g.[254],inordertobetterprepareCIstodealwiththemin therealworld,butalsotoproposefast,efficientandreliableresponsetactics.

9.5.2

Developriskassessmentandmanagementmethodologiesforsystemicandsupply-chainrisks

Aswashighlightedveryclearlythroughtheeventstriggeredbythewarin Ukraine,severalCIsectors,suchasenergyandtransport,maytriggersystemic,cross-sectorandcross-borderrisksforsocietyandtheeconomy(e.g.by disruptingenergyorfoodsufficiencyonaEuropeanorglobalscale).Given theirspecialcharacteristics[256],thereisaneedtodevelopnovelmethodsfor theearlyidentificationandproactivemanagementofsuchrisks,especiallyin across-sectorandcross-bordercontext.

9.5.3 ResilienceofCriticalInfrastructures

IncreasingtheresilienceofCIsisanongoingresearchgoal,atbotha“microscopic"andata“macroscopic"level[193].Fromacomponent-wisepoint ofview,thereisaneedforadditionalresearchinto resilientandfault-tolerant embeddedsystems,whichareessentialforthepropermonitoringandcontrol ofcriticalcyber-physicalsystems.Fromasystem-wiseview,assuringalevel of resilienceforcriticaloperations andservicesinacost-efficientwayisanopen challenge.Thereisaneedfor(re)designingresilient-by-designinfrastructures,bydevelopingnearlycost-optimalsolutionsthatassurethecontrolled redundancy,resourcefulnessandquickrecoveryofcriticaloperations[210]. Aresilientdesignshouldalsoconsidertheintegrationofcost-efficient(semi)automatedresponsecapabilitiestoeffectivelyminimisetheimpactofcyber attacksattheearliestpossiblestage.

9.5.4

ImprovedAI/MLassistedmodelsfor(inter)dependencyanalysis

Despitepastandrecentresearchefforts(e.g.[204, 208, 224, 243]),thereisstill aneedforimprovedmodelsfortheanalysisofCI(inter)dependencies,exploitingreal-timethreatandriskmonitoringsystemsassistedbyartificial intelligence(AI)andmachinelearning(ML).Forexample,thereisaneed todevelopmodelsforunderstandingtheperturbationfluxfromonesystem toanother,whichencompassnon-localeffectswithlargedifferenceintime scales.Asmodeltrainingshouldbebasedonrealdatafromactualsystems, thisrequiresamoredirectinvolvementoftheinfrastructureoperators.

9.5.5

Eventpredictionbasedonalltypesofdependencies

Althoughcyberandphysicaldependenciesaregenerallycapturedindepth incurrenteventpredictionmodels,geographicaldependenciesarenotadequatelycaptured.Becauseofthis,predictionofdisruptiveeventsisnotaccuratelymappedonaspecificterritory.Effortsshouldbemadeinthatdirection, forexampleusinggeographicinformationsystemstocapturethemaximum possiblelevelofspatialresolutionandtomapthisinformationontodata concerningdislocationofassets,alsoconsideringthemostimportantperturbationtypesforeachasset(e.g.ground-shaking,rain,wind,ortemperature).

9.5.6

CollaborativesituationalawarenessfortheCIecosystem

Asacontinuouslyincreasingamountofcybersecuritydata(e.g.emerging threats,zero-dayvulnerabilities)becomesavailableonadailybasis,itbecomesmoredifficulttoeffectivelyutilisethatdatatoimprovethecybersecuritysituationinanorganisation[11].Improvingsituationalawarenessfor CIoperatorsrequiresamulti-disciplinarysocio-technicalapproach,whichincludespeopleaspartofthesolution.Methodsandtoolsareneededtofacilitatecooperationandcollaboration within and between theCIoperatorsand therelevantsectoral,nationalandEuropeanauthorities.

9.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

DigitaltwinsofCIs. Develop4-dimensionalmodelstocontinuouslymonitorthebehaviouroftheinfrastructure,byconstantlyreceivinginput fromIoTdevices.Thismaybeapriorityforinfrastructuressuffering fromageingproblems,inordertocontinuouslyanalysetheirexpected stateandtoperceivepossibledeviationsfromtheirnormalstructural behaviour.

SupplychainsecurityforCIs. CIoperatorsdependonvarioussupplychains todelivertheirservicestoend-users.Attackersareincreasinglyusing thewholesupply-chaintoperformattacks.Assupplychainsecurityis alsomandatedbytheNIS/NIS2,supplychainsecurityforCIsrequires furtherinvestigation.

Developtoolstosupporteffectiverecoveryfromcriticalall-hazardevents. Theincreasednumberofinterdependenciesrequiresthedevelopmentof innovativedecisionsupporttoolstoassistintheearlyidentificationof criticaleventsandinthemosteffectiverecoverystrategy,takinginto considerationtheirdependenciesandotherconstraints.

Developmethodsandtoolsfortheearlydetectionofcyber-physicalevents. SinceCIsarecyber-physicalsystemswithstronglytightdimensions, thereisaneedtodevelopmethodsandtoolstodetectandassesscyberphysicalattacks,byconcurrentlyconsideringboththeircyberandphysicalvulnerabilitiesfromaholisticperspective.

Designimprovedmulti-disciplinaryregulatoryframework. Competentauthorities,asdefinedinNIS/NIS2,areplayingakeyroleinCIprotection.Itisnecessarytodevelopsustainableandcollaborativeregulatory frameworksthatcanconsiderallpertinentrisksandhandleincidents involvingvariousdimensionssuchastechnical,societalorlegalaspects.

10 Metaverses

10.1 Introduction

Despitewhatmanyreadersmaythink,themetaverseisnotaproduct,not evenabrandofsomesocialnetworkcompany,butanamegiventoasetof technologiesappliedinplatformsfortheWebontheInternet.Infact,the conceptofvirtualworldsdatesbackatleasttothe19th Century[27].Still,the termmetaversewasusedtonameafuturisticconcept,describedinascience fictionbookin1992,whichpopularisedit[223],andwasshowninavisual formatinamovie20yearslater(i.e.10yearsago)[159].Metaverseinpractice today,referstoanewtypeofWebplatform,whichissupportedthrougha comprehensivesetoftechnologies,someofwhichalreadyconsolidatedand othersinevolution,whichwillallowusersgreaterinteractivityandsocialisationinimmersive3Ddigitalenvironments,representedbyauniverseofnew digitalvirtualworlds,preferablymirroredinthephysicalworld.

Metaverseresearchwitnessedafirstwaveof"hype"betweentheyears 2000and2006,withmanyresultsandvisibility.Currently,in2022,itisgoing throughasecondwaveofinterest,nowbroughtaboutbycommercialplayers thatstartedtomarkettheirmetaversesandeventsheldinsidethem,butalso byawidelypublicisedmetaverse-relatedpublicannouncementbyoneofthe WesternBig-Techsinlate2021.

Today,itispossibletounderstandwhatmetaversesare,orcouldbe,by browsingthroughWebplatformssuchasSecondLife,Decentraland,SomniumSpace,TheSandbox,Roblox,HorizonWorlds,AvakinLife,Mesh,among others.

Theconceptsofdigitalvirtualworldsoftoday’smetaversesaretypically basedonWeb2.0technologiesthatinclude2Dand3DVirtualRealityspaces, withcomputergraphicsimagesrangingfromlowtohighresolution,and someplatformsusingAugmentedRealitytechnologiesinvariousactivities. Therepresentationofusersisalwaysthroughavatars,and,asaccesstoplatformsdependsonanexclusivelogin,thereisalackofinteroperability,as avatarsareconfinedtoasinglemetaverseanditsworlds,notbeingallowed

tomovefromonemetaversetoanother,onanotherplatform,withoutlogging inagaininthephysicalworld.

Themeansofaccessingtheplatformscanbedonethroughvariousdevices thatincludesmartphones,tablets,laptops,desktops,workstations,andeven head-mounteddisplaysorvirtualrealityglasses.Someplatformsalready usemonetisationthroughblockchainandcryptocurrencies,withtheadoption ofsmartcontractsandfungibleandnon-fungibletokens(NFTs)thatenable mercantileactivities.Note,however,thattodaytherearestillfewplatforms formetaversesthatuseWeb3.0technologies,theHTTP/3protocol,andother moreadvancedandsecuretechnologicalresources,butthisisclearlythepath forthefuture.

Tosupportthesefeatures,themostadvancedtechnologiessuchasWeb 3.0(latestInternetversion),ArtificialIntelligence,Brain-ComputerInterfaces, IoT(InternetofThings),Blockchains,andVirtual,Augmented,Extended,and MixedRealitywillusherinalargenumberofopportunitiesthatwillprobably impactlargepartsofoursocieties,justlikeSocialNetworksdid.

10.2 WhoIsGoingtoBeAffected?

Inordertoanalysetheplausibleimpactofmetaversesinfuture,let’sembraceinthischaptertheirfullvisionasdigitalworldsthataremassive,immersive,persistent,openandeconomicallydeveloped,asfollows[95].

• Massive: Theycanhostanunlimitednumber,oratleastaveryhigh numberofconcurrentusers,asthecomputingpoweroftheWebplatformsandoftheusers’machinesevolvesintermsofgraphicsprocessing andconnectivity.

• Immersive: Theyofferthree-dimensionalandembodiedexperiences, basedonVirtualReality(VR)andExtendedReality(XR).Imaginethat afterworkyougotoasmallroominyourhouseorneighbourhood, dressupinaconnected“sensorysuit”,andtellthecomputerthemetaverseofyourchoiceand,fromthere,youenterthesite,havingthesensationofbeingpresentandliving“inside”achosendigitalvirtualworld, controllingmanythingswithyourthoughts.Thisisincontrasttothe currentexperienceofmostgameuniverses,whicharetwo-dimensional, confinedtoscreens,andmediatedbyclicks,typing,andeitherscreenor mouse.

• Persistent: Metaverseswillneverstoporreset.Oratleastthatwillbe theperceptionoftheirusers.Thelifeandsocietyofametaversewill continuouslyevolve,evenifsomeavatarsarenotpresent,asithappens tonormallifeinourworld.

• Open: AnyonewithgoodInternetconnectivityandVR/XRcomputing powercangointometaverses,movewithinthemasanavatar,interact withotheravatars,socialise,trade,build,produceintellectually,andso on.

• Economicallydeveloped: Therewillbeextensivetradeingoodsand serviceswithinthemetaverses,whichmayormaynothaveanimpactin thephysicalworldoutsidethem.TheywilllikelybesupportedbyDecentralizedFinance(DeFi)architecturesanddigitalmonetarysystems thatencompassblockchaintechnologies,cryptocurrencies,smartcontracts,andfungibleandnon-fungibletokensthatwillenableproperty rightsassurancepractices.

Clearly,suchanambitiousvisionpointstoahighlikelihoodofarenewed collisionbetweenIndustrialAgeGovernanceandDigitalAgeGovernance, whichwouldaffectalllayersofthepopulation,fromsimplemetaverseusers topolicymakers.

Infact,governmentsarealreadynervous.IntheEUtheEuropeanParliamentisconcernedmainlyaboutCompetition,DataProtection,Responsibilities,FinancialTransactions,Cybersecurity,Health,Accessibility,andInclusion[145],whiletheEUCouncil’smainpointsofpreoccupationareGeopolitics,Economicgrowth,Jurisdiction,Health,Consumerprotection,Civiland Penalcodes,andClimatechange[19].Wenotethatmassiveintellectualinvestmentwouldberequiredinorderforpracticalsolutionstobefoundand implementedineachoftheseareas.Besides,therewillbethornyissues aroundreachingconsensusinanyofthesetopics.Thesearesomereasons whytheEuropeanCommissionhasjustincludedmetaversepolicyamongits priorities[44, 50].

10.3

WhatIsExpectedtoHappen?

Ananalysisoftheevolutionofmetaversesupporttechnologies,suchas thosedescribedabove,theInternet,andtheWeb–fromtheWeb1.0version andthecurrentWeb2.0,tothenewlevelofWeb3.0,especiallywhenthinkingaboutWebplatformswithgreatinteractivityandgreatersocialreach–, bringsmanyquestionandconcerns,especiallyregardingcybersecurity,privacyandprotectionof(personal)data,regulations,andvariousaspectsofthe governanceofsuchdigitalworlds[216].

Takegovernance inside ofmetaversesasanexample.Theconceptof“inside”ishighlightedbecauseitisdifferentfromtheconceptofinterfacebetweenthedigitalworldofametaverseandourphysicalworldsincesuchan

interfaceisbecomingregulated,atleastintheEuropeanUnion(EU),since 2016.

IntheEU,therule-of-lawisdominantanditsinstitutionsaremostlyfit forpurpose.However,inthisnewtechnologicalfrontierthataremetaverses, itisnotclearwhatwillberegulated,whowillestablishandenforcerules,or howthiswillbedone.Butanyplace,physicalordigital,atsomepointof populationdensitywillneedsomekindofordermaintenance,includingthe notionoffundamentalrights.

Indeed,thinkingofunregulatedparalleldigitaluniversesisworrisome. Andascommercewillbeubiquitous,products,transactions,propertyrights, andotherbusinesseswillneedsomekindofprotocolsformarketstothrive. Thenallkindsofconflictingsituationswillhavetoberesolvedbysomeform ofauthorities,police,andcourts.Aswell,theremustberulesoftrade,taxation,income,etc.Butthen,ifalargesetofruleshastobeestablished,another importantquestioniswhoisgoingtosetthem:Aretheygoingtobetheownersoftheplatformsofmetaverses,sincetheseuniversesareprivatelyowned? Willtheyputuserstohelpsetuplocalrules?Orarepublicauthoritiesfrom thephysicalworldstartingoutandexpandingtheirreachintothedigital worldaswell?Whosepublicauthoritiestostartwith?Orarelibertarians thinkingaboutcreativetechnologiestogovernthemetaverses,promotingthe ideologythat"codeislaw"?Likewise,whatformdoessuchabodyofrules take?Accordingly,wecanthinkofthefollowingformsofregulation.

• Signingofusagecontracts.However,theymaybeaslongasconstitutions.

• Replicationoflawsandregulationsfromthephysicalworld.However, thismayhinderinnovation,andgoodjustificationswouldbeexpected forthechoiceofonemodeloveranother.

• Distributedmodels,basedondigitaltechnologies,likeblockchain,bitcoins,NFTs,smartcontracts(i.e.,persistentscripts).

Inaddition,theverytechnologicalofferofinteractivityandimmersionof next-generationmetaverseswillheavilydependonwearabledevicesmonitoringbothbiometric(e.g.,gait,facialexpressions,temperature)andneurometric (e.g.,fear,satisfaction,attention)data,whichwillimplycontinuousandfull surveillanceofusers.InWesternsocieties,whereprivacyandprotectionof personaldataarefundamentalrights,commercialandpublicinterestswill haveaverydifficultrelationshipconcerningthistopic.

Tocompoundsuchissues,theattacksurfaceforsecuritybreachesandprivacyinvasionscanbecomeverylargeinthemetaverse,becauseitintegrates avarietyofolder,current,aswellasuntestednewtechnologiesandsystems

10.4.WhatIstheWorstThatCanHappen?

whoseintrinsicvulnerabilitiesandflawswillbeinheritedbythelargersystem.Asaconsequence,existingsecuritythreatswillbeamplified,withmore severeeffects.Theyincludethefollowing(nonexhaustive)[244]:

• Lackofsecurityculturefromthepartofusersinsuchnewenvironments,

• Mismanagementofmassivedatastreams,

• Widespreaduser-profilingactivities,

• UnfairresultsfromArtificialIntelligence(AI)algorithms,

• Digitaltwinssecurity,

• Securityofmetaversephysicalinfrastructures,

• Personaldatainvolvedinthemetaversewillbemoregranularandbiometric,includingemotional,etc.

Finally,theenlargementoftheattacksurfacebroughtbymetaverseswill facilitateexistingthreatsinphysicalandcyberspaces,likepersecution,harassment,andespionage,whichmayincreaseinfrequencyandimpact.The useofemergingtechnologieswillmakemorelikelysecurityincidents,like hijackingwearabledevicesorcloudstorage,virtualcurrencytheft,orAImisconducttoproducefakenewsautonomouslywithinmetaverses[67].

10.4 WhatIstheWorstThatCanHappen?

Manythingscangowrongifprovisionandusageofmetaversesrunamok infuture,andmostofthemarerelatedtothenotionoftrustinthem.

Itiscertainthatthevastmajorityofmetaverseusersareandwillbelawabidingcitizensandpeoplewhovaluecivilisedbehaviour.However,among theusersisalsocertainthattherewillbecheatersandotherlesshonestpersonswhowilljoininjusttotryandmakeeasymoneyoutofwhatwould bedefinedinmostpartsofthephysicalworldascriminalactivity.Suchan environmentwouldnotinvitetrustfromusers,andlicitcommercialreturns overinvestmentmayplungeasaconsequence,whileillicitundertakingsmay flourish.

Onanother,perhapsmoreimportantregistry,metaversesplacemajorchallengestoprivacyandgovernanceandtheymayhavethepotentialtoacceleratethegeopoliticalshiftofpowerfromNationStatestoprivatecompanies. Remindthatalreadytodaysomesocialnetworkcompanieshavepopulations thatarelargerthanthatofthelargestcountryonEarth.Ifnationalgovernmentscannottrustthatmetaverseswilltreattheircitizensinalegalmanner,

thengovernmentsmaydecidetoover-regulatemetaverses,hamperinginnovationandincreasingfragmentation.

Accordingly,thelackoftrustworthygovernanceandofsecurityandprivacyregulationsinsidemetaversesmayturnthishigh-techEldoradointoa 21st CenturyWild-West,wherefortuneswillbemadeandlawlessnesswillbe theruleratherthantheexception.

10.5 ResearchGaps

Asseenabove,thefieldforStateregulationofmetaversesisvast,ranging fromissuesatmacrolevels(e.g.,geopolitics)tomicrolevels(e.g.,sellingadigitalbraceletinthemetaverse).Inanutshell,themajorcurrentEUlegislation andpoliciesgoverningthedigitalsphereareasfollows.

• DigitalMarketsAct:Regulationofcompetitionforonlinemarkets.It establishesharmonisedrulesthatdefineandprohibitunfairpractices, suchastheuseofcompetitors’dataandlackofinteroperability,onthe partof“gatekeepers”oftheWeb.

• DigitalServicesAct:Duediligenceobligationsonalldigitalservices thatconnectconsumerstogoods,services,orcontent,includingproceduresforfasterremovalofillegalcontentaswellascomprehensive protectionforthefundamentalrightsofonlineusers[46].

• GDPR:Protectionofpersonaldata.Duediligenceandcybersecurity[49].

• DataGovernanceRegulationandDataAct:WhiletheDataGovernance Regulationcreatestheprocessesandstructurestofacilitatedata,the DataActclarifieswhocancreatevaluefromdataandunderwhichconditions.[48]

• Variousincybersecurity:CybersecurityAct(egcertification),NIS2,ENISA, ECCC/NCCs,JointCyberUnit,CyberResilienceAct,etc.[45]

However,fromagovernanceandpolicyviewpointsuchexistinglegislationareprobablynotsufficienttoinducetrustinthedomain,andperhaps notevensuitableformetaverses.Consequently,muchresearchisneededin theseareasinthenearfuture.Forinstance,therewillbeaneedtoregulate securityandprivacyinmultipleuniversesthatarebeingbuiltfromscratch.

Questionsmaybesimpleextensionsofexistingconcerns,likewhethermetaversesshouldbesubjecttoexistinglawsforthephysicalworldand,ifso,how nottohinderinnovationandcreativity.Ortheymaybeturnedmuchmore towardsfutureconcepts,likewhetheravatarsshouldbegivencitizenstatus.

Likewise,thetechnologiesneededtobuildmetaversesasenvisionedhere arejustemerging,andagreatdealoftechnologicalresearchwillberequired inthenextfewyears.Moreover,onelikelyresultofmarketforcesisthat severalmetaverseswillbecreated,representingparalleluniverses,notonly betweenthem,butalsotothephysicaloneweareusedtolivein.

Whatiscertainisthatanewgoldrushhasalreadybegun.Required researchareascanbepresentedinclusters,asfollows.

10.5.1 Buildingtrustworthymetaverses

Onegovernanceresearchareashouldanalyseallaspectswithinmetaverses thatwouldimpactindividualusers.TheseencompassinteraliaDataprotection,Liability,DigitalIdentities,Cybersecurityattheuserlevel,Mental andPhysicalHealth,Accessibility,Inclusion,Financialtransactions,andConsumerprotection.

10.5.2

Metaversesandthephysicalworld

Anothergovernanceresearchareashouldproposenewsocietalsystemsfor metaversesandtheirinterrelationwithexistingformsofgovernanceandgovernment.ThesewouldincludeCybersecurityatphysicalinfrastructureand atsystemslevels,Privacy,Competition,Globalgovernance,Jurisdiction,Civil rights,Penalcode,Climatechange,Innovation.

10.5.3

Compliancebydesign

Theemergenceofmetaversesraiseawiderangeofconcernsregardingtheir compatibilitywiththelaw,asseenabove.Therefore,itwillbenecessarytogo beyondthewell-knownconceptsofsecurity-by-designandprivacy-by-design towardsanencompassingcompliance-by-designparadigm,ifatallpossible. Forinstance,researchwillberequiredaboutadaptedtechnicalregulationsto guidehardwaremanufacturersandsoftwaredeveloperswithrespecttocompliance,includingdatagovernanceandoperationalgovernancerules.

Suchgovernancetopicsshouldbeaddressedtogetherwithresearchin thenewtechnologiesandsystemsintegrationthatwillbeneededinorder toachievethefullmetaverseconceptdescribedaboveinthischapter.Some technologicalandsystemsresearchareasareasfollows.Notethattheyare intrinsicallytransdisciplinary.

10.5.4

Interactivityandimmersivetechnologies

Makingthemetaversefullyinteractiveandimmersiveisanevolutionaryresearcharea.itshouldbefocusedonthemassivecaptureandfastanalysisofdata(telemetry,biometric,andneurometrictracking,amongothers)of

usersandtheiravatars.Datawillbecollectedthrough"wearableinterfaces" (wearabledevices)ofdifferenttypesthatwillgraduallybringtometaverse XRplatformsmoreandmoresensitivepersonalinformation,whichwillneed systemicprotection.

10.5.5 Metaversesdesign

Theareaofresearchontheestablishmentofstructuredprojectsanddesign ofdigitalvirtualworldsinametaverseenvironmentnowhasgreatpotential tostudyandestablishaminimumnecessaryarchitecture.Thesecanbeplatforminfrastructures,usualprotocolstandards,securitysystems,oreventhe constructiveandoperationalaspectsoftheapplicationofXRin3D.Theestablishmentofaminimumstandardshouldnotmakecreativityunfeasible,but encouragetheeffectiveconstructionofinteroperablemetaverseswithrulesfor socialcoexistenceamongavatars,whichareacceptableinethicalandmoral terms,universally,whetherindigitalorphysicalworlds.

10.5.6

Interoperabilitybetweenmetaverseplatforms

Interoperabilityofmetaversesneedstobeintensified,sothatitshouldbe possibleforavatars(users)whoareexperiencingadigitalvirtualworldona particularmetaverseplatformofacompany,tobeabletomove,withoutimpedimentsandinatransparentway,intoanotherplatformofmetaverse,from anothercompany,withouttheneedtoidentifythemselvesagaininthephysicalworld.Researchoninteroperabilityinmetaverseenvironmentswould touchupondigitalidentitiesandallowtheestablishmentofaseamlesscollectionofmetaverses,maybeusingtheconceptofself-sovereigndigitalidentities anddigitalpassports[248].

10.5.7

MetaversesandEnvironmental,Social,andGovernance(ESG)issues

Oneofthekeyresearchpointsconcerningmetaversesrelatestotheirimpact onclimatechange,becauseoftheirneedtorelyonhugedatacentres,high performancecomputing,andevenblockchainplatforms,allofwhichnecessitateveryhighelectricityconsumption.Thisareaofresearchrequiresadvancesinarchitecturesandalgorithms,butalsoinotherareassuchascooling techniques,thatcanenabletheuseofthosetechnologieswithoutmajorenvironmentalimpact.ESGconsiderationswillplayamajorroleintheprovision andadoptionofmetaversesinfuture.

It’sworthnoticingthatintheareasmentionedabove,isolatedandunconsolidatedactionsarealreadyongoing,whichaimtocovertheexistinggaps inmetaverseresearch.Wecanmentiontheactionsof:theWorldEconomic Forum[84],theMetaversesStandardsForum[150],theOpenMetaversesIn-

teroperabilityGroup[175],andtheMetaversesInteroperabilityCommunity GroupattheW3C[229],amongothers.

10.6

Exampleproblems

Tangibleexampleproblemsinclude:

Dataprotectioninsidethemetaverse. Personaldatacollectedinthemetaversewillbemoregranular,biometric,andneurometric.Thequestion isthenhowtoreconcilethefundamentalneedofmetaverseimmersion technologiestoimplementwidespreaduser-profilingandthefundamentalrighttodataprotection,includingbioethics.Notethatsucha questiontouchesuponprotectingthedatafromboththephysicaluser andthedigitalavatar.Morespecifically,itshouldbeinvestigatedhow toensurethatmetaverseswillnotmakeillegaluseofsuchdata,forexampleforsalesandmonetisation(suchassocialnetworksalreadydo), forpromotingmediainfluence,orintheeffectiveproductionofsubliminaladvertisements,amongotheraspectsofactiveandinteractive persuasion.

Protectingavatarsfromidentitytheft. Theprotectionofavatars’identityis averyimportantissuetobesolved.Althoughtherearealreadyseveral proposalsandstrategiesforapplyingsecurityindatabases,withthe useoftechnologiessuchasdistributedledgersandscatterorhashtree structures,suchasMerkleTrees(whichare,bytheway,keyelements ofBlockchain),thereisstillnoconsensusonhowtokeepavatars’digitalidentitieswithoutcompromisingtheirLifelogging(metaverseslife history).

Regulationofcreationofmetaverses. Thetechnologiescurrentlyappliedby manyWebplatformsalreadyprovideeasy-to-usetoolsthatallowusers tocreatetheirownmetaverses.Evenifthesearesimple,theyaretotally undertheusers’control.Theproblemhereiscentredontheimproper creationofmetaversesthatcamouflagedigitalworldsmeanttoharbour avatargangsforcriminalpractices,socialactivism,racism,andterrorism,amongotherunethicalandillegalpractices.

Equalopportunitiesinthemetaverse. Ensureaccessibilityandinclusionin themetaverseinordertosafeguardequalopportunities.TheWebplatformsthathostmetaverseswillbeabletosegregateavatarsbasedon theirphysicalusers’hardwarecharacteristics,computingcapacity,personalprofile,oraccordingtothegeographicregionoftheiraccess,givingmoreprivilegestosomethanothers.

CryptocurrenciesandNFTsusageinthemetaverse. Issuesofownership,misuse,interoperabilityandportability.AstheWebplatformsareproprietary,theymaintaincontroloverthedigitalassetsownedbyavatars,as wellas,determinethemonetarystandardsused.Someplatformshave theirowninternalcryptocurrencies,afactthatcanjeopardisetheportabilityandinteroperabilityofavatars’digitalassetsbetweenplatforms.

11 Malware

11.1 Introduction

Modernmalwarecomesindifferentforms:viruses,worms,spyware,adware,trojans,backdoors,andransomware,tonameafew.Although,computerviruseswerethemostfrequentformofmalwareacoupleofdecades ago,nowadays,itisransomwarethatseemstobethemostprevalent.This isbecauseransomwareprovidesahighlyprofitableanddirectwayformaliciousactorstomonetisetheinfectedsystems.Indeed,usingransomware theseactorsinfectvictimsystems,encryptalldata,andthenaskformoney (ransom)inordertoprovidethedecryptionkey.Withoutthedecryptionkey, thelegitimateownersofthevictimsystemscannotreallyusethemasall informationisencrypted.

Todefendagainstmalware,computersecuritypractitionersusuallyneedaway to detect itinthefirstplace. Detectingafilecontaining malwareusedtobeeasy: computersecuritycompaniescomputedahash(a summary)ofthemalicious fileandjusttriedtofind filesthatmatchedthishash value.Antivirussystems usedtobenothingmore thanasetofhashvalues (onehashvalueforeachpieceofmalware)andjustsearchedforfilesthat matchedanyofthesehashvalues.Toavoidthistypeof(static)detection, modernmalwaremutatessothattwo“copies”ofthesamemalwarearenot thesame.Forexample,ineach“copy”ofthemalwaretheyintroducesmall changesthat,whilenotchangingthemainfunctionalityofthesoftware,do changeitsappearance,andconsequentlyitshashvalue.Followingthisphi-

losophy,malwareauthorsobfuscatetheircodetodeteroratleastimpedethe reverseengineeringoftheirbinariesbutalsotoremovepossiblecodepatternsthatcouldbeusedtodetectthemalware.Thismaycomeintheform ofpackers,programsthattrytocompressand/orencryptthecodeofthe malwaresothatthemaliciouscodeisunpackedandexecutedafterseveral stepsthatwouldmakethelifeofamalwareauthordifficult.Finally,modern malwareisarmouredinthesensethatithasanti-analysisfunctionalitiessuch asanti-debugging,anti-hooking,andanti-VMtonameafew.

Settingasidethedifferencesofscopethatmalwaremayhave,e.g.worms, trojans,miners,itisimportanttohighlightsomedifferencesinthesophisticationandrangeoftargets.Practically,highlysophisticatedmalware,from PegasustoStuxnet,ismostlyfocusedonattackingaspecificindividualor craftedforasingleorganisation.Inthiscase,achainofexploitsisused, manyofwhichmaybezero-daysyettheattackerisnotfinanciallymotivated. However,thesophisticationofthemalwaresignificantlydecreaseswhenthe attackistargetedatgeneralinformationsystems.Thissophisticationiswhat canmakemalwarestaybelowtheradarandincreaseitsimpactonitsvictims.

11.2

WhoIsGoingtoBeAffected?

Commonpracticeprovesthatmalwarecaninfectalmostanycomputing device.Indeed,inthepastfewyearswehavewitnessedalonglistofhighprofileorganisationsbeingcompromised:e.g.theColonialPipeline[236], Uber[52],AXA[194]tonameafew,however,theransomwarecasesarethe onesthatsurfaceinthenewsmainlybecauseofthemonetisationmethodthat theransomwaregroupsadopt.Inessence,beyondencryptingthedataand askingforransominreturnforprovidingthedecryptionkey,theattackers alsoexfiltratethedatasothattheycanstillthreatenthevictimwithpublicationofthesensitivedata.Malwaremayalsotrytoexfiltratesensitiveuser informationviakeyloggers,compromisedrecordingmedia(e.g.,camerasand microphones),etc.

Attackersmayalsousemalwaretoinfectthousandsofhostsandusethem asanarmythatobedientlycarriesoutallthetasksthatitisassigned.The networkiscalledabotnetandmayalsobeusedfordenial-of-serviceattacks. Onestrikingexamplethatstandsoutinthiscategory,notbecauseofitssize butbecauseofthedevicesthatcomprisedit,isMirai.Mirai[14]isabotnet thatmainlyinfectsinsecureIoTdevicesandusesthemtoperformdenial ofserviceattacks.Thesizeoftheproducedbandwidthtargetedapopular DNSproviderDYNandasaresult,high-profilewebsitesandservicessuch asGitHub,Twitter,andNetflixwereinaccessible[34, 251].BeyondMirai, thereareseveralbotnetswhicharecurrentlyactive,e.g.Emotet[185]which

wasresurrected[117]afteritsshutdown[80]aftercompromisingahostused todeliverseveralothermalwaresuchasTrickbotandRyuk,Mozi[20],and Mantis[252].Notably,theyhavealsobeenusedbystateactorstolaunch attacksforcyberwarfare[237].

11.3 WhatIsExpectedtoHappen?

Theimpactofmalwareismultifaceted.Thereareseveraldirectcoststhat canberelativelyeasilyquantified,suchastheamountofransomrequested. However,therearealsocoststhataremoredifficulttoquantify,suchaslost customers,lostproductivity,etc.AccordingtoSophos,theaveragecostto recoverfromaransomwareattackisontheorderof$810,000fororganisations thatdidnotpayransomanddoublethatfororganisationsthatdidpay[218]. Thesecostscoveralltheoperationalcostsanddowntimecostscausedbythe ransomware.Infact,thedamagescausedbyransomwareareestimatedto reachthestaggeringamountof$265billionby2031[29].

EvenorganisationsthatarenotIT-orientedmaysufferfrommalware.Consider,forexample,thecaseofahotel.Whileitscorebusinessisnotdelivering ITproductsandservices,hotelsthathavesufferedaransomwareattack[160] havewitnessedtheirguestsbeinglockedoutoftheirrooms,andtheirbilling, reservations,check-in/outsystemsrendereduselesseffectivelyblockingany possiblebusinesstransaction.Similarly,severalhealthorganisationshavesufferedmalwareattacks,andwehavereachedapointwhereitisjustamatter oftimeuntiltherearecasualties[176].

Basedontheabove,thereareobviousmonetaryandreputationlosesfor organisationsandindividualswhosesystemsarecompromisedbymalware. Considerthatforindividuals,othermechanismssuchassextortionmaybe usedtoharmthevictimfurtheronthepersonallevel[183].

11.4

WhatIstheWorstThatCanHappen?

Ashighlightedinthepreviousparagraph,weareonthevergeofhaving casualtiesduetomalwareattacks.However,thisisnottheonlynefarious scenario.Stuxnet[137]wasawormtargetedtodisruptIran’snuclearprogram.Whilethismaybemorethanadecadeago,consideringthecurrent turmoilinthepoliticallandscape,malwareattacksareexpectedtobefurther utilisedasameanstoattackacountry’sdigitalinfrastructure.Inthisregard, malwareattackstocripplesmartcities,criticalinfrastructuresorbigservice providersareexpectedtoincrease,asprovedbytherecentcyberattackson HSE[191],theColonialPipeline,andtheDanishtrainoperator[195].Unfortunately,thisisalignedwiththemodusoperandiofseveraladvancedpersistent

threat(APT)groupswhicharenotnecessarilyfinanciallymotivatedbutare stateactorsorstate-supported.Infact,asrecentlyreportedbyENISA,APT groupswereresponsibleformorethanhalfofthesupplychainattacksthat wereinvestigated[75].Indeed,thisleadstoseveralunprecedentedattacks, e.g.therecentSandwormattackwhichtargetedaUkrainianagricultural firm’snetworktodisruptgrainproductionandexports[155].Thepresence ofAPTgroupsinconjunctionwiththeshiftstoIoTandremoteworkingis significantlyincreasingthepotentialimpactofacyber-attack.Indeed,using theZmapnetworkscanner[66],onecaneasilyseethatmillionsofvulnerabledevicesareconnectedtotheInternet.Tomakemattersworse,aquick searchinsearchengines,suchasShodan(https://shodan.io/)andCensys (https://censys.io/),revealssimilarresults.Suchtechniqueshavebeen usedby,e.g.APT41totargetU.S.StateGovernments[30],

Allthiscreatesadangerousmixwherestrategicallymotivatedthreatactorshaveaccesstoamyriadofvulnerabledevicesthatmayaccess,directlyor indirectly,systemsthatstore,exchangeandprocesssensitiveand/orcritical information.Therefore,inthecomingyears,cyberwarfareasanextension ofgeopoliticalturbulence,andtheresultinguseofmalware,isgoingtolead tolarge-scalecyber-attacksoncriticalinfrastructuressignificantlyimpacting sectorssuchasbanking,energy,telecommunicationstonamebutafew,or evenorganizationsinthedefenceindustry[92].Thelatterimpliesthatwe mayfaceunprecedentedattacksthatmayparalysemission-criticalsystems andservices,andimpactorganisations,individuals,andthesocialfabricin boththecyberandthephysicallayer.

11.5

ResearchGaps

11.5.1

Provablysecuresystems

Asdiscussed,malwareoftenexploitssystemvulnerabilities.Therefore,anobviousquestionishowdowebuildsystemsfreefromanyvulnerabilitiesthat malwarecanexploit?Whilethislineofresearchmightbetoobroad,there isstillplentyofsecuritytobehadfromimperviouscontainersorsandboxes. Forexample,whilesomemalwaremaycompromisetheunderlyingoperatingsystemorfirmwaretheresearchquestioniswhetherwecanbuildmicrokernelandsandboxingarchitecturesthatareprovablysecure.Ofcourse,this wouldstillleaveusvulnerabletomalwarethatcompromisesapplication-level software,butcontainingtheadversaryinasandboxwouldallowustokeep coresystemfunctionalitysecureandalsomaintainseparationbetweendifferentapplicationsandservicesrunningindifferentsandboxes.seL4(https:

//sel4.systems/)asmicrokernelandQubes(https://www.qubes-os.org/) asOScanbeconsideredwell-knownexamplesinthisdirection.

11.5.2 Malwaredetection

Currentlythereisanongoingarmsracebetweenmalwareauthorsandthe “defenders",whethertheyaremalwareanalysts,digitalforensicsinvestigators,SOCs,CERTs,CSIRTsetc.Asalreadydiscussed,modernmalwareis armouredtopreventanalysisandtobemorestealthy.Therefore,malware detectionisstillacoreissueinthisresearchfield.Althoughmodernantivirus (AV)softwaremaybefarmoreaccuratethaninthepast,itisnotenough topreventtheinfectionofmillionsofdevices,primarilybecauseAVsarefocusedonstaticfeatures.Anewstreamofanti-malwaremechanisms,namely endpointdetectionandresponsesystemshasemergedduringthepastfew years.These,alongwiththeirvariants,e.g.extendeddetectionandresponse systems(XDR),trytoexploitbehaviouralfeaturesandAI/MLmechanismsto detectandblockmalwareattacks.WhilemoreefficientthanAVsastheycan detectadvancedtechniquesandlateralmovement,EDRsarefarfrombeing consideredsilverbullets[126].Tothisend,acriticalresearchquestionishow todeterminethatafileismaliciousatruntime,andblockitonceitperforms amaliciousactionwithoutallocatingalotofresources.

Thisresearchquestionalsohasmanymoreextensions.Forinstance,when analysingmalwareweoftenexecuteitinsandboxestorecordandunderstand itscapabilitiesinahighlymonitoredenvironment.Thissandboxed-basedexecutionhastwomaindisadvantages:(i)itconsumesalotofresourcesand(ii) ifthemalwarerealisesthatitisbeingexecutedinasandbox,itmayalterits behaviourtoavoidbeingdetected.Therefore,theresearchhereliesinhowto performdynamicmalwareanalysiswithoutwastingpreciousresourcesand howthiscanbeperformedagainstevasivemalware[132, 133].Moreover, weneedtofindmethodstoautomaticallytriggerthemalwareappropriately withoutcreatinglongexecutionpathsandunlockitsfunctionality.Tothis end,binaryemulationandsymbolicexecutionmaycometotherescue.Finally,wehavetohighlightthatmanysystemcallsperformedbymalware,if treatedindividually,donotalwaysdiffermuchfromthoseissuedbybenign programs;thus,evasivemalwarecanstillbypassmanyclassifiersthatcannot seethewholepicture[171].

11.5.3

Machinelearninginmalwaredetectionandclassification

Thecontinuoususeofmachinelearningandartificialintelligenceincyber securityhasalsopavedthewayforitsapplicationinmalwaredetectionand analysis.Nevertheless,wehavetoconsiderthatitcanalsobeleveragedby malwareauthorstobypassthedetectionmechanisms.Hence,itisessentialto

considerthatmalwareauthorswilltrytoexploitfeatureselectionalgorithms tomaketheirmalwareundetectablebysomeclassifiers.Asaresult,machine learningandartificialintelligencecannotsimplybeusedandexpectedtoprovideexcellentresults.First,wehavetodevotemajorresearcheffortsinorder tounderstandhowtofillinthegapinimbalanceddatasetswhereamalware familymaybeunderrepresented.Next,wehavetostudyadversarialmachine learningandhowtomakeourmechanismsrobustagainstpossiblefeature injectionorblinding[255].Oneshouldalsoconsidertheexplainabilityand interpretabilityoftheresultsofmachinelearningandartificialintelligence algorithmsandhowfeatureengineeringcanimpactthemasmalwaresamplesmayhavethousandsofsparselydistributedfeatures.Finally,oneshould alsoconsidertherelevanceofthedatasetsandmodelsovertime.Usingolder datasetsandmodelsthatmightbestateoftheartnowmaysoonbeoutperformedorperformpoorlyduetotheevolutionofbothmalwareandICT systems.

11.5.4

Extendtheplatformscope

WhilemostusersofpersonalcomputersareusingWindowsandrepresent oneofthebiggesttargetsofmalwareattacks,theyarenottheonlyones. Similarly,inmobiledevicesAndroidmayhavethebiggestshareinsmart phones,butitisnottheonlyplatformformobiledevices.Moreover,weall knowthatasignificantpartoftheInternetisnotrunningononlythesetwo platformsandthatmalwarehasbeendevelopedfor,e.g.,IoTdevices,Linuxbasedhosts,andMacOS,amongothers,focusingresearchonlyonWindows andAndroidcreatesahugegapthatisexploitedbythreatactorswhofind manyoftheless-focusedplatformsunprepared.Forinstance,thebulkof researchfocusesonPE32files,overlookinge.g.ELFfilesthattargetUnix andLinuxhosts.EvenwhenresearcherstrytostudyELFfilesthedatasets arehighlyunbalancedasmostsamplescomefromasinglefamily,i.e.,Mirai, whichmayseverelybiastheoutcomes.Therefore,thereisadefiniteneed toextendthescopeofplatformsandarchitecturesthatareusedinmalware analysisresearchandtodevelopnewmethodsandtools.

11.5.5 Commandandcontrolservers

Finally,aratherthornyissueisthegradualintegrationofdecentralisedmechanismsbymalwaretocontrolthebotnetbutalsotodeliverpayloads.For example,blockchainsanddecentralisedstorage(e.g.IPFS)havebeenproven tobeaveryrobustmechanismstoactasCommandandControlserversbut alsotohostpayloads[12,184,188].Thecrucialpointhereisthatmostofthese decentralisedmechanismsarenotregulated(indeedsomeofthemcannotbe) andtakedownmechanismsmaynotbepossible,forexample,oncesomething

iscommittedinbitcoin’sblockchain,itcannotbeerased.Thus,thereisalot ofresearchonhowtoprotectagainstsuchmalwareandhowtominimisethe exploitationofsuchecosystems.

11.5.6 Post-infectionmanagement

Acknowledgingthatthereisno100%accuracyinmalwaredetectionandpreventionmeansthatinpracticeagivensystemwillbeinfectedwithmalware atleastonce.Naturally,onemaywonderwhatshouldbethenextstepswhen malwareisdetected.Mostoftheexistingantimalwaresolutionstrytothwart attemptsatinfectionandcleanup.Perhapsthereareotherthingswecan dopost-infectiontominimiseharmorfacilitatedigitalforensics.Thiscould involveautomaticallyrollingthesystembacktoastatejustbeforeinfection. Whileforstoragetheoptionofincrementalfilesystemsmayprovideasolution,thesamedoesnotapplyformemory.

11.6

Exampleproblems

Tangibleexampleproblemsmightinclude:

CommandandControl(C2)serversanddefencemechanisms

Tomanagethe compromisedhosts,manymalwareauthorsuseC2servers,someof whicharecommercial,e.g.CobaltStrike,whosecopieshavebeenleaked butarelegitimatelyusedinredteamscenarios.Regardlessoftheirorigin,C2serversallowthreatactorstocoordinatetheactionsoftheirbots, issuecommands,exfiltratedataandperformotherattacks.Currently, therearemanyC2servers,manyofwhichareopensource,anditwould beinterestingtostudyhowdifferentsecuritymechanisms,e.g.,AVs, EDRs,firewalls,treatthesebeaconsandwhethertheyaredetectedas malicious.Maliciouspatternsinmemoryandsystemcallscanbeleveragedthroughmemoryscannersandhookingtopromptlyblocktheir functionalities.

Malwareclassifiers Thesheeramountofmalwaresamplesonadailybasis imposesmanyconstraintsonresourcesandtiming.Binaryclassification (benignandmalware)isatraditionalprobleminthefield.Goingastep further,familyclassificationandclusteringareveryimportant.Regardlessofwhethertheseanalysesareperformedbasedonbinarysimilarity measures,staticordynamicfeatures,itiscrucialtodeterminetheiraccuracyandrobustness,especiallyinanadversarialscenariowherethe threatactorsmaywanttobypasssecuritymechanismsbutifthisfails, raiseafalseflag[25].

Anti-evasionmechanismsandtriggeringmechanisms

Malwaremaytryto evadedetectionandanalysisinvariousways.Automatingthebypassing ofsuchmechanismsandcollectingrobustresultsfrommalwarethrough thecorrelationofstaticanddynamicfeaturesisabigchallenge.How dowetriggerthemalwareproperlytoexhibititsbehaviourwhenstatic analysisindicatesthatafileismaliciousyetthedynamicanalysisfails todetectthemaliciousnessofthefileinquestion?

Covertcommunicationchannelsandmalware

Manymalwareinstanceswould trytohidetheircommunicationchannelsbymixingtheirinterventions withlegitimatetraffic,e.g.usingasocialnetworkoranotherlegitimate servicetocommunicatebetweentheC2serverandthecompromised host.However,malwaremayusesteganographyandothercovertchannelstoexfiltratedataortodisseminatecommands.Detectingpossiblemaliciouscovertcommunicationandstegomalwareisachallenging problem.

Abuseoflegitimateprocesses LivingOffTheLandBinariesandScripts(and alsoLibraries)1,commonlyreferredtoasLOLBin/Script/Libarefiles thatareshippedfromMicrosoftinWindowsandothertools(e.g.Office,VisualStudio),whichbearthesignatureofMicrosoftandcanexecuteadditionalfunctionalitiestothosewithwhichtheywereinitially designed,e.g.downloadfiles,executearbitrarycontent,etc.Because oftheirsignature,whenexecuted,theydonotrequestanyuserinteraction,arewhitelistedbymostsecuritymechanisms,andcanbefoundin almostallWindowsmachines.

Threatactorshaverepeatedlyusedthesefilesinmaliciouscampaigns totrojaniseMicrosoftOfficedocumentstoexecutemaliciouspayloads. Thisapproachhasgraduallybeenabusedbyothermalware,especially filelessmalwareattacks[230].Moreover,theyareabusedbyransomware todeleteshadowcopies,e.g. cmd tolaunch vssadmin anddeletethe shadowcopies[136].

Basedontheabove,theresearchproblemliesintofindingways,based on,forexampleAPIcallcontext,processparentsandchildren,andcall argumentstodeterminewhetheracalltoalegitimateprocess,API,library,orbinaryisbeingabusedbymalwareorwhetheritisinfacta benigncall.

SoftwareLifeCycle

12.1 Introduction

Softwareisatthefoundationofalldigitaltechnologies,thusitisatthecoreof theinfrastructures,servicesandproductsthatdriveoursocieties.Thelifecycleofsoftwareconsistsofseveralphases,startingfromconception,andgoing throughdesign,realisation,deployment,operation,maintenanceand,eventually,decommissioning.Currentsoftwaredevelopmentapproachesprioritise fastdeploymentoversecurity,whichoftenresultsininsecure,expensiveto repair[202],applications.Securityconcernsare,unfortunately,stillnotfully andsuitablyintegratedwithinthelifecycleoftoday’sincreasinglycomplex softwaresystems[241].Moreover,softwareisusuallybuiltbyassembling componentsfromthird-partysources,whichraisestrustconcerns(e.g.asevidencedinsupplychainattacks[72]),makesithardtocomplywithsecurity requirementsandlegislation,andcompromisesdigitalsovereignty.Theraise ofartificiallysynthesisedsoftwareisexpectedtoaggravatethis.Last,security andprivacyregulationssuchastheGDPR[93]ortheCybersecurityAct[7],as wellascitizenexpectationschangefrequentlyandsoftwareissubjecttocontinuousupdate.Asaconsequence,softwarecompliancecannotbeassessed onceandforallandneedstobeaninherentpartofitslifecycle[135].The fieldhasseenadvancessinceearlyinitiativestobuildsecurityinsoftware systems[149]andeffortsinthisdirectionhavebeenmade,suchasNIST’sSecureSoftwareDevelopmentFramework[167],OWASP’sSoftwareAssurance MaturityModel[201],Microsoft’sSDL[153],andETSI’sstandard303645[78] (seealsoChapter12oftheCybersecurityBodyofKnowledge[56]).Nonetheless,manychallengesremain(seethelastpartofthischapterandalso[135], asexamples).

12.2

WhoIsGoingtoBeAffected?

Traditionally,ensuringhigh-qualitysoftwarewasconsideredtobemainlyrelevantforcriticalinfrastructures:finance,healthcare,energy,andsoon.However,softwareisbecomingmorepervasiveandintrinsic,uptothepointthat itcanbeseenasthecirculatorysystemofoursociety’sbody:youmaynot

noticethatitisthere,untilitsqualitystartstoaffectyourhealth.Nowadays,weusesoftwaretoregulatetheindoorclimateofourhouses,toplan ourcommutetoworkandschools,tocarryoutourdailyactivities,tocommunicatewithcolleagues,family,andfriends,toaccessmedicalservicesand treatments,andsoon.Ultimatelythequalityoflifeofeverysinglecitizenwill behighlydependentonthequalityofthelifeofthesoftwarefacilitatinghis orheractivities.

12.3 WhatIsExpectedtoHappen?

Softwarevulnerabilitiesincriticalsectorscanhavecatastrophicconsequences forourlives:companiesandindividualscanlosemoneybecauseofflaws infinancialsoftware,accesstotreatmentscanbedelayedbymalfunctionin softwareplatformsusedinhospitals,livescanbelostasaresultofsoftware bugsinmedicaldevicesorcarassistancesystems.Chapter 13 providesa representativesampleof(in)famouscasessuchastheAriane5disaster,and thelossoftheMarsclimateorbiter,andmanyotherscanbeadded.Justto mentionarecentexample,avulnerabilityinthePolyNetworksmartcontract leadtothelossof600MUSD[89].Butevenvulnerabilitiesincasesthatwe traditionallydonotconsiderascriticalcanhavesevereconsequencesforindividualcitizens:violationofpersonalprivacyisarguablythemostarchetypal example.

12.4 WhatIstheWorstThatCanHappen?

Softwarevulnerabilitiescanhaveallsortsofcatastrophicconsequences,and certainlyneedtobeaddressed.However,ensuringqualityofsoftwareisthe leastthingwecando.Softwarecanalsobeofhigh-qualityandadhereto themoststrictsecurityandprivacyregulationssuchasthehighestlevelsof theCommonCriteria[36],butharmcanstillbeobtainedifthereislackof trustworthinessinthewayitisdeveloped,acquired,used,maintainedand dismantled.Considerforexample,whatcanhappenifcitizensdonottrust thesoftwarebeingusedinthenextdemocraticelections.Eventheentire democraticsystemofacountrycanbeatrisk.

12.5 ResearchGaps

Securitymustbebetterintegratedintheentirelifecycleofsoftware,from conceptiontodismantlement.Weconsiderthefollowinggapsandpossible waysforwardtoaddressthis.1

12.5.1

VerifiableandAuditableSoftware

Agreatportionofthesoftwarecomponentsthatconstituteasoftwareproduct orserviceisobtainedfromthirdparties;thusitispotentiallyuntrustworthty asitmaynotcomplywiththeexpectedsecurityrequirements.Toachieve digitalsovereignty,thereisaneedtobeabletorelyonsoftwarethatcan beverifiedandaudited.Thepotentialsecuritygainofusingopen-source softwareamenabletoautomatedanalysisshouldbefurtherexplored.

12.5.2

ContinuousSoftwareAssessment

Securityandprivacyregulationsandcitizenexpectationschangefrequently andsoftwareissubjecttocontinuousupdate.Thereforethecomplianceof softwaresystemscannotbeassessedonceandforall,andhencemethodsand toolingtoperformcontinuousassessmentsareneeded.Giventhehighcostof securityandsoftwareassessments,theuseofautomatedproceduresiscritical toensuresustainabilityandscalability.Ifthisisnotimplementedeffectively, thesoftwarebecomestoocomplex,andmaintenanceandevolutionbecome tooexpensive,untiltheyarenolongersustainable.Wemustbreakthisvicious cycle,andfindnewwaystocreatesoftwarethatislong-lastingandthatcan becost-efficientlyupgraded,assessedandmigratedtonewtechnologies.

12.5.3

Secure-by-designAgileSoftwareDevelopment

Thedominatingapproachestodevelopmentareagileandprioritisefastdeploymentoversecurityguarantees.Moreresearchisneededtoeffectively andefficientlydeveloptoolsandtechniquestosupportsecure-by-designtechniqueswithinagileapproaches,sothatcompetitivenessandfastdeployment arenotcompromisedbysecurityrequirementsandsothatchangesinthose requirementscanbeefficientlyreassessedatanypoint,evenwhilethesoftwareisrunning.

12.5.4

LightweightFormalMethods

Manyformalmethodstechnologieshavebeendevelopedtoimprovesoftwarereliability,suchasmodelchecking,theoremproving,andmonitoring systems,butapplyingthemonalargescaletomodernsoftwaresystemsremainsachallenge.Moreeffortsareneededtofurtherdevelopandpromote lightweight,accesibleformalmethodsthatcanbegraduallyappliedtoincreasethelevelsofassurancesobtained.Methodsmustbedevelopedtosupportaspectrumofguaranteelevels,eachprovidinggreaterassurance,ina waymoreapproachablethanthecommoncriteria.Eventualenforcementsin regulationsmustbegradualinordernottoclosetheopportunityforSMEsto deliversoftwareproductsandservices,andappropriatetoolsupportisneed.

12.5.5 DecentralisedSoftwareGovernance

Softwarewithdecentralisedgovernancesuchassmartcontracts,blockchain technologies,andcrypto-assets,poseseveralchallengestothemanagementof thesoftwarelifecycle.Inthosesystems,itisunclearwhetherandhowvulnerabilitiesshouldbereportedandrepairedinawaythatharmonisesconsensus andsecurityacrossthehistoryofthesystem.

12.5.6

TrustworthyAI-poweredSoftwareLifeCycle

Artificialintelligencetechniquesarealreadybeingusedtosynthesisesmall piecesofcode.Oneshouldexpectthatinthenearfutureallactivitiesofthe lifecycleofsoftware(requirementselicitation,codesynthesis,verification, monitoring,etc.)willbesupportedbyintelligentagents.Whilethiswill certainlybringhugeadvancesintermsofscalabilityandproductivity,itis stillunclearhowsoftwarecomponentsandmethodologieswithintelligent componentscanberigorouslyanalysed.

12.5.7

SoftwareSupplyChainSecurity

Nowadays,creationanddeploymentofsoftwareinvolestheintegrationof codeandcomponentsfromthirdparties,whosedevelopmentisoutsideour control.Thesecomponentscanbethetargetofcyberattacks(e.g.theSolarWindincident).Weneedtodefineamethodologyforreducingsupply chainsecurityrisks,bymeansofassessingandguaranteeingthetrustworthinessofcomponents.Thismethodologymustbebasedonformalmodelsof contract-basedsoftwarelinedevelopmentandintegration,inordertoenable theimplementationof(semi)automatictoolsfortheverificationofsecurity properties.Thedevelopmentofthesemodelsandthecorrespondingformal methodsareanimportantresearchpriority.

12.5.8

SecureArchitecturesandPlatforms

Forbuildingsafety-andsecurity-criticalsystems,itisnotenoughtohavea trustedsoftwaresupplychain:weneedtodeploythissoftwareontrusted platforms.Thisincludesthehardwarelevel,butinparticulartheoperating systemlevel.Therefore,animportantresearchpriorityistodevelopaverifiedplatformthatprovidesfine-grainedaccesscontrolthroughcapabilities, andcontrolscommunicationbetweencomponentsofthesystem.Thiskindof platformsishighlysoughtinoperationalscenarios(e.g.theSCADAofcriticalinfrastructures)butalsoindatacentresthatprovidecloudservices.This wouldhelptorecoverdatasovereignityintheEU.

12.5.9 SecureEconomics

Anotherinterestingresearchdirectionisrelatedtosecurityeconomics,i.e.,the studyoftheincentivesfacingdifferentplayers[71].Itisnowwellestablished thatpurelytechnologicalsolutionswillnotfitthebill.Accordingly,each alternativemechanismsmustbescrutinisedagainstmarketdynamics.

12.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

Verificationatthescaleofpublicopensourcecoderepositories. Formalverificationtechniquesofferthehighestlevelofassuranceforsoftwaresecurity.Themainchallengesofcurrenttechniquesare,arguably,dueto scalabilityissuesintermsofthecomputationalandhumanexpertise needed.Howcanweraisesuccessfulverificationtechniquestothescale ofcodebasesofthesizeofaveragepopularpubliccoderepositories?

Formalmethods-poweredDevSecOps DevSecOpshasbeenadvocatedasan idealapproachtocombineDevOpsandsecurity,inordertoprovide asecurity-awareagileandfast-adaptingcontinuouslifecycle.Onthe otherhand,formalmethods,whichprovidethehighestpossiblelevel ofassuranceintermsofsecurity,safetyandperformance,havebeen traditionallyconceivedinawater-fallmind-set,rootedonformalspecificationsasthefirststep.CanwedevelopagileformalmethodmethodologiesinwhatcouldbecalledformalDevSecOps?

FormalAnalysisofSocio-TechnicalandCyber-PhysicalSoftwareSystems. Socio-technicalsystems,whosesecuritydependsintrinsicallyonhuman users,andcyber-physicalsystems,whereoneneedstoexplicitlyconsidertheunderlyingphysicalprocessesposeseveralchallengestoformalautomatedmodelling,analysisandtesting.Canwedevelopeffectiveandscalableformalandautomatedtoolsfortheanalysisandtesting ofsuchsystems?

VerificationofMLapplications. Probabilisticandrandomisedsoftwarecomponentsareatthecoreofmanysoftwareapplications,fromcryptographytomachinelearning(ML),toprivacyprotection.Recentyearshave seenadvancesinprobabilisticprogrammingtechniquesandverification techniquesforML.However,thefieldisstillinitsinfancy,while,onthe otherhand,theapplicationofMLhasbeenadvancingswiftly.Howcan weextendprobabilisticprogrammingtocopewithreal-worldML-based applications?

ResilientSmartContractRepair Ifasecurityvulnerabilityisdiscoveredina smartcontract,reportingit-ortryingtorepairit-couldtriggeraracefor

itsexploitationthatislikelytoendupwithfinancialgainformalicious agents.Canwedesigndisclosureandrepairtechniquesthataresilient w.r.t.maliciousagentstryingtakeprofit?

SecureandPrivacy-friendlyExplainability. ExplainablesecurityextendsexplainableAIwiththeneedtoconsidersecurityandprivacyaspectsof theexplanationprocessandoftheexplanationsthemselves.Howcan weadaptapproachestoexplainabilitytotakeintoaccountsecurityand privacyconsiderations?

13 TestingandCertification

13.1 Introduction

Informationtechnology(IT)is pervasiveinbothworkandsocial sectors.Home,industries,offices, cars,streetsandpublicbuildingsare fullofITdevices,systemsapps,or electronicequipment.Inourdaily lives,undernormalconditions,we areusuallynotworriedaboutthe technologyaroundus.Wearereasonablysurethatourmobilephone, PC,refrigerator,electronicdevice, car,orevenapps,cannotdamageourlife,stealdata,orcausesecurityor safetyissues,becausetheyshouldhavebeenbuiltaccordingtotherequired standards,properlytestedandfullycertified.

However,wehaverecentlywitnessedvariousexamplesofmalfunctioning orissueslikethefollowing:Teslahadafailureinaflashmemorydevice,causingasafetyriskinmorethan135,000vehicles[163];theNewJerseyhospital vaccineschedulingsystembugcaused10to11thousandduplicateappointments[65];theZoomappsufferedfromsecurityissuesduringthecoronavirus pandemicin2020[1].

AsreportedintherecentCybersecurityact,“Hardwareandsoftwareproductsareincreasinglysubjecttosuccessfulcyberattacks,leadingtoanestimatedglobalannualcostofcybercrimeof €5.5trillionby2021”[79]

Humansandsocietygenerallytrustindustriesandthebestpracticesthey adoptintestingandcertificationprocesses.However,consideringthatthe overallcostoftestingisaround40%ofthetotaldevelopmentcostsofatypicalsoftwareproject[91],ifnotstringentandwithoutconcretesafetyrisks, oftenverification,validationandassessmentproceduresarethefirsttobe reducedorskippedtosavecostandtime.Additionally,pressurefromthe needtoresearchnewproducts,thetimetomarket,andcompetitionforcesin-

dustriesanddeveloperstowardsmassivewidespreadintegrationandtheuse ofavailablethird-partyoropen-sourcecomponentsthatcouldsurreptitiously increasethecybersecurityrisksifnotproperlytestedandcertified.

InanITworldthatisgoingtobemorehuman-centricandfocusedonpeople’sneeds(suchastheInternetofPeople[IoP]manifesto[157]),thepresence ofevidenceofthetestingandcertificationactivityperformedneedstobecome acommonpractice.Weneedtoincreaseourawarenesstoavoid"poisoned"IT productsaswellaspoisonedfood.Therefore,theassessedorcertifiedquality levelmustbealabelforeachITproductinordertoestablishtrustandreduce riskstosecurityandprivacy.

Thequalityofdigitalproducts(combinationofsoftwareandhardware) mustbecomeaguaranteelabel,inthesamewayasthelabelwefindonthe foodwebuyinsupermarkets.

13.2 WhoIsGoingtoBeAffected?

Everyonedirectlyorindirectlyusingproductsortechnologiescanbeaffectedbythelackoftestingandcertificationprocesses.Forinstance,babies couldbedamagedbyatoygoingoutofcontrol,GenerationAlphaorZeta couldbeunconsciouslydeceivedbyappealingappsmaliciouslystealingtheir pictures,companiescanbeaffectedbyransomwarehiddeninusefulplug-ins orlibraries,organisationsandgovernmentscanbesubjectedtocybersecurity attacks.Ofcourse,testingandcertificationarenottheonlymeansofavoidingsuchcriticalsituations.Everythinghastobeexecutedcorrectlyatevery phaseofthedevelopmentprocess(seeChapter 12 fordetails).Conceiving anddeveloping(by-design)qualityproductsiscrucial,butnotsufficientper setomeetthefinalrequirements:buildingtheproductrightdoesnotguaranteebuildingtherightproduct[217].Testingandcertificationremainpivotal activitiesfortrustworthinessandcybersecurityassuranceandforguaranteeingthataproductisdesignedandmanufacturedwithqualityasaprimary objective.

However,aslongasstakeholders(ordinarypeople,companies,organisations,andgovernments)donotfirmlydemandtransparent,labelled,tested andcertifiedproducts,thesituationwillhardlychangeandcybersecurity riskswillstillbeontheagenda.

13.3 WhatIsExpectedtoHappen?

Whatistheexpecteddamageintheabsenceofanadequatetestingand certificationprocess?Unfortunately,therearemanyaspectstobeconsidered:

Hardware/softwarefailure: Ithasbeenestimatedthatnearly80%ofunexpecteddowntimecanbeascribedtoHW/SWfailuresandpoweroutages.Properstoragebackupscanbeanadhocsolutioninmostcases, butpreventingfailurewouldbelesscostlyandrisky.

Naturaldisastersandemergencysituations: Lackoftestingandcertification oftheprocessesandproceduresforresumingoperations/dataandsystemsincaseof(natural)disasteroremergencysituationscanbeextremelycostlyandcausethelossofbusinesscontinuity.

Humanfactor: Evennotintentionally,humansmayinevitablycausemistakes orexecutionofunexpectedprocedures.Testingbasedonuserprofiles orexploitingmachinelearningapproachescouldavoidorpredictpossiblemisbehaviouroraccidentalsituations.User-centredassessment processesandtrainingprogrammescouldbeessentialforminimising humandamageandavoidingpermanentlosses.

Cybersecurityattack: Becausesocietyandorganisationsincreasinglyrelyon digitalinformationfordailyoperations,cybersecurityattackscanbe moredangerous.Currently,95%ofcompaniesinvestintestingand certificationactivitiesonlyafteradisasterandthenactuatearecoveryplan(reactivebehaviour).Predictingvulnerabilitiesbeforehandand providingsolutionsbeforeacybersecurityattackis,therefore,mandatory(proactivebehaviour).Thepenetrationtestispivotalforavoiding andanticipatingcyberattacksbyhackerswhoaretryingtoexploitpotentialvulnerabilitiesinordertoaccesscompanynetworksandtosteal confidentialdataortoinjectmaliciouscodes.

Highexpectations: :Inourhyper-connectedworld,whereITproductsneed tobeavailable24h7dwithoutdisruptions,failuresandlossofservices arecostlydisastersforcompaniesandfavourtheircompetitors.Therefore,robusttestingandcertificationprocesses,whichcanassurethe qualityofservicesandmakeitpossibletoestablishasuitablerecovery plan,arepivotalactivities.

Trustorreputationdamage: Lossoftrustordamagetoareputationismostly translatedintoalossofcustomers,andhencealossofrevenue:trustand reputationarenearlyimpossibletoregain.Testingandcertificationare amongthemosteffectivemeansofavoidingthisproblem.

Compliancerequirements: Nowadays,businesscontinuityisnotjustamere desire:itisbecomingarequirement,especiallyforOperatorsofEssentialServices(OESs)[47].Allofthemmustfollowspecificandstrict regulationsandstandards.Thatmeansthatadoptingcertificationprocessesandmaintainingtheirproductcertificationisbecomingalegal

obligationandoffersacompetitiveadvantagewithinthereferencemarket.

13.4 WhatIstheWorstThatCanHappen?

Figuringoutwhatcouldhappenwithouttestingandcertificationshouldnot pointtothefuturebutsimplytothepast.Mostworst-casescenarioshave alreadybeencoveredinthenewspapers,thedefaultreportsanddisasterdocumentation.Theworst-casebugshistorystartedassoonasthefirstcomputer wasmassivelyusedandincluded:

• TheAriane5Disaster,4thJune,1996.DuringthelaunchoftheAriane 5spacecraft,37secondsafterthefirstrocketigniteditstartedflipping inthewrongdirection,andlessthantwosecondslaterthewholeworld observeditsself-destruction.Theproblemwasquicklyidentifiedas asoftwarebugintherocket’sinertialreferencesystemand,unfortunately,couldhavebeeneasilysolvedwithatrivialintegrationtesting procedure[247].

• TheMarsClimateOrbiter,23rdSeptember,1999.Duringitsdescent intotheMartianatmosphere,theMarsClimateOrbiterwasreoriented topassbehindMarsandsuccessfullyenteritsorbit.Unfortunately,this didnothappen:thecraftwasnotonthecorrecttrajectoryanditwasfinallylostwithoutatrace.Therootcauseanalysisofthiserroryieldeda longchainofwrongorunexpectedevents,whichincluded:theincidentalarrangementofsolarpanelsonthecraftduetothesolarsaileffect; theuseoftwodifferentunitsintheGroundControlsoftware(dataprovidedusingimperialunitsandpound-secondsonthesendersidebut expectedinmetricunitsonthereceiverside);andfinally,humanerrorsincommunications.Again,properintegrationtestingprocedures andcorrectuseofstandardsandassessmentprocedureswouldhave avoidedsuchacriticaldisaster[105].

• Therac-25 Duringtheperiodfrom1992to1998,thereportsaboutradiationoverdosescausedbythe80’scomputer-controlledradiationtherapywerepublished.Inparticular,sixdocumentedaccidentsoccurred, resultingindeathsorsevereinjuries.Thecauseswereidentifiedasthe applicationofincorrectproceduresbypersonnelandtheweaknesses ofthesoftwareusedforassuringsafety.Inparticular,allaccidentsinvolvingsoftwarehadresultedfromflawedsoftwarerequirements.Applicationofcertificationprocessesandapropersystemandacceptance testingprocesswouldhaveagainavoidedsignificantlossoflife[140].

• KnightCapitalGroup On1stAugust2012,duringasoftwareupdateof theproductionserver,anincorrectconfigurationofanold(2003)system

caused97emailnotificationsandtheexecutionof4millionunexpected trades.Thatledtoa$460millionlossandtheriskofbankruptcy.The post-analysishighlightedthattheprogrambelieveditwasinatestenvironmentandexecutedtradesasquicklyaspossiblewithoutworrying aboutlosingthespreadvalue.Asinthepreviouscases,thetesting processwouldhavediscoveredthatmisbehaviourandavoidedusing obsoleted,notalignedsoftware[189].

Itislikelythatpastmistakeshavebeenresolvedandlessonslearnt,but challenges,vulnerabilitiesandnewscenariosareconstantlyemerging.Who doesnotremembertheMillenniumbug[4]?Orthe2018cyberattackthat interruptedcommunicationsontheMidcontinentIndependentSystemOperator?Oreventhesix/sevenhoursoftheglobalunavailabilityofthesocial networkFacebookanditssubsidiariesinOctober2021[228]?Ortherecent ransomwareattacksontheITnetwork?

Thesmartandquickdiscoveryandprovisionofnewtechnologies,programminglanguagesandsystemsobligestestingandcertificationtocontinuouslyjump “BacktotheFuture” andprovidenewmeans,strategiesand processestopreventfutureworst-casescenarios.Indeed,historyteachesthat thepastcanalwaysturnintothefutureand vice-versa

WhatIstheWorstThatCanHappen?Alifewithouttestingandcertification,becauseitmeansalackofquality,efficiencyandtrustineverysystem andsoftwarepackage.

Indeed,testingandcertificationseektomitigatetherisksofsafety,security andprivacylossorabsenceforanyoneworldwide.Whowoulduseamachine withoutitbeingtested?Whowouldbewillingtosetupamedicalfacility withoutbeingcertified?Whocouldthinktogiveachildtoysthatputtheir lifeatrisk?

Unsafe,notsecureornottrustableHWorSWproducts,elements,components,andlibrariesmaketheworlddangerous:theycancauseenvironmental disasters;theycanplayaroleinthedefaultorbankruptcyofcompanies, industriesandevennations;theycanimpactessentialservices(i.e.energy, transport,financialandbanking,healthcare,drinkingwatersupply&distribution,anddigitalinfrastructures);theycancompromisehealthsystems ormedicaldevices.Thecurrentinternationalsituationcanalsopainteven moredramaticscenarios:HW/SWvulnerabilitiesandsecuritythreatscould beexploitedtoallowterroristattacksonnuclearpowerplantsandmilitary bases.

Luckily,inthiscatastrophicapocalypticscenario,learningfromthepast andfocusingonthefuture,researchandindustryarestartingtounderstand theimportanceofstrictcollaborationintestingandcertificationtoeffectively preventdisastersbeforetheyhappen.

13.5 ResearchGaps

Consideringthat“Programtestingcanbeusedtoshowthepresenceofbugs, butnevertoshowtheirabsence.(Dijkstra)“[61],exhaustivetestingisusually impossible,andissuesandproblemsintestingandcertificationarefarfrom beingexhausted.Newchallengesarecontinuouslyaddedinparallelwith thedevelopmentofnewtechnologies,features,languagesandapplication domains,andthediscoveryofnewvulnerabilitiesandthreats.Inparticular, thefollowingareasarerecenttrendsinresearchactivities.

13.5.1 Human-centredTestingandCertification

Supportinghuman-centredtestingandcertificationapproachesthatareable toguide,improveandassesstechnologicaldevelopmentinlinewithsocial andethicalvalues,sustainabilityandtrustworthiness.Additionally,increasinginclusivenessbysupportingthegenderanddiversitybalanceofdifferent stakeholdersinvolvedinthetestingandcertificationapproachcanensure trustworthypublicawareness,thebroadadoptionofITmethods,andthe adoptionofstandardstoincreasetransparencyandopenness.

13.5.2

Integratedcybersecurityandfunctionalsafetycertification

Besidesinterleavingandoverlappingseveralaspectsofcybersecurityand safety,thereisstillagapinprovidingacomprehensiveframeworkandtechnicalstandardsfortheirfullintegration.Indeed,safetyassurance/certification cannotbeachievedwithoutconsideringtheimpactofcybersecurityvulnerabilitiesandthreatsonthesystem.Thus,thereisaneedtoprovideafunctional safety/cybersecurityassurancerisk-basedintegratedapproach.

13.5.3

Quantitativeandqualitativetestingandcertification

Accountabilityandreplicabilityareessentialcharacteristicsofcybersecurity modelling,testingandcertificationapproaches,andrequiremethodsand meansforquantitativeandqualitativecollectionandtheanalysisofresults anddata.Thus,theavailabilityofopen-sourcedatasetsandconformance testsuitesasthefacilitiesforthesettingupandexecutionofcontrolledexperimentsshouldbeimproved.Inparticular,challengesfocuson: (1) Improving formalmethodsforquantitativesecuritymodellingandanalysisandtheir applicationtoriskmanagement,enrichingtheirdata-drivenaspects,e.g.synthesisingandrefiningmodelsfrom(possiblyunderspecified)attackscenarios andvalidatingthemconcerningdatafrompreviousattacks. (2) Realisationof modelling,testing,andcertificationapproachesdrivenbycybersecurityrisks (3) Makingdatacollection,quantificationapproaches/tools,andresultanalysismoreaccessibletopractitionersandopen-accesscommunities. (4) Improvingtheefficacyandefficiencyofthetestingandcertificationprocesses,

makingthemmorefocusedonqualitativeproperties. (5) Makingtestingand certificationbydesign,guidedbyuserstories,domain-specificneedsrequirements,andstandards. (6) Providingmetrics,guidelines,andapproachesfor securingproductsandservicesthroughouttheirlifetime.

13.5.4 AutomationofTestingandCertification

Testingandcertificationarecomplex,costlyandtime-consumingactivities. Reducingtheeffortandmitigatingthecybersecuritycostandriskisasignificantchallengeforattainableautomation.Importantdirectionsare:

1. Developingadvancedtechniques,findinginnovativesupportprocedures to(fully)automatethedifferentactivities,orprovidingmetrics,guidelinesandapproachesapplicablethroughouttheoverallprocesslifetime

2. Providingaholisticmethodologythatintegratesruntimeanddesigntimemethodsapplicableatdifferentspecificationlevels—suchasfirmware, communicationprotocols,stacks,operatingsystems(OSs),andapplicationprogramminginterfaces(APIs)—andthatconsiderstheintegration ofsoftwareandhardware.

3. Specifyinganddevelopingmanageableandhuman-centricKPIs,metrics,procedures,andtoolsfordynamicandautomaticcybersecurity certificationfromchiptosoftwareandservicelevels.

13.5.5 Diversity,heterogeneityandflexibilityofenvironments

Diversity,heterogeneity,andflexibilityarechallengingattributesoftesting andcertificationproposals.Inparticular,anyapproachesandsolutionsprovidedshouldmoveaccordingtoverticalandhorizontalresearchlevels.Indeed,ecosystemsandsystemsofsystems(SoS)relyonthecontinuousintegrationofcomponents,appsanddevicesdevelopedusingdifferentlanguages andoperatingsystems,andoncombiningandaccessingthousandsofdevicebrowser-platformcombinationssimultaneously.Toavoidtheriskofbecoming outdated,testingandcertificationneedhighlyflexibleandmodularschemes thatrapidlyadapttothechangesandupdatesofthetechnologicalenvironmentandelementsateachhorizontalorverticallevel.Additionally,tofollow therapidandpervasiveevolutionofthedifferentsupplychainenvironments (suchasthecriticalinfrastructuresdescribedinChapter 9),andnewtechnologies(likethemetaversesdescribedinChapter 10),holistic,modularproposals arenecessary,abletoeffectivelyandefficientlyvalidate,verifyandcertifythe differentHW/SWelementsunderrealuserconditionsandconsideringother interactingsystemsandapplicationdomains.

13.5.6

Includinglegalaspectsinsidetestingandcertification

TheinterplaybetweenHWandSWelementsincurrentsystemspromotesa newdirectionforcybersecuritytestingandcertificationresearch:toinclude legalaspectsintheverification,validationandassessmentprocedures.The legalframeworkandtechnicalstandardsmustbeconsiderednecessaryparametersduringthedevelopmentlifecycle(formoredetailsrefertoChapter 12).Indeed,cybersecurityvulnerabilitiesmaycauselegalviolations,especiallyinsensitiveapplicationssuchashealthcare.Thefuturedirectionisto ensurethatcybersecurity,safetyandlegalrequirementsaretestedandcertifiedasinseparableaspectsofthesameprocess.

13.6 Exampleproblems

Tangibleexampleproblemsmightinclude:

Testingtheunknown. SoSscontinuouslyintegratevariousnewdevicesand components;someofthemcouldbeuntestedandanyintrinsicflaws willbeinherited.Theresearchshouldpavethewaytonewtesting paradigmstoachieveself-adaptivetestingmethodologiesaimingatensuringthatunknownanduntestedcomponentsanddevicesaretrustable andhavegoodqualitybeforetheyjointheSoS.Inotherwords,thisresearchshouldpromote“FullQuality–positive-sum,notzero-sum.” 1

TestingofAI/ML/DL. Providetestingmethodologiesandtoolsthatcanbe suitableforrevealingbugsinartificialintelligence(AI),machinelearning(ML)ordeeplearning(DL)applications.Thestudyshouldconsider thefollowingthreemainaspects:therequiredconditions(correctness, robustness,securityandprivacy);theAI,MLorDLitems(e.g.thedata, thelearningprogram,ortheframeworkused);andtheinvolvedtesting activities(testcasegeneration,testoracleidentificationanddefinition, andtestcaseadequacycriteria).

UsingAI/ML/DLfortesting. ProvideAI/ML/DL-basedmethodologiesand toolsthatcanhelpperformmosttestingtasks,suchastest-casegeneration,test-caseclassification,oraclederivationormutationanalysis, tociteafew.Therefore,thisresearchaimstoleveragestate-of-the-art AI/ML/DLtechnologiestoaidsoftwareandhardwaretestersinachievingthedesiredqualitydrivenbytestingdata.

Understandingthetestabilityofthemetaverse. Improvetheunderstanding ofthechallengesoftestingthemetaversebyconsideringthreetesting

1Thistermisinspiredbythewell-knownprivacybydesignprinciple“Fullfunctionality: positive-sum,notzero-sum”[35].SeealsoChapter 8

pillars:cybersecurity,aimedatsecuritytesting;APItesting,crucialfor guaranteeinginteroperability,whichisafundamentalcharacteristicof themetaexperience;andinteractiveandimmersivetesting,whichputs thehumanatthecoreoftestingmetaexperiences.

Wearealltesters. Improvetheunderstandingoftheroleofhumansinthe testingprocess.Theresearchshouldprovidetheories,insights,and practicalsolutionsforengagingpeopleinthetestingandassessmentof digitalproductsandservices,consideringdifferentdimensionsof(digital)ethnography.Thestartingpointforthiskindofresearchshouldbe gamification,whichaimstoconverttestingtaskstogameplaycomponents,andcrowd-sourcedtesting(alsoknownascrowdtesting),which isanemergingapproachforinvolvingusersandexpertsintestingactivities.

14 IoTSecurity

14.1 Introduction

TheInternetofthings (IoT)isacollectionofdevices(i.e.things)thatcontainsensorsand/oractuators,software,andcommunicationcapabilitiestosend andexchangedatawith otherdevicesontheInternet.TheideaoftheIoThas beenwithusforquitesome timenow,anditssecurity hasalwaysbeenandstillis oneofitsmainchallenges.

Overtimethetypesandcapabilitiesofeverydaydevicesconnectedtoa networkcontinuetogrowquickly.IoTAnalytics[106]estimatedthenumber ofconnectedIoTdevicesfor2021tohavebeen12.2billionglobally.Meanwhile,Statista[238]hasestimatedamoremodest11.3billionconnectedIoT devicesin2021.Thepredictionisforthenumberofdevicestomorethan doubleby2030,withStatistaestimating29.4billionconnectedIoTdevices. IoTrepresentsoneofthebiggestsecurityconcernsatthepresenttimeandin thefuture,asthenumberofsuchdevicesisprojectedtogrowandpermeate allaspectsoflifeevenmoredeeply.ThisiscorroboratedbythefactthatIoT devices,onaverage,getattackedwithinfiveminutesofbeingconnectedto theInternet[158].

Whilemostmightthinkofsmartrefrigerators,robotvacuumsorsmart watcheswhentheyhearthewordIoT,therearemanymoresuchdevices thatsupportandsurroundusinourdailylivesaswellasinindustries.IoT canbeusedinmanyareastohelpoptimiseand/orautomateprocessesby gatheringlivedata.ThemostcommonapplicationsofIoTincludesmart homes,smartcities,wastemanagement,smartgridsandpowermanagement,

industry(i.e.Industry4.0),agriculture,smarthealthcare,smartwarehouses, smarttransportandlogistics,etc.Fromtheenvironmentwheretheyareused andtheirpurpose,wecanderivedifferenttypesofIoTs.Forexample,Internet ofIndustrialThingsorIndustrialInternetofThings(IIoT),ConsumerInternet ofThings(CIoT),InternetofMedicalThings(IoMT)orInternetofHealthcare Things(IoHT),InternetofAgriculturalThingsorInternetofFarmingThings (IoFT),InternetofEnergyThings(IoET),InternetofVehicles(IoV),Internetof TransportationThings(IoTT),InternetofEducationThings(IoEdT),etc.

ThereisaconsiderablelistofpropertiesthatmakeIoTdevicesandnetworksvulnerable.Forexample,theubiquityofIoTdevicesmakeitdifficult toprotectthemagainstphysicalaccess.Atthesametime,thediversityof devicesmakesitdifficulttodesign"one-size-fits-all"securityconstructsthat couldbefreelyappliedtothedevices.Evenmore,therapidlifecycle(ofthe devicesthemselvesandthedevelopmentprocess)alsomakesithardtotrack thedevicesonthemarketandapplysoftwarepatches.Discoveredvulnerabilitiescangounpatchedforanextendedamountoftime,andevenifthereis apatch,mostoftheusersfailtoregularlyupdateIoTdevices.

Tomakemattersworse,IoTdevicesareoftenleftwiththeirdefaultsecurityconfigurations(e.g.factorypasswords)whichleavesthemevenmore vulnerable.Andfinally,severalIoTdevicesarerathersmall,withlimited power,memoryandcomputationalcapabilities.Thisoftenmeanstheyare notcapableofrunningthebestsecuritymechanismsandprotocolsandmust insteaduselesscomputationallydemandingandresource-intensesolutions thataregenerallynotassecure.

AddressingcommonchallengesintheIoTecosystem,ofwhichsecurityis certainlyoneofthemostimportant,iskeytothefutureofIoT,especiallyas IoTbecomesmoreandmoreingrainedinourlivesandnolongerrepresents athreatonlytooursensitiveinformation,butalsotoourphysicalassetsand health.ForalloftheconvenienceandvaluethatIoTprovides,therisksare alsounparalleled.

14.2

WhoIsGoingtoBeAffected?

IoTdevicesaffectnearlyeverybody.Forexample,individualscanbeaffectedinmanyways.Ifasmarthomecomesunderattackandstopsfunctioningcorrectly,theinhabitantscanlosepower,heating,light,entertainment, etc.WebcamerasandbabymonitorsarealsoverycommonhouseholdIoT devicesthatregularlygetattackedandhavepreviouslybeenusedtospyon theirownersortoformpartofabotnet.Individualsmayalsobeindirectly affectediftheattacktargetstheirorganisation,theirgovernmentoranyother entitytheyarepartof.

IndustriesmayalsobenegativelyimpactedbyIoTattacks.Indeed,asuccessfulattackonsuchanIoTsystemwouldcauseoperationstocease.Any organisationinthesupplychainwouldalsosufferconsequences,especiallyif theattackwasaimedatpostal/transportorganisationsthatmanagethetransportationofgoods.Whilethisistruerfororganisationsorindustriesdealing withmanufacturing,onlineservicesaremorevulnerabletothingslikeDDoS (DistributedDenialofService)attacks,whichmakeonlineservicesinaccessiblebyoverloadingtheserviceproviderswithfakerequests.Someofthe largestsuchattackswerelaunchedfromhijackedIoTdevicesthatformeda botnet(alargecollectionofdevicesthatweresuccessfullyattackedandsubvertedtodotheattacker’sbidding:e.g.[178, 206]).

Forefficiencyandtransparency,manycriticalinfrastructuresandgovernmentalservices(e.g.power,water,andwastemanagement)arebecomingIoT supported.Anyattackthatwouldundermineanyofthemforanyextended amountoftimewouldcausehavocinthepopulationandresentmenttowards thegovernment.AttacksagainstIoTcouldalsobeusedtospyonpoliticians or,again,usingDDoSattacks,tomakedigitallysupportedgovernmentalservicesunavailable(e.g.eHealth).

14.3

WhatIsExpectedtoHappen?

AsIoTprogressestobecomepartofeverything,manythingscouldbe affectedwhensomethinggoeswrong.Webcameras,babymonitors,voice assistants,smarttoysandsimilartoolscanmonitorpeoples’activitiesand conversations(e.g.[108, 165, 192]).Medicaldevicescollecthighlysensitiveinformation,includingprotectedhealthdata.Smarttemperaturesensorscan tellpeoplewhensomebodyisathome,andsmartlockscanlettheminwhen theyarenot.IoTdevicesusedinmanufacturingcouldbeusedforindustrial espionagetoobtainsensitiveinformationaboutmanufacturingprocessesand procedures,orthewholemanufacturingprocesscouldbeshutdown.AttacksonIoTinsmarttransportandwarehousingwilldisturbsupplychains. Attackscanaffecttrafficwheresmarttrafficmanagementisused,andattackerscantakeoversmartcarsiftheycangainaccess(e.g.[100]).Byattacking smartwatermanagementandpowergrids,largeregionscanbeleftwithoutpowerandwater,whichbringsanyindustrytoastopandcausespeople theredifficultieswithcookingandkeepingwarminthewinter.Inagriculture,asuccessfulattackthatisnotnoticedquicklyenoughcanleadtoruined cropsordeadlivestock.HavingIoTdevicesexpandstheattacksurface,sowe canexpectmoresuccessfulattacksbyattackersgainingaccesstoprotected networksthroughseeminglyinconsequentialIoTdevices(e.g.[245]).Large amountsofsuccessfullycorruptedIoTdeviceswillbemergedintobotnets

thatwillthenbeusedforcryptominingortoperformlargeattacks,suchas DDoS,tocrippleonlineservicesorwholepartsoftheInternet(e.g.[58]).IoT attackswillregularlycomeintheformofransom,wheretheattackerswill demandmoneytostopanattackornotbeginitinthefirstplace.

Giventhesefewexamples,thepotentialdamagethatcouldbecausedby losingsecurityoverIoTsystemsisimmense.Consequencesincludelossof privacy,identitytheft,effectonhealthorevenlossoflives,stealingofintellectualproperty/competitiveadvantage,lossofproperty,goodsshortages, decreasedfoodproduction,unavailabilityofonlineservices,difficultieswith thesupplyofelectricityandotherenergysources,etc.

14.4 WhatIstheWorstThatCanHappen?

Intheprevioussection,wetriedtoshowhowmuchcouldgowrongif IoTsystemsgetcompromised.Inthissection,however,wewanttogivesome worst-caseescalationsofthoseexamples.Inthecaseoflosingprivacy,there aretworeallybadoutcomes.Thefirstisthelossofanonymity,whichwas alreadycoveredinapreviousbookchapter,whilethesecondisidentitytheft, whichisconsiderablyalarming,especiallyifithappensinlargenumbers.MaliciousmedicalIoTdevicescancausehealthdegradationorevendeath,but evenworsearedevicesimplantedinhumans(i.e.pacemakers).Ransomware onsuchdevicesisbasicallyremotekidnappingthatdoesnotleavethevictim withanynegotiationoptionsoralternativetopayingtheransom.Thelossof runningwaterandpowerisbad,butifalargeenoughareaisaffected,that wouldplungetheinhabitantsintoadarkage,whichinmoderntimeswould becatastrophic.Anattackonthewatersupplycannotonlystoptherunning water,butitcanalsomakeitpoisonousbyalteringthewatertreatmentat thewaterplant.Anywidespreadsuccessfulattackoncriticalinfrastructure wouldhavedevastatingconsequencesforgeneralsecurity(e.g.military),nationaleconomicsecurity,nationalpublichealthorsafety.Maliciousattackson manufacturingplantscanalsocausetheproductionmachinerytobreak,stoppingproductionforaverylongtimeorevencausinginjuriesordeathsamong employees.UsinglargeenoughbotnetstoperformDDoSorothertypesof attackcouldcripplelargesectionsoftheInternetand,withit,everythingthat reliesonthatinfrastructure(e.g.communications).

14.5 ResearchGaps

IoTsecurityisaproblem,anditwillgetworseasthepotentialattack surfaceexpandswithmanymoredevicesandwithmorecriticaldevices(e.g. medicaldevices).

14.5.1

ArtificialIntelligenceandMachineLearning

Artificialintelligence(AI)andmachinelearning(ML)promisetobeahuge helpinsecuringandidentifyingattacksinIoT[10, 114, 203].TheintroductionofAIandMLintotheIoTenvironmenthassomeassociateddifficulties, suchasdeploymentonconstrainedanddistributeddevicesandtheneedfor updatingAI/MLmodelsovertime,whichcanbeproblematicforreasonsof accessibilityandgeneralupdatingpractices-aswediscusslaterinthissection.Overcomingtheseissues,AIandMLcanprovideagreatdealinterms ofsecurityforIoTsystems.AIandMLcancopewithheterogeneousdataand gothroughlargevolumesofdataproducedbyIoTmuchmorequickly(i.e. inrealtime)thantraditionalmethods,enablingthemtodiscoverattacksas theyhappen.Suchsolutionscanbeutilisedforaccesscontrol,security,malwaredetectionandanalysis,riskassessment,threatanalysis,privacy,attack detection,andpotentiallytracingtheattacktroughthesystem.AI/MLisalso agoodfoundationforprovidingadditionalsystemresilience.Deeplearning hasalreadyshownpromisingresultsinidentifyingIPSpoofingandDDoS attacks,anddecentralisedmachinelearningcouldbeespeciallycompatible withIoT.Weneedsolutionsthatareabletoidentifythesubtletiesofsecurity breachesandmitigatethemwhileconformingtothelimitedresourcesofIoT devices.Thisincludesefficientlabellingofinputstreamsandlearningwith smallersetsoftrainingdata.Weneedmethodsforsuchsolutionstowork notonlyinenterprises,butpotentiallyalsoinmuchsmallerenvironments, regardlessofthetypeofdatatransmittedthroughtheIoTnetwork.

14.5.2

StrongandUniversalSecurityStandardsforIoTTechnology

SecuritystandardsinIoTandtheirapplication,ingeneral,needsomework [13].Thequickdevelopmentofsolutionsandtheheterogeneityofthedevices certainlydonotmakestandardisingIoTsecurityanyeasier.UniversalstandardsorguidelinesshouldbesetforIoTdevices,includingdataprotection atrestandduringcommunication,authenticationandauthorisationofIoT devices,maintenanceandmanagementofIoTdevices,auditingandlogging, andsecureinterfaces(web,applicationAPI,cloud,andmobile),andIoTsecurityincidentresponseprocesses.Ingeneral,moreIoTdevelopmentshould followthe"securityandprivacybydesign"paradigm,especiallyfordevices thatcollectpersonaldataand/orcanhaveasignificantimpactontheirowners’healthorassets(e.g.smartlock).

14.5.3

DevelopStrongandLightweightCryptographyforIoT

SomeIoTdeviceshaveseverelylimitedresources,andtoretainfullfunctionality,securityandusability,theyrequirelightweight(cryptographic)protocols. Lightweightsolutionsmustbeefficientintheircomputational,memoryand

14.IoTSecurity

powerconsumption.Forthispurpose,weneed(standardised)lightweight IoTsolutionsfordataencryption(atrestandintransit),keymanagement, routing,authentication,andaccesscontrol.Additionally,malwareisalsoa largeproblemforIoTsystemsandforthesamereasonsoflimitedresources, malwaredetectionsolutionsthatcanbeeffectiveinsuchenvironmentshave tobefurtherdeveloped.

14.5.4 EstablishTrustandTraceability

TakingintoaccountthesecurityconcernssurroundingtheIoT,establishing trustinthedevices,theirprocesses,andthecollectedandtransmitteddatais important.CurrentIoTsystemslacktransparency,makingitimpossiblefor ordinaryuserstoknowwhatisgoingon,whatdataisbeingcollectedand whathappenstoit.Thisincludeslivemonitoringthatcannotifyusersin realtimeofanymaliciousbehaviourinIoTsystems.Monitoringisalsovery importantforself-healingcybersecurityIoTsystemsthathavethepotentialto automatecybersecurity.

Datatraceabilityandintegrityarevitalforincreasingtrustindataand, consequently,thewholeIoTsystem.Distributedledgershavebecomethe primarysolutionfordatatraceability;however,somedevelopment,especially inscalability,isstillneededbeforetheycanbefreelyappliedtolargerIoT networks.Atthesametime,trustisalsorequiredamongstIoTdevicesina network.Thispreventsattackersfromjoiningthenetworkormasquerading asoneofthedevicesinthenetwork.Forthis,weneedbettersecuretrust managementsystems.

14.5.5

IoTSecurityAwarenessandEducation

IoTusersarecurrentlynotwellawareofthesecurityrisksandespecially theavailablemitigationcontrolstoreducetheserisks[115].Thisisespeciallytrueinpersonal/homeenvironmentsandsmallerbusinesses,butitis unfortunatelyalsooftentrueinenterpriseenvironments.Themostcommon problem,andonethathasbeenexploitedverysuccessfullyevenintherecent past,istheuseofthedefaultpasswordsthatthedeviceswereshippedwith ortheuseofweakpasswords.Moreeffortisrequiredforeffectiveawareness methodsandtoolsforinformingthepublicofthedangersofinsecureIoT(eitherinsecuredevicesorweakconfigurations).IoTproductsshouldcomewith clearerinstructionsfortheusersonhowtosetuptheirdeviceswithanemphasisontheimportanceofsecurityandprivacysettings(thiscouldbepart ofthemanualand/orashardwiredpolicies,e.g.defaultpasswordswould havetobechangedduringthesetuptoapasswordofsomeminimumquality).Reportedlythereisalsoalargeshortageofprofessionalstoimplement

IoTnetworksinbusinesses,includingcybersecuritytalent[118].Appropriate trainingandupskillingprogramsshouldbedesignedandputinplace.

14.5.6 HardwareSecurity

WithIoTdevices,itisimportanttorememberthattheycoverawiderange ofusecases,andinsomeofthem(e.g.whendevicesareinstalledoutside protectedenvironments),thephysicalorhardwaresecurityofthedeviceitself isasimportantasanythingelse[18].Thisaspectoftenseemstobeforgotten, andIoTdeviceslackhardwaresecurity,suchascryptographiccoprocessorsor anti-tamperingtechnologies.Therefore,weneedmorelow-cost,efficientand well-testedmodules,whichincludehardwaresecuritythatmanufacturerscan reliablyuseintheirIoTproducts,andwemustprovideincentivesforthemto beused.Inthissection,trustedgatewayscanalsobementionedasawayto minimisetheattacksurfaceandthedamagetoorganisations.

14.5.7

PrivacyinIoT

PrivacyisanimportantchallengeinIoT[107].Privacypreservationrestricts theprocessingofdatatoonlythestrictlynecessary,andinawaythatpreventsadditionalsensitivedatafrombeinginferredthroughoutthedata’slifecycle.Itmustalsostrikeabalancebetweendatautilityandprivacy.Weneed moreemphasisonprivacyduringthedesignanddevelopmentofIoTandbetterprivacy-preservingtechniques(e.g.anonymisation)thatcouldbewidely adoptedinIoT.

14.5.8

Lifecyclemanagement

Adevicecanbesecuretoday,butthisconditioncouldchangeduringitslife cyclebecauseofanewlydiscoveredvulnerability.Thesecuritymanagement shouldbescalableandasautomaticaspossibleifwewanttodealwithalarge numberofheterogeneousIoTdevices[109].However,thismightnotalways bepossible.SinceIoTdevicesarenotusuallyequippedwithtraditionalinterfaces,andupdatesarenotpushedtothedevices,usersdonotknowthereare newupdatesorpatchestheyshouldinstall.Weneedmethodsofnotifying deviceownerswhentherearecrucialupdatesorpatchestheyneedtoinstall withoutthemlosinganyfunctionalityofthesystemstheyhavesetup(ifupdatingmeanslosingdataordeviceconfiguration,manywillchoosenotto update).Finally,anadditionalchallengethatneedsfurtherresearchistodevelopefficientupdateproceduresforIoTdeviceswithverylimitedresources (e.g.notenoughmemorytodownloadanupdate).

14.5.9

IoTRegulationandPolicies

Attheendoftheday,evenifthetechnologyexiststhatcanmakeIoTsecure, itisstillimportantforthetechnologytobeimplemented.Asisoftenthe case,regulationtakessometimetocatchupwithtechnologicaladvances,and whilewehaverecentlyseensomemovementonregulatingIoTsolutionsand theexpectedlevelsofsecuritytheyshouldprovide,thereshouldbemore.We needsomewayofimposingminimumsecuritystandardsforIoTdevices(e.g. certification).

Onecrucialmatterthatcouldbealleviatedwithregulationisthelong-term supportofIoTdevices.Todayyoucanbuyadevice,andthemanufacturer willenditssupport(ifithadanyinthefirstplace)atanypointinthefuture, withoutevennotifyingthedeviceowners.Giventhecurrentpoliciesofsustainabledevelopment,minimumcriticalsecuritysupportcouldbeprescribed byregulation,ortherecouldbearequirementforproductstohaveaclearly markedsupportdurationontheirpackagingatthetimeofsale,whichthe manufacturerwillguarantee.

14.6

Exampleproblems

Tangibleexampleproblemsmightinclude:

Machinelearning-basedcybersecurityforIoT. StudyIoTattackpatternsand developIoT-friendlyrawdata-labellingmethodsfornewmachinelearningsolutionstorecogniseattacks.Createanomalydatasets.Develop newdeeplearningsolutionsfordetectingattacksand/ormalwareon IoTnetworks.

IoTdevicesecurityclassifications. ToalleviatetheproblemofIoTdeviceheterogeneity,developaclassificationschemeforIoTdevicesbasedontheir resourcelimitationsandpurpose(i.e.howcrucialissecurityforthe device,basedonwhatitismeanttodoandwhattypesofdataare involved).Theclassificationcouldbeusedtodeterminewhatarethe minimumsecurityfeatures(e.g.securityprotocols)thedevicehasto supportforittobeconsideredtohaveanacceptablelevelofsecurity, givenitssecurityclass.

SmarthoneypotsforIoT. EstablishemulationofIoTdevicesonuniversal computerplatforms.Enablemonitoringandcollectionofdatafromthe distributedIoThoneypotnetwork.

LightweightprotocolsforIoT. Findoradaptsuitableexistingprotocolsor developnewcryptographyprotocolsforIoT(potentiallyforeachIoT devicesecurityclassificationfromthepreviousexample).Theselection

14.6.Exampleproblems

ofprotocols(fordataencryption,bothfordataatrestandintransit,key management,routing,mutualauthenticationofdevicesinthenetwork, etc.)canbepromotedasgoodpracticesand/orstandardised.

Updateandpatchnotificationsforordinaryusers. CompileadatabaseofIoT devicesandhardwareusedandanyconsequentupdatesorpatchesreleasedfortheirsoftwareorfirmware.Giveusersoptionstofindtheir devicesinthedatabaseandsubscribetobenotifiedifupdatesorpatches areeveravailablefortheirdevices.Provideinstructionsonwheretoget themandhowtoinstallthem.

Improvedauthentication. IoTdevicessufferfromoveruseofdefaultand weakpasswords.Effortsshouldbeputintodevelopingconvenientways ofincorporatingmulti-factorauthenticationintoIoTdevicesanddevelopingandimplementingpasswordlessauthenticationforIoTdevices.

EffectiveThreatModelling

15.1 Introduction

Thereisgrowingtrendinsecurityofshiftingleft,thatisapplyingsecurityactivitiesearlierinthesoftwaredevelopmentlifecycle.Threatmodellingstarts fromanarchitecture-level(ordesign-level)descriptionofthesoftwaresystemorservicethatisbeingdeveloped,andstrivesforearlyimprovementsin termsofsecurityandprivacyby(1)identifyingthreats,(2)prioritisingthese threatsintermsofriskandpossibledamage,and(3)suggesting/offeringpossiblemitigationsatthearchitecturallevel.Suchanapproachisbeneficial,as itenablestheidentificationofsecurityflawsearlyontoreducetheimpactof changes[232].Therelevanceandusefulnessoftechniqueslikesecurityand privacythreatmodellingisdemonstratedbythegrowinginterestinthreat modelling.Indeed,organisationssuchasMicrosofthavemadegreatstrides inaddressingsecurityintheearlyphasesofthedevelopmentlifecycleaspart oftheirsecuritypushintheearly2000s[112, 130, 227],withtheintroduction ofsecuritythreatmodellingandthesecuritydevelopmentlifecycle.Therelevanceandimportanceofconsideringsecurityinthesephasescontinuestobe recognisedandisconfirmedwiththe2021releaseoftheOwasp top10[180] whichexplicitlyincludesinsecuredesignasatop10entryandspecifiesthe needtoperformmorethreatmodelling[212].Furthermore,ithasbeenappliedtomanysystemsinpractice.Forseveralofthese,concretethreatmodels areavailable,suchastheSecureDropwhistle-blowersubmissionsystem[86] andKubernetes[233].Suchasystematicandcomprehensiveanalysiscanbe anindispensabletooltoidentifyproblematicdataflowsinapplicationsthat arelaterleveragedaspartofransomwareattackstofurtherpropagatethemselves.

15.2

WhoIsGoingtoBeAffected?

Clearly,theactivityofthreatmodellinginvolvessoftwarearchitectsandsecurityexperts.Itintroducesanadditionalandpossiblycostlyactivitytothe developmentprocess,yettheyieldcanbearelativelyhighlevelofassurance: manyclassicalsecurityandprivacyflawscanbeavoided“byDesign”.Ifthis

Modelling Manual construction

Consistentwith implementation

Analysis Analysis effort

Traceabilityto implementation

Prioritisation

Identify keythreats

Explainable priorities

Figure15.1:Challengesineachthreatmodellingphase

werenotcovered,thesameflawscouldbehiddenandnotdiscovereduntil later,attheimplementationlevel.Thiswouldentailmuchlargerinvestments andeffortstodealwiththesespecificsecurityproblems.Ineffect,theserviceorsoftwareprovider(company)remainsinneedofacost-effectivethreat modellingprocess.Usersanduserorganisationsofthecorrespondingproductorservicemightnotbeawareofthismatter,yettheywouldstillhaveto paythebillattheendofthejourneyofsolvingstructuralsecurityproblems.

Whiletechniquesforthreatmodellinghavealreadyshowngreatpotentialinsupportingthedesignanddevelopmentofsecuresoftwaresystems, thebroaderapplicationofthesetechniquesasapartofthesoftwaredevelopmentprocessesintroducesanumberofchallenges(showninfigure 15.1)for practitionerswithregardtothecostofapplyingthesetechniquesincontemporarydevelopmentprocesses[214].First,theapplicationofthesetechniques istypicallyanactivitythatincludestheinvolvementofsecurityexperts,a scarceresourceinmanycompanies,whichhindersthebroaderandmorefrequentapplicationofthesetechniques[253].Second,theapplicationofthese techniquesentailssomemanualeffortincreatingandmaintainingarepresentationofthesystemandanalysingsucharepresentationtoidentifysecurity threats.Anymanualeffortaspartofanactivitythatis,ideally,frequently repeatedasasoftwaresystemisfurtherdevelopedandextended,introduces anon-trivialoverheadthatimpedesitsfrequentapplication.Furthermore, thecostofmaintainingandre-analysingthisrepresentationisexacerbatedin thecontextofcontemporarydevelopmentpracticesthatarecharacterisedby frequentiterationsandfast-paceddevelopment.

15.3 WhatIsExpectedtoHappen?

Thedrawbackofnotperformingthreatmodellinghasbeensuggestedabove. Yetthecurrentcostofthreatmodellingishigh,andtheresearchchallenges introducedinthischapterareofutmostimportancetoincreasethecosteffectivenessofcurrentandfuturethreatmodellingpractices.

Asmentionedabove,theapplicationofsecurityandprivacythreatmodellingcommonlyinvolvesamanualinputorassessmentsbythreatmodellers, suchasthecreationofamodelrepresentationofthesystemunderconsideration,theelicitationofthesecurityandprivacythreats,theprioritisationof thesethreatstodeterminethemostimportantonesand,finally,suggestingappropriatemitigationstoaddresstheidentifiedthreats.Practitionersencounter severalchallengeswhenapplyingthesethreatmodellingactivities:(i)acomprehensiveanalysisofasoftwaresystementailsasignificantamountofwork, inbothconstructingthemodelofthesystemandtheactualthreatelicitation; (ii)theanalysescanfrequentlyleadtolonglistsofthreats,buttheseresults lacksinformationontherelevanceofthesethreats,hinderingtheidentificationofthemostcriticalones;(iii)itisessentialtoensurethatthemodelused fortheanalysisremainsconsistentwiththeactualimplementationofthesystemunderdevelopment.Eachofthesechallengeswillbeexplainedinmore detailbelow.

15.3.1

Manualwork

Oneofthelargestchallengestothecost-effectivenessofsecurityandprivacy threatmodellingistherelianceonmanualeffortinboththecreationofthe modelsandtheanalysisforelicitingsecurityandprivacythreats.Sincethe threatmodellingreliesonusingadesignrepresentationofthesystem(typicallyadataflowdiagram[59,112])toanalyseforsecurityandprivacythreats, sucharepresentationmustberetrievedorconstructedbeforethethreatanalysiscanstart.However,frequentlysuchdesigndocumentationisnotavailable forthesystemsthathavebeenbuiltorarebeingextended.Becauseofthat, thedesignofthesystemunderanalysiswillhavetobereconstructedbyrelyingondocumentation(totheextentitisavailable)andgoingthroughthe implementationoftheapplication.Thisreconstructioneffortalreadyimposes additionalcostwhenperformingathreatmodellingexercise,andthiseffort mayhavetoberepeatedfrequentlyifthemodeldocumentationisnotkept uptodatewiththeapplicationasitisfurtherdeveloped.Asecondsource ofmanualeffortcanbetheanalysisitself.Theamountofeffortintroduced bythisstepdependsontheextenttowhichpractitionerscanrelyontool supportfortheanalysisorinsteadperformtheanalysismanually.Themore informalthesystemdescriptionsare,themoretheanalysiswillhavetorely onamanualassessmentbyathreatmodeller,asautomatedtoolsrequirea richermodelinput,includingmoreinformation,toenablethetooltomake thethreatelicitationdecisionsautomatically.

15.3.2 Prioritisation

Thesecondchallengeisrelatedtousingtheresultsofthethreatanalysis insubsequentphasestosupportdecisionsonapplyingsecurityandprivacy countermeasuresintheapplicationunderdevelopment.Astheavailableresourcestoaddresssecurityandprivacythreatsarelimited,practitionersneed tobeabletodeterminewhichthreatsarethemostrelevantandimportantto address.However,thesecurityandprivacythreatelicitationonlyrendersa (large)listofthreatsthatareapplicable.Itdoesnotprovideanysupportin identifyingthemostrelevantthreatsamongthemthatshouldbeaddressed first.Theseelicitedthreatscommonlylackinformationneededtoprioritise theresultingthreats.Asaconsequence,theprioritisationofthethreatsinvolvesamanualactivityinwhicheachthreathastobemanuallyassessed todetermineitsrelevance.Whilesuchanapproachmaybeappropriatefor asingle-shotanalysis,itisineffectiveifthethreatelicitationisfrequentlyrepeatedandtheresultinglistofthreatschangesaswell.Furthermore,support fortrackingthepriorityorimportanceofthreattypesisfrequentlylimitedto averycoarsegrainedclassification(e.g.low,medium,orhigh)thatdoesnot includeanykindoftraceabilityinformationwhensuchaclassificationdecisionwillhavetobereassessedlateron.Becauseofthelackofinformation, itisnotpossibletoassesswhythatparticularprioritywasassignedtothat threatatthetime.Ifcertainassumptionsunderlyingthatdecisionturnoutto beinvalid,itisnotpossibletoidentifyallrelevantthreatsthatwouldrequire areassessment.

15.3.3

Ensuringuptodateresults

Afinalchallengeforpractitionersistoensurethatthethreatanalysisresultsremainuptodateandrelevanttotheapplicationunderdevelopment. Especiallywithcontemporarydevelopmentpracticesthatinvolvefast-paced developmentandfrequentiterations,thedesignoftheapplicationcanchange frequently.Theresultisthatthethreatanalysisresultsfrompreviousversions ofthedesignarenolongerrelevant,assomethreatsmaynolongerbeapplicable(forexample,duetotheremovalofcertainelementsinthedesign).This introducesachallengeinkeepingthedesignrepresentationsofthesystemup todatewiththeimplementationasitevolvesduringdevelopmentleadingto additionalmaintenancecoststoensurethethreatresultsarecurrent.

15.4 WhatIstheWorstThatCanHappen?

Theprevioussectionoutlinedthedifferentchallengesandproblemsexperiencedbypractitionersintheapplicationofthreatmodelling,especiallyin termsofoverheadandcost-effectivenessoftheseapproaches.whileagreat numberofscenarioscanbeconstructedtoillustratetheimpactofvariousse-

Modelling

Model reconstruction Modelcompliance checking

Analysis

Automated elicitation Linkingto code

Prioritisation

Risk-driven prioritisation

Traceable riskresults

Figure15.2:Opportunitiesandimprovementsineachthreatmodellingphase

curityflawsbeingmissedinthedevelopmentofaconcretesoftwareproduct. Theworstcaseisactuallyunknown,astheactualimpactofnotperforming anyofthesesecurityanalysescannotbepredictedbecauseoftheuncertainties intheapplications,theorganisations,thecontextsinwhichtheapplications areused,thetypesofdataprocessed,etc.Hence,themainfocusisonthe roleofautomationinreducingoverheadandeffortasawaytoenablethe broaderuseandapplicationofthesetechniques.Thesuccessfulenhancement ofthreatmodelling,largelythroughautomation,willbenecessarytodriveto adoption,whichinitsturnwillenabletheavoidanceofexpensiveworst-case scenariosinthereengineeringandfixingofcomplexsoftwaresystemsand services.Itishardtoestimateworst-casescenariosintermsofdamage.

15.5 ResearchGaps

Theresearchagendathatisessentialtodrivethissubdomainofthesecure softwaredevelopmentlifecycleisrelativelystraight-forward.Itmainlyincludesactivitiesthatrelatetoreusableknowledge,automationandtoolsupport,etc.Theessentialresearchthemesandactivitiesaresummarisedbelow.

15.5.1 Automation

Akeyelementinthestrategytoaddressthesechallengesistostrengthen automationandapplyitinmanythreatmodellingactivitiestoreducethe manualstepsandenablefrequentreassessmentaspartofiterativedevelopmentpractices.Indeed,automationcanplayacrucialroleinreducingthecost ofthreatmodellingbyautomatingmanystepsthatinvolvecostlyandmanualinputsbydevelopersandexperts.Webrieflyoutlineeachofthephases (showninfigure 15.2)ofthethreatmodellingprocessesinwhichautomationcansignificantlyimprovecost-effectivenessbyreducingoreliminating manuallabour.

Modelling Oneofthefirststepswhereautomationcanbeappliedisinthe constructionofthemodelrepresentationthatservesastheinputforthethreat

modellingactivity.Thisisalsooneofthemostchallengingareastoapplyautomation.Therearetwomainapproachesthatcanbetakenthatprovide differentdegreesofreducedmanualeffort.First,aftertheconstructionofan initialmodelrepresentationofthesystem,modelcompliancechecking[186] orarchitecturaldriftanalysis[231]canbeusedtoverifywhetherthemodel representationactuallycorrespondswiththesourcecodeimplementationof thesystem.Suchanapproachstillrequiresaninitialmodel,butcanreduce thecostofkeepingthemodeluptodateasthesystemcontinuoustobe furtherdeveloped.Second,amorecomplexandmorefullyautomatedapproachistorelyonmodelconstruction.Thisapproachemploystoolingto automaticallycreateamodelstartingfromthesourceoftheapplication,thus eliminatingtheinitialeffortinmodelconstruction.Thesetechniquescanof coursebecombinedwiththecompliancecheckingtoverifytheaccuracyof thereconstructedmodels.

ThreatElicitation

Thesecondstepwhereautomationcanbeleveragedis duringthreatelicitation.Therearetwoareasinwhichautomationreduceseffortandmanualinput:theelicitationitselfandtheautomatedapplicationof expertknowledge.Forthethreatelicitationitself,theuseofautomationcan ensureacomprehensive,systematic,andrepeatableanalysisofthesystem. Manyexistingthreatmodellingtools[120, 156]doprovidethisfunctionality already,rangingfromsimplecriteriatomorecomplexmodelpatterns[235]. Automatedtoolscanconsistentlyapplycomplexrulesetstosystemdesignsto ensurerepeatablethreatelicitation.Thesecondbenefitofautomationinthe contextofthreatelicitationisthatitallowsexpertknowledgeaboutsecurity andprivacythreatstobeencodedintotoolsupport,enablingtheautomated applicationofthisknowledgewithouthavingtorelyonsecurityandprivacy expertstoassistintheassessments,asthesearescarceresourcesfororganisations.

Prioritisation

Thethirdstepwhereautomationintroducesbenefitsisinthe prioritisationoftheelicitedsecurityandprivacythreats.Giventhesubstantial numberofsecurityandprivacythreatsthatmaybeelicited,beingabletoprioritisethembecomesessential.Thelargenumberofthreatsmakesitincreasinglydifficulttoreviewthemforprioritisation,especiallyiftheanalysisis frequentlyrepeatedinresponsetochangestothesystemdesign.Automation providestwokeybenefitsinthiscontext.First,becausetheautomationwill relyonadditionalinformationintheinputmodelstodeterminethepriorities ofthethreat,itactuallyforcestheexplicitspecificationofthisinformationin theinputmodels.Whilethisintroducessomeoverheadtoprovideadditional input,italsoallowstraceabilityoftheresults,astheresultingprioritiescan beexplainedthroughtheinputsandrevisitedlater.Second,itremovesthe needformanualassessmentandprioritisationofthethreats,makingitmuch

moreeconomicaltofrequentlyreanalyseasystem.Suchautomationrequires theconstructionofriskmodels[87, 215]thatcanbesystematicallyapplied.

15.5.2 Toolsupport

Asillustratedabove,therearemanyopportunitiesforautomationtoreduce manualeffortandenableamorecost-effectthreatanalysisofasystem.Tool supportiscrucialforachievingautomationinthesedifferentphasesofthreat modelling.Thenecessarytoolsupportrangesfrom:(1)sourcecodeanalysis toolstoperformcompliancecheckingormodelreconstruction;(2)automated threatelicitation,leveragingencodedexpertknowledge;and(3)automated prioritisationofelicitedthreatsusingriskassessment.

15.5.3

Educationandtraining

Afinalareaofimprovementistoprovideeducationandtrainingtoenableall personneltoparticipateinthreatmodellingandfurtherreducetherelianceon securityandprivacyexpertsforthreatmodellingactivities.Togetherwithtool support,educationandtrainingfacilitatestheembeddingofthreatmodelling inexistingsoftwaredevelopmentprocesses.

15.6

Exampleproblems

Tangibleexampleproblemsmightinclude:

Creatingandmaintainingmodels. Anythreatmodellingactivityrelieson thecreationofaninitialmodelofthesystemtobeanalysed.Thecreationandmaintenanceofthesemodelscanintroducesignificantoverheadforthreatmodellershinderingthefrequentapplicationofthese techniquesduringdevelopment.Therehavebeenseveraladvances[186] thatmakeiteasiertodeterminewhetherthesemodelsarestillcompliantwiththecode,thusreducingtheeffortinvolvedinmaintenance.The analysisofsourcecodetoconstructmodelsthatarereadilyuseablein threatmodellinganalysisisstillachallengingproblem.

Automatingthreatknowledge. Therearemanypubliclyavailableresources withinformationaboutpreviouslyidentifiedsecuritybugs,weaknesses, andflaws(e.g. cves, cwes,etc.).Theseresourcesarehighlydynamic, astheyarefrequentlyupdatedwhennewissuesareidentified.While someoftheseresourceshavealreadybeensuccessfullyintegratedinto automatedanalysisactivities,suchasthedetectionofvulnerabledependencies,notallresourcesareeasilytranslatedandappliedinathreat modellingcontext.

Integrationindevelopmentprocesses. Theapplicationofthreatmodelling isusuallyanactivitythathappensinisolation.Thisintroducessome

additionaloverheadandcomplexityintranslatingthethreatsidentifiedinthesystem’sdesignintoveryconcreteandactionableitemsfor developerstoworkon.Thereareseveralchallengesinimprovingthe actionabilityoftheresultsofthreatanalysesbysupportingatighter integrationindevelopmentprocessesandrelatingthreatmodellingresultstoconcretesourcecodeartefacts,forexample,byguidingtowards startingpointswhenmitigatingtheidentifiedthreats.

16 GrandChallenges

Inthissectionwedescribesome“grandchallenges”thatwewillneedtoface inthenextfewyears.Thesechallengesrequirethecollaborationofhundreds ofpeoplefromseveraldifferentrealmsofscience.Mostofthesechallenges notonlyinvolvenovelresearch,butalsoneedappropriateregulationand possiblylegalframeworksinplace.Wehopethatthefundingagencieswill providesupporttotheseareasandthattheresearchcommunitywillstart workingtowardsthesechallenges.

16.1

Giveusersassuranceaboutthesecurityoftheirdevices

Mostcomputingdevicestodayofferlittle, ifany,assuranceaboutthelevelofsecurity theyprovide.Althoughsomeofthem(such asmedicaldevices)mayadhereto safety standards,mostofthemdonotadhereto any securitystandards atall.Asaresult, theyprovidenoguaranteestotheirusers: theymaycrashatanytime;theymayget compromisedatanytime;theymayturn hostileatanytime.Webelievethatwe shouldprovideuserswith(i)bettertransparencyand(ii)betterguarantees aboutthesecurityoftheirdevices.Althoughthissoundslikeataskthatcan beachievedthroughregulation,ithassignificantresearchanddevelopment dimensionsincludingcontinuousmonitoring,aggressivepenetrationtesting, andcontinuousbugdetectiontonameafew.

16.2

Overthepastyearswehavemovedseveralofoureverydayactivitiestocyberspace.TheCOVID-19pandemicintensifiedthistrendsothatatthepeak ofthepandemictheonlywaystointeractwithotherpeopleinvolvedthedigitalworldatsomelevel.Asaresult,westarteddoingallourshoppingonline,

ourvisitsmovedtoteleconferences,ourschoolingwasdoneviaZoom,severalaspectsofworkalsomovedonline,etc.Whatwedidnoteasilyrealise, though,wasthatinordertocarryouttheseactivitiesonlinewehadtoprovide agreatdealofpersonalinformation,andinthiswaysacrificeourprivacy.For example,inthepastitwaspossibletodomostofourshoppingpractically anonymously.Wecouldvisitstoresanonymously,browseforvariousproductsanonymously,wecouldevenpayanonymouslyusingcash.Atnopoint inthisprocessdidwehavetorevealourname,ouraddress,ourtelephone numberetc.Wecouldrevealthisinformation(ifwewantedto),butwedid nothaveto.Todayitisalmostimpossibletodoanyshoppingonlinewithout revealingagreatdealofpersonalinformationsuchasourname,ourtelephonenumber,ouraddress,etc.Suchpersonalinformationisrevealedtoa widerangeofdifferentactorsincludingthemerchant,onlineadvertisers,the couriercompany,etc.Webelievethatitisnowtimetoreclaimourprivacy andrevealaslittleinformationaspossible.The guidingprinciple hereisthat ifitcanbedoneanonymouslyoffline,itcanalsobedoneanonymously online.Thisisnotaneasytaskanditmayinvolveseveralaspectsbesides researchincluding,forexample,awarenessanddeployment.Itmaynoteven bepossibleinsomecasesandwithsomeproviders.However,havingthisasa guidingprinciplewillhelpustrimdownallthecaseswhereprivacyhasbeen unnecessarilysacrificed.

16.3

MakeAISafeforPeople

AIisspreadingwidelyandrapidly.For example,arecentwhitepaperbyDeloitte showedthattheworldwillseeAI-driven GDPgrowthof$15.7trillionby2030.ThecapabilityofAI,andMLmodelsinparticular, toextract/learncomplexfeaturesfrommassivevolumesof(often)unstructureddatais whatmakesthemapopularchoicefortacklingvariousproblems.Yet,asdiscussedin Chapter 4,ML-poweredapplicationsoffera wholenewspectrumofsecurityandprivacy exploitsforpotentialadversaries.

First,MLmodelsareoftenappliedtosectorswherewrongdecisionmakingcanhaveseriousimplications.Yetitmay oftennotpossibletoofferformalsecurityguarantees,giventhosemodels’ non-deterministicnature.Second,MLmodelsareoftentrainedonpersonal/sensitivedata,especiallymodelsdeployedinthehealthcarefield.Thus,

16.4.Makesystemsresilientunderattack

revealingtraininginstancesconstitutesaseriousviolationofindividuals’privacy.

Asaresult,weneedtodeveloptechniquesandmechanismsfor makingAI safeforpeople.Notethatdoingsoisnotaneasytaskandinvolvesbringing togetherresearchersandpractitionersfromawiderangeoffields,suchas mathematics,linguistics,informatics,etc.Infact,forspecificusecases,itmay notevenbepossibletoprovidethedesiredguaranteeswithoutsacrificingthe model’sperformance.However,workingtowardsthisdirectionwillsurely leadtosignificantimprovementsandnoveltechniquesofferingacceptable trade-offs.

16.4

Makesystemsresilientunderattack

Computersystemscanberemarkablyfragile.Indeed,awrong ifstatement,a wrong assignmentstatement,oranundefined globalvariable isallittakestocrash anapplicationoreventocompromiseacomputer.Tomakemattersworse,ifaprogram withthewrong ifstatement runsonmillionsofcomputers,allthesecomputersmay becompromisedinamatterofhoursoreven minutes!Thegrandchallengehereistodevelopcomputersystemsthatare abletotoleratecyberattacks.Wewouldliketohavesystemsthatfailgracefullywhenareattackedbycyberattackers.Wecannotavoidhavingmillions (orevenbillions)ofcopiesofaprogramrunningonvariousdevices.Indeed, therearebillionsofpeopleandtensofbillionsofdevicesrunningasmall numberofultra-popularapplications.Thechallengeinthisenvironmentis tomaketheseultra-popularprograms(andallcomputersingeneral)resilient tocyberattacks.Thereareseveraldifferentpathsonecanexploreinorderto achievethisresilience.Althoughthepathsmaybedifferent,mostofthem agreethatanapplicationshouldfailgracefullyunderattack.Thisgraceful failuremaymeanthatonlyasmallfractionofthecomputerswillbecompromised,orthatonlyatinypartofthefunctionalitywillbecompromised, orsomethingelse.Theunifyingpoint,however,istomakesystemsmore resilienttocyberattacks;onewrong ifstatement shouldnotbeabletocompromisemillionsofcomputers.Weshoulddomuchbetterthanthat.

16.5

EnhanceGeneralPublicAwarenessofCybersecurity

Peopleareoftenperceivedastheweakestlinkinthecybersecuritychain. Theyareamajorcontributingfactortothemajorityofcybersecuritybreaches, ascybercriminalsfrequentlyemploytechniquesthatexploitinnatehuman

weaknessestocarryoutattacks.Enhancingcybersecuritycompetencedevelopmentthroughtrainingandawarenessinitiativesfocusesonenablingpeopletoestablishtechnicalandoperationalbarrierstocybersecuritythreats,and tooperatethemselvesassuch,throughthevigilantprocessingofactionable intelligence.Boostingthepotentialimpactofsuchinitiativesrequiresthepersonalisationandtailoringoftheawarenessortrainingexperience.Thismust takeintoaccount,amongotherthings,personnelroles,knowledgefoundations,competences,andexperiences.Itshouldalsoincludetheoperational contextoftheinvolvedorganisations,includingpolicies,processes,andapplicableregulatoryframeworks.

Thefirstgrandchallengeherehastodowithcreatingamappingofthe competencebenchmarksthataretobeachieved,dependingonthedistinctorganisationalcontextsandthecorrespondingpersonnelroles.Thisalsoreflects onthepersonalspherewhenreferringtosocietalhardeningandawareness. Thesecondgrandchallengehastodowiththedeliveryofcompetencedevelopmentprograms,whichmeansstructuringtheappropriatemessageto achievethetargetedlearningobjectives,selectingasuitablemediumofcommunication,anddeterminingthetimeintervalsandotherparametersthatare dependentontheparticipantsandcansignificantlyaffectparticipationand retention.Addressingthesechallengesrequiresamultidisciplinaryapproach, involvingexpertisenotonlyinpedagogicalsciencesandcybersecurity,but alsopsychology,domain(i.e.sectorspecific)experience,andotherareas.Furthermore,socialsciencesanddataanalyticscanbecontributingfactorsthat canenhanceandfacilitatetheaforementionedmapping,whilealsocontributingtotailoreddelivery.

Bibliography

[1] 8zoomsecurityissuesyouneedtoknowabout. https://www.sigmundsoftware.com/ blog/zoom-security-issues-coronavirus/.Accessed:2022-11-119.

[2] Digitaleconomyandsocietystatistics-householdsandindividuals. https://ec.europa.eu/eurostat/statistics-explained/index.php?title= Digitaleconomyandsocietystatistics-householdsandindividuals.Accessed: 2022-11-119.

[3] Howdoburglarsusesocialmediatofindtargets? https://www.homewatchgroup.com/ how-do-burglars-use-social-media-to-find-targets/.Accessed:2022-11-119.

[4] Year2000problem. https://en.wikipedia.org/wiki/Year_2000_problem.Accessed: 2022-11-119.

[5] Securesme:Cybertipsforpasswords,Sep2021.

[6] Openauthenticationstandardsmoresecurethanpasswords,Nov2022.

[7] N.AchiagaandM.D.Mar.TheNIS2Directive:Ahighcommonlevelofcybersecurityin theEU. https://www.europarl.europa.eu/thinktank/en/document/EPRS_BRI(2021) 689333.[Accessed:07November2022].

[8] A.AdamsandM.A.Sasse.Usersarenottheenemy. Commun.ACM,42(12):40–46,dec 1999.

[9] R.AgrawalandR.Srikant.Privacy-preservingdatamining. SIGMODRec.,29(2):439–450, may2000.

[10] R.AhmadandI.Alsmadi.Machinelearningapproachestoiotsecurity:Asystematic literaturereview. InternetofThings,14:100365,2021.

[11] C.AlcarazandJ.Lopez.Wide-areasituationalawarenessforcriticalinfrastructureprotection. Computer,46(4):30–37,2013.

[12] S.T.Ali,P.McCorry,P.H.-J.Lee,andF.Hao.Zombiecoin2.0:managingnext-generation botnetsusingbitcoin. InternationalJournalofInformationSecurity,17(4):411–422,2018.

[13] E.Andrukiewicz,S.Cadzow,andS.Górniak.Iotsecuritystandardsgapanalysis. = https://www.enisa.europa.eu/publications/iot-security-standards-gap-analysis,12019. [Accessed:07November2022].

[14] M.Antonakakis,T.April,M.Bailey,M.Bernhard,E.Bursztein,J.Cochran,Z.Durumeric, J.A.Halderman,L.Invernizzi,M.Kallitsis,D.Kumar,C.Lever,Z.Ma,J.Mason,D.Menscher,C.Seaman,N.Sullivan,K.Thomas,andY.Zhou.Understandingthemiraibotnet.In Proceedingsofthe26thUSENIXConferenceonSecuritySymposium,SEC’17,page 1093–1110,USA,2017.USENIXAssociation.

[15] S.AralandD.Eckles.Protectingelectionsfromsocialmediamanipulation. Science, 365(6456):858–861,2019.

[16] R.Arizon-Peretz,I.Hadar,G.Luria,andS.Sherman.Understandingdevelopers’privacy andsecuritymindsetsviaclimatetheory. EmpiricalSoftw.Engg.,26(6),nov2021.

[17] ARM.Buildingasecuresystemusingtrustzonetechnology.In ARMSecurityTechnology. ARM,April2009.[Accessed:16November2022].

[18] Arrow.Understandingtheincreasedimportanceofhardwaresecurityiniottechnologies. = https://www.arrow.com/en/research-and-events/articles/understandingthe-importance-of-hardware-security,52020.[Accessed:07November2022].

[19] ART.Metaverse:Virtualworld,realchallenges.Technicalreport,AnalysisandResearch TeamoftheCounciloftheEuropeanUnion,Mar.2022.[Accessed:07November2022].

[20] D.Atch,G.Regev,andR.Bevington. https://www.microsoft.com/en-us/security/ blog/2021/08/19/how-to-proactively-defend-against-mozi-iot-botnet/,2021.

[21] M.Azure.Openenclavesdk. https://openenclave.io/sdk/.[Accessed:17November 2022].

[22] M.Bada,A.Sasse,andJ.Nurse.Cybersecurityawarenesscampaigns:Whydotheyfail tochangebehaviour?In InternationalConferenceonCyberSecurityforSustainableSociety, pages118–131,012015.

[23] O.Barajas.Howtheinternetofthings(iot)ischangingthecybersecuritylandscape. https://securityintelligence.com/how-the-internet-of-things-iot-ischanging-the-cybersecurity-landscape/,092014.[Accessed:07November2022].

[24] R.Barrett. BuildingaValues-drivenOrganization:AWholeSystemApproachtoCulturalTransformation.Butterworth-Heinemann,2006.

[25] B.BartholomewandJ.A.Guerrero-Saade.Waveyourfalseflags!deceptiontacticsmuddyingattributionintargetedattacks.In VirusBulletinConference,pages1–9,2016.

[26] V.Boehme-Neßler.Privacy:amatterofdemocracy.whydemocracyneedsprivacyand dataprotection. InternationalDataPrivacyLaw,6(3):222–229,2016.

[27] T.Boellstorff.Themetaverseisn’thereyet,butitalreadyhasalonghistory.Technical Report186083,TheConversation,Aug.2022.[Accessed:07November2022].

[28] J.Bonneau,C.Herley,P.C.v.Oorschot,andF.Stajano.Thequesttoreplacepasswords: Aframeworkforcomparativeevaluationofwebauthenticationschemes.In 2012IEEE SymposiumonSecurityandPrivacy,pages553–567,2012.

[29] D.Braue.Globalransomwaredamagecostspredictedtoexceed$265billion by2031. https://cybersecurityventures.com/global-ransomware-damage-costspredicted-to-reach-250-billion-usd-by-2031,Jun2022.[Accessed:07November 2022].

[30] R.Brown,V.Ta,D.Bienstock,G.Ackerman,andJ.Wolfram.Doesthislookinfected? AsummaryofAPT41targetingU.S.stategovernments. https://www.mandiant.com/ resources/blog/apt41-us-state-governments,2022.[Accessed:07November2022].

[31] M.A.S.BubukayrandM.A.Almaiah.Cybersecurityconcernsinsmart-phonesand applications:Asurvey.In 2021InternationalConferenceonInformationTechnology(ICIT), pages725–731,2021.

[32] B.Bulgurcu,H.Cavusoglu,andI.Benbasat.Informationsecuritypolicycompliance:An empiricalstudyofrationality-basedbeliefsandinformationsecurityawareness. MISQ., 34(3):523–548,sep2010.

[33] M.BULL.Ex-burglarswarnhomeownersofsocialmediapoststhatputpropertyatrisk ofabreak-in. https://www.express.co.uk/life-style/property/1559309/burglartips-hacks-social-media-posts-break-ins-property.Accessed:2022-11-119.

[34] E.Bursztein.InsidetheinfamousMiraiIOTBotnet:Aretrospectiveanalysis. https://blog.cloudflare.com/inside-mirai-the-infamous-iot-botnet-aretrospective-analysis,Sep2021.[Accessed:07November2022].

[35] A.Cavoukianetal.Privacybydesign:The7foundationalprinciples. Informationand privacycommissionerofOntario,Canada,5:2009,2009.

[36] CC.Commoncriteriaforinformationtechnologysecurityevaluation. https://www. google.com/search?client=safari&rls=en&q=common+criteria&ie=UTF-8&oe=UTF-8. [Accessed:07November2022].

[37] D.ChampagneandR.B.Lee.Scalablearchitecturalsupportfortrustedsoftware.In HPCA -162010TheSixteenthInternationalSymposiumonHigh-PerformanceComputerArchitecture, pages1–12,2010.

[38] H.ChangandR.Shokri.Ontheprivacyrisksofalgorithmicfairness.In 2021IEEE EuropeanSymposiumonSecurityandPrivacy(EuroS&P),pages292–303,2021.

[39] S.Chaudhary,V.Gkioulos,andS.Katsikas.Developingmetricstoassesstheeffectiveness ofcybersecurityawarenessprogram. JournalofCybersecurity,8(1),052022.tyac006.

[40] G.Cheng,P.Zhou,andJ.Han.Learningrotation-invariantconvolutionalneuralnetworks forobjectdetectioninvhropticalremotesensingimages. IEEETransactionsonGeoscience andRemoteSensing,54(12):7405–7415,2016.

[41] R.ChoudhryandK.Garg.Ahybridmachinelearningsystemforstockmarketforecasting. WorldAcademyofScience,EngineeringandTechnology,39,012008.

[42] N.Chouliaras,G.Kittes,I.Kantzavelou,L.Maglaras,G.Pantziou,andM.A.Ferrag. Cyberrangesandtestbedsforeducation,training,andresearch. AppliedSciences,11(4), 2021.

[43] N.ChowdhuryandV.Gkioulos.Cybersecuritytrainingforcriticalinfrastructureprotection:Aliteraturereview. ComputerScienceReview,40:100361,2021.

[44] E.Commission.Commissionworkprogramme2023. https://ec.europa.eu/info/ sites/default/files/cwp_2023.pdf.[Accessed:07November2022].

[45] E.Commission.Cybersecuritypolicies. https://digital-strategy.ec.europa.eu/en/ policies/cybersecurity-policies.[Accessed:07November2022].

[46] E.Commission.Thedigitalservicesactpackage. https://digital-strategy.ec. europa.eu/en/policies/digital-services-act-package.[Accessed:07November 2022].

[47] E.Commission.Directive(eu)2016/1148oftheeuropeanparliamentandofthe councilof6july2016concerningmeasuresforahighcommonlevelofsecurityofnetworkandinformationsystemsacrosstheunion(NIS). = https://eurlex.europa.eu/eli/dir/2016/1148/oj.[Accessed:07November2022].

[48] E.Commission.Europeandatagovernanceact. https://digital-strategy.ec.europa. eu/en/policies/data-act.[Accessed:07November2022].

[49] E.Commission.Generaldataprotectionregulation. https://eur-lex.europa.eu/eli/ reg/2016/679/oj.[Accessed:07November2022].

[50] E.Commission.People,technologies&infrastructure–europe’splantothriveinthe metaverse. https://ec.europa.eu/commission/presscorner/detail/en/STATEMENT_ 22_5525.[Accessed:07November2022].

[51] ConferenceoftheIndependentDataProtectionSupervisoryAuthoritiesoftheFederationandtheLänder.Thestandarddataprotectionmodel–amethodfordataprotectionadvisingandcontrollingonthebasisofuniformprotectiongoals,version2.0b(englishversion). https://www.datenschutzzentrum.de/uploads/sdm/SDM-Methodology_ V2.0b.pdf,2020.[Accessed:07November2022].

[52] K.CongerandK.Roose.Uberinvestigatingbreachofitscomputersystems. https: //www.nytimes.com/2022/09/15/technology/uber-hacking-breach.html,Sep2022. [Accessed:07November2022].

[53] M.Corporation.Commonvulnerabilitiesandexposures(cve)details:Theultimatesecurityvulnerabilitydatasource. https://www.cvedetails.com/browse-by-date.php.[Accessed:20November2022].

[54] V.Costan,I.Lebedev,andS.Devadas.Sanctum:Minimalhardwareextensionsforstrong softwareisolation.In 25thUSENIXSecuritySymposium(USENIXSecurity16),pages857–874,Austin,TX,Aug.2016.USENIXAssociation.

[55] CyberSec4Europe.Flagship2:Thesuccessfulsecondcybersecurityexercisehostedbycybersec4europe. https://cybersec4europe.eu/flagship-2-the-successful-secondcybersecurity-exercise-hosted-by-cybersec4europe/,032022.[Accessed:07 November2022].

[56] CyBOK.Thecybersecuritybodyofknowledge. https://www.cybok.org.[Accessed:07 November2022].

[57] A.DaVeigaandN.Martins.Informationsecuritycultureandinformationprotection culture:Avalidatedassessmentinstrument. ComputerLaw&SecurityReview,31(2):243–256,2015.

[58] S.DangeandM.Chatterjee.Iotbotnet:Thelargestthreattotheiotnetwork. Advancesin IntelligentSystemsandComputing,1049:137–157,2020.

[59] T.DeMarco. StructuredAnalysisandSystemSpecification.YourdonPress,1979.

[60] M.Deng,K.Wuyts,R.Scandariato,B.Preneel,andW.Joosen.Aprivacythreatanalysis framework:Supportingtheelicitationandfulfillmentofprivacyrequirements. Requir. Eng.,16(1):3–32,mar2011.

[61] E.W.Dijkstraetal.Notesonstructuredprogramming. Section3OnTheReliabilityof Mechanisms,corollaryattheend,1970.

[62] R.Dingledine,N.Mathewson,andP.Syverson.Tor:Thesecond-generationonionrouter. In Proceedingsofthe13thConferenceonUSENIXSecuritySymposium-Volume13,SSYM’04, page21,USA,2004.USENIXAssociation.

[63] A.Dionysiou,M.Agathocleous,C.Christodoulou,andV.Promponas.Convolutionalneuralnetworksincombinationwithsupportvectormachinesforcomplexsequentialdata classification.InV.K˚urková,Y.Manolopoulos,B.Hammer,L.Iliadis,andI.Maglogiannis,editors, ArtificialNeuralNetworksandMachineLearning–ICANN2018,pages444–455, Cham,2018.SpringerInternationalPublishing.

[64] C.Directive.Councildirective2008/114/ecof8december2008–ontheidentificationand designationofeuropeancriticalinfrastructuresandtheassessmentoftheneedtoimprove theirprotection. OfficialJournaloftheEuropeanUnion.L,345:75–82,2008.

[65] J.Drees.Softwarebuginnewjerseyhospital’svaccineschedulingsystemcauses thousandsofduplicateappointments. https://www.beckershospitalreview.com/ healthcare-information-technology/software-bug-in-new-jersey-hospital-svaccine-scheduling-system-causes-thousands-of-duplicate-appointments.html Accessed:2022-11-119.

[66] Z.Durumeric,E.Wustrow,andJ.A.Halderman. {ZMap}:Fastinternet-widescanning anditssecurityapplications.In 22ndUSENIXSecuritySymposium(USENIXSecurity13), pages605–620,Washington,D.C.,2013.USENIXAssociation.

[67] Y.K.Dwivedi,L.Hughes,A.M.Baabdullah,S.Ribeiro-Navarrete,M.Giannakis,M.M. Al-Debei,D.Dennehy,B.Metri,D.Buhalis,C.M.Cheung,K.Conboy,R.Doyle,R.Dubey, V.Dutot,R.Felix,D.Goyal,A.Gustafsson,C.Hinsch,I.Jebabli,M.Janssen,Y.-G.Kim, J.Kim,S.Koos,D.Kreps,N.Kshetri,V.Kumar,K.-B.Ooi,S.Papagiannidis,I.O.Pappas,A.Polyviou,S.-M.Park,N.Pandey,M.M.Queiroz,R.Raman,P.A.Rauschnabel, A.Shirish,M.Sigala,K.Spanaki,G.Wei-HanTan,M.K.Tiwari,G.Viglia,andS.F. Wamba.Metaversebeyondthehype:Multidisciplinaryperspectivesonemergingchallenges,opportunities,andagendaforresearch,practiceandpolicy. InternationalJournalof InformationManagement,66:102542,2022.

[68] C.Dwork.Differentialprivacy:Asurveyofresults.InM.Agrawal,D.Du,Z.Duan, andA.Li,editors, TheoryandApplicationsofModelsofComputation,pages1–19,Berlin, Heidelberg,2008.SpringerBerlinHeidelberg.

[69] C.DworkandD.K.Mulligan.It’snotprivacy,andit’snotfair. Stan.L.Rev.Online,66:35, 2013.

[70] Enarx. https://enarx.dev/.[Accessed:07November2022].

[71] ENISA.Securityeconomicsandtheinternalmarket. https://www.enisa.europa.eu/ publications/archive/economics-sec.[Accessed:07November2022].

[72] ENISA.Understandingtheincreaseinsupplychainsecurityattacks. https://www.enisa.europa.eu/news/enisa-news/understanding-the-increase-insupply-chain-security-attacks.[Accessed:07November2022].

[73] ENISA.Artificialintelligencecybersecuritychallenges. EuropeanUnionAgencyforCybersecurity(ENISA),Aug2021.

[74] ENISA.Cybersecuritychallengesintheuptakeofartificialintelligenceinautonomous driving. EuropeanUnionAgencyforCybersecurity(ENISA),Aug2021.

[75] ENISA.ENISAthreatlandscapeforsupplychainattacks. EuropeanUnionAgencyfor Cybersecurity(ENISA),2021.

[76] ENISA.Securingmachinelearningalgorithms. EuropeanUnionAgencyforCybersecurity (ENISA),Dec2021.

[77] ENISA.Tipsforsecureuserauthentication,Aug2021.

[78] ETSI.Etsien303645. https://www.etsi.org/deliver/etsi_en/303600_303699/ 303645/02.01.00_30/en_303645v020100v.pdf.[Accessed:07November2022].

[79] EuropeanCommission.Cyberresilienceact. https://digital-strategy.ec.europa. eu/en/library/cyber-resilience-act.Accessed:2022-11-119.

[80] Europol.World’smostdangerousmalwareemotetdisruptedthroughglobalaction. https://www.europol.europa.eu/media-press/newsroom/news/world%E2%80% 99s-most-dangerous-malware-emotet-disrupted-through-global-action,2021.

[81] Eurostat.Individuals-internetactivities. https://ec.europa.eu/eurostat/ databrowser/view/isoc_ci_ac_i/default/table?lang=en.Accessed:2022-11-119.

[82] S.Fischer-Hübner,C.Alcaraz,A.Ferreira,C.Fernandez-Gago,J.Lopez,E.Markatos, L.Islami,andM.Akil.Stakeholderperspectivesandrequirementsoncybersecurityin europe. JournalofInformationSecurityandApplications,61:102916,2021.

[83] D.FlorencioandC.Herley.Alarge-scalestudyofwebpasswordhabits.In Proceedingsof the16thInternationalConferenceonWorldWideWeb,WWW’07,page657–666,NewYork, NY,USA,2007.AssociationforComputingMachinery.

[84] T.W.E.Forum.Definingandbuildingthemetaverse.Technicalreport,weforum.org,Jan. 2022.[Accessed28-Sep-2022].

[85] M.Fredrikson,S.Jha,andT.Ristenpart.Modelinversionattacksthatexploitconfidence informationandbasiccountermeasures.In Proceedingsofthe22ndACMSIGSACConference onComputerandCommunicationsSecurity,CCS’15,page1322–1333,NewYork,NY,USA, 2015.AssociationforComputingMachinery.

[86] FreedomofthePressFoundation.SecureDropThreatModel. https://docs. securedrop.org/en/stable/threat_model/threat_model.html,2022.[Accessed:07 November2022].

[87] J.FreundandJ.Jones. MeasuringandManagingInformationRisk:AFAIRApproach Butterworth-Heinemann,2014.

[88] J.E.Gaffney.Estimatingthenumberoffaultsincode. IEEETransactionsonSoftware Engineering,SE-10(4):459–464,1984.

[89] T.Gagliardoni.Thepolynetworkhackexplained. https://research. kudelskisecurity.com/2021/08/12/the-poly-network-hack-explained.[Accessed: 07November2022].

[90] B.GardnerandV.Thomas. BuildinganInformationSecurityAwarenessProgram:Defending AgainstSocialEngineeringandTechnicalThreats.SyngressPublishing,1stedition,2014.

[91] V.Garousi,A.Rainer,P.Lauvås,andA.Arcuri.Software-testingeducation:Asystematic literaturemapping. JournalofSystemsandSoftware,165:110570,2020.

[92] S.Gatlan.Chinesehackersusenewwindowsmalwaretobackdoorgovt,defenseorgs. https://www.bleepingcomputer.com/news/security/chinese-hackersuse-new-windows-malware-to-backdoor-govt-defense-orgs/,Aug2022.[Accessed: 07November2022].

[93] GDPR.GeneralDataProtectionRegulation. https://gdpr-info.eu.[Accessed:07 November2022].

[94] T.Geppert,S.Deml,D.Sturzenegger,andN.Ebert.Trustedexecutionenvironments: Applicationsandorganizationalchallenges. FrontiersinComputerScience,4,2022.

[95] S.Gilbert.Thepoliticaleconomyofthemetaverse.Technicalreport,Briefingsdel’IFRI, IFRI,June2022.[Accessed:07November2022].

[96] I.Goodfellow,J.Shlens,andC.Szegedy.Explainingandharnessingadversarialexamples. arXiv1412.6572,122014.

[97] Google.Asylo. https://asylo.dev/.[Accessed:17November2022].

[98] gramine.Gramine. https://gramineproject.io/.[Accessed:07November2022].

[99] A.Graves,A.-r.Mohamed,andG.Hinton.Speechrecognitionwithdeeprecurrentneural networks.In 2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing, pages6645–6649,2013.

[100] A.Greenberg.Hackersremotelykillajeeponthehighway—withmeinit. https: //www.wired.com/2015/07/hackers-remotely-kill-jeep-highway,72015.[Accessed: 07November2022].

[101] L.Grindstaff.Throughyourmind’seye:Whatbiasesareimpactingyoursecurityposture? https://www.mcafee.com/blogs/other-blogs/executive-perspectives/throughyour-minds-eye-what-biases-are-impacting-your-security-posture/,052021. [Accessed:07November2022].

[102] S.GürsesandJ.M.DelAlamo.Privacyengineering:Shapinganemergingfieldofresearch andpractice. IEEESecurity&Privacy,14(2):40–46,2016.

[103] I.Hadar,T.Hasson,O.Ayalon,E.Toch,M.Birnhack,S.Sherman,andA.Balissa.Privacy bydesigners:Softwaredevelopers’privacymindset. EmpiricalSoftw.Engg.,23(1):259–289, feb2018.

[104] M.Hansen,M.Jensen,andM.Rost.Protectiongoalsforprivacyengineering.In Proceedingsofthe2015IEEESecurityandPrivacyWorkshops,SPW’15,page159–166,USA,2015. IEEEComputerSociety.

[105] A.Harish.Whennasalostaspacecraftduetoametricmathmistake. https://www. simscale.com/blog/nasa-mars-climate-orbiter-metric/.Accessed:2022-11-119.

[106] M.Hasan.Numberofconnectediotdevicesgrowing18%to14.4billionglobally. https://iot-analytics.com/number-connected-iot-devices/,52022.[Accessed:07 November2022].

[107] N.Hasan,A.Chamoli,andM.Alam.Privacychallengesandtheirsolutionsiniot. Internet ofThings(IoT):ConceptsandApplications,pages219–231,12020.

[108] J.Haworth.Zero-dayflawsiniotbabymonitorscouldgiveattackersaccessto camerafeeds. https://portswigger.net/daily-swig/zero-day-flaws-in-iot-babymonitors-could-give-attackers-access-to-camera-feeds,92021.[Accessed:07 November2022].

[109] J.L.Hernández-Ramos,G.Baldini,S.N.Matheu,andA.Skarmeta.Updatingiotdevices:challengesandpotentialapproaches. GIoTS2020-GlobalInternetofThingsSummit, Proceedings,pages1–5,2020.

[110] J.Hodges,J.Jones,M.B.Jones,A.Kumar,andE.Lundberg.Webauthentication:Anapi foraccessingpublickeycredentialslevel2.

[111] J.-H.Hoepman.PrivacyDesignStrategies.InN.Cuppens-Boulahia,F.Cuppens,S.Jajodia, A.AbouElKalam,andT.Sans,editors, ICTSystemsSecurityandPrivacyProtection,pages 446–459,Berlin,Heidelberg,2014.SpringerBerlinHeidelberg.

[112] M.HowardandS.Lipner. TheSecurityDevelopmentLifecycle.MicrosoftPress,2006.

[113] T.Hunt.Pwnedwebsites.

[114] F.Hussain,R.Hussain,S.A.Hassan,andE.Hossain.Machinelearninginiotsecurity:Currentsolutionsandfuturechallenges. IEEECommunicationsSurveys&Tutorials, 22(3):1686–1721,2020.

[115] iSCOOP.Iotsecurityandtheconsumer:thechallengesandeducationquestion. = https://www.i-scoop.eu/iot-security-consumer-education/.[Accessed:07November 2022].

[116] IBM.Costofadatabreachreport2022. https://newsroom.ibm.com/2022-07-27-IBMReport-Consumers-Pay-the-Price-as-Data-Breach-Costs-Reach-All-Time-High, July2022.[Accessed:07November2022].

[117] J.Inclan.Emotetexposed:Alookinsidethecybercriminalsupplychain. https://blogs.vmware.com/security/2022/10/emotet-exposed-a-look-insidethe-cybercriminal-supply-chain.html,2022.

[118] Inmarsat.Industrialiotinthetimeofcovid-19. https://www.inmarsat.com/en/ insights/enterprise/2021/research-programme-2021-industrial-iot-covid19.html,2021.[Accessed:07November2022].

[119] Intel.In IntelSoftwareGuardExtensionsProgrammingReference.ARM,October2014.[Accessed:16November2022].

[120] IriusRisk.IriusRisk. https://www.iriusrisk.com/,2022.[Accessed:07November2022].

[121] L.Islami,S.Fischer-Hübner,andP.Papadimitratos.Capturingdrivers’privacypreferencesforintelligenttransportationsystems:Aninterculturalperspective. Computers& Security,123:102913,2022.

[122] L.H.Iwaya,G.H.Iwaya,S.Fischer-Hübner,andA.V.Steil.Organisationalprivacyculture andclimate:Ascopingreview. IEEEAccess,10:73907–73930,2022.

[123] J.Jalkanen. IsHumantheWeakestLinkinInformationSecurity?SystematicLiteratureReview. UniversityofJyväskylä,Jyväskylä,Finland,2019.

[124] S.V.Joshi,D.Stubbe,S.-T.T.Li,andD.M.Hilty.Theuseoftechnologybyyouth: Implicationsforpsychiatriceducators. AcademicPsychiatry,43(1):101–109,2019.

[125] D.Kaplan,J.Powell,andT.Woller.In AMDMemoryEncryption.AMD,April2016.[Accessed:16November2022].

[126] G.KarantzasandC.Patsakis.Anempiricalassessmentofendpointdetectionandresponsesystemsagainstadvancedpersistentthreatsattackvectors. JournalofCybersecurity andPrivacy,1(3):387–421,2021.

[127] T.Karras,S.Laine,andT.Aila.Astyle-basedgeneratorarchitectureforgenerativeadversarialnetworks.In 2019IEEE/CVFConferenceonComputerVisionandPatternRecognition (CVPR),pages4396–4405,2019.

[128] Kaspersky.Thehumanfactorinitsecurity:Howemployeesaremakingbusinessesvulnerablefromwithin. https://www.kaspersky.com/blog/the-human-factor-in-itsecurity/,112022.[Accessed:07November2022].

[129] N.KohlandP.Stone.Policygradientreinforcementlearningforfastquadrupedallocomotion.In IEEEInternationalConferenceonRoboticsandAutomation,2004.Proceedings. ICRA’04.2004,volume3,pages2619–2624Vol.3,2004.

[130] L.KohnfelderandP.Garg.Thethreatstoourproducts. MicrosoftInterface,Microsoft Corporation,33,1999.

[131] I.Kononenko.Machinelearningformedicaldiagnosis:history,stateoftheartandperspective. ArtificialIntelligenceinMedicine,23(1):89–109,2001.

[132] V.KoutsokostasandC.Patsakis.Pythonandmalware:Developingstealthandevasive malwarewithoutobfuscation.InS.D.C.diVimercatiandP.Samarati,editors, Proceedings ofthe18thInternationalConferenceonSecurityandCryptography,SECRYPT2021,July6-8, 2021,pages125–136.SCITEPRESS,2021.

[133] A.Küchler,A.Mantovani,Y.Han,L.Bilge,andD.Balzarotti.Doeseverysecondcount? time-basedevolutionofmalwarebehaviorinsandboxes.In 28thAnnualNetworkandDistributedSystemSecuritySymposium,NDSS2021,virtually,February21-25,2021.TheInternet Society,2021.

[134] I.Kuzminykh,B.Ghita,andJ.M.Such.Thechallengeswithinternetofthingsforbusiness. https://arxiv.org/abs/2012.03589,122020.[Accessed:07November2022].

[135] A.L.Lafuente,F.Nielson,S.Mödersheim,A.Schlichtkrull,A.Sforzin,C.Sorientea, L.Kamm,R.Martins,J.Soares,L.Antunes,L.Durante,M.Cheminod,E.Athanasopoulos,B.Hamid,A.Omerovic,K.Bernsmed,andR.S.PerHMeland.Researchchallenges andrequirementsforsecuresoftwaredevelopment. https://cybersec4europe.eu/wpcontent/uploads/2020/09/CS4E-D3.9-Research-challenges-and-requirementsfor-secure-software-development-v1.1-Submitted.pdf,2020.[Accessed:07 November2022].

[136] T.LAMBERTandB.DONOHUE.It’sallfunandgamesuntilransomwaredeletes theshadowcopies. https://redcanary.com/blog/its-all-fun-and-games-untilransomware-deletes-the-shadow-copies,2022.[Accessed:07November2022].

[137] R.Langner.Stuxnet:Dissectingacyberwarfareweapon. IEEESecurity&Privacy,9(3):49–51,2011.

[138] D.Lee,D.Kohlbrenner,S.Shinde,K.Asanovi´c,andD.Song.Keystone:Anopenframeworkforarchitectingtrustedexecutionenvironments.In ProceedingsoftheFifteenthEuropeanConferenceonComputerSystems,EuroSys’20,NewYork,NY,USA,2020.Association forComputingMachinery.

[139] K.LeFevre,D.DeWitt,andR.Ramakrishnan.Mondrianmultidimensionalk-anonymity. In 22ndInternationalConferenceonDataEngineering(ICDE’06),pages25–25,2006.

[140] N.G.Leveson.Thetherac-25:30yearslater. Computer,50(11):8–11,2017.

[141] M.N.Lintvedt.Puttingapriceondataprotectioninfringement. InternationalDataPrivacy Law,12(1):1–15,122021.

[142] B.Liu,M.Ding,S.Shaham,W.Rahayu,F.Farokhi,andZ.Lin.Whenmachinelearning meetsprivacy:Asurveyandoutlook. ACMComput.Surv.,54(2),mar2021.

[143] J.Lopez,C.Alcaraz,andR.Roman.Smartcontrolofoperationalthreatsincontrolsubstations. Computers&Security,38:14–27,2013.CybercrimeintheDigitalEconomy.

[144] P.Lorenzo,F.Stefano,A.Ferreira,andP.Carolina.Artificialintelligenceandcybersecurity:Technology,governanceandpolicychallenges. https://www.ceps.eu/wp-content/ uploads/2021/05/CEPS-TFR-Artificial-Intelligence-and-Cybersecurity.pdf, 2021.[Accessed:07November2022].

[145] T.Madiega,P.Car,andM.N.withLouiseVandePol.Metaverse:Opportunities,risks andpolicyimplications.TechnicalReportPE733.557,EuropeanParliamentaryResearch Service,June2022.[Accessed:07November2022].

[146] S.Maes,K.Tuyls,B.Vanschoenwinkel,andB.Manderick.Creditcardfrauddetection usingbayesianandneuralnetworks.In Proceedingsofthe1stinternationalnaisocongresson neurofuzzytechnologies,pages261–270,082002.

[147] K.ManheimandL.Kaplan.Artificialintelligence:Riskstoprivacyanddemocracy. Yale JL&Tech.,21:106,2019.

[148] K.Marky,K.Ragozin,G.Chernyshov,A.Matviienko,M.Schmitz,M.Mühlhäuser, C.Eghtebas,andK.Kunze.“nah,it’sjustannoying!”adeepdiveintouserperceptions oftwo-factorauthentication. ACMTransactionsonComputer-HumanInteraction,29(5),oct 2022.

[149] G.McGraw. SoftwareSecurity:BuildingSecurityIn.Addison-WesleyProfessional,2006.

[150] metaversestandards.org.Themetaversestandardsforum.Technicalreport,metaversestandards.org,June2022.[Accessed:07November2022].

[151] MicroAge.Thebenefitsofcybersecurityawarenesstraining. https://microage.ca/thebenefits-of-cybersecurity-awareness-training/,102022.[Accessed:07November 2022].

[152] Microsoft.Createandusestrongpasswords—support.microsoft.com. https://support.microsoft.com/en-us/windows/create-and-use-strongpasswords-c5cebb49-8c53-4f5e-2bc4-fe357ca048eb.[Accessed:07November 2022].

[153] Microsoft.Microsoftsdl. https://www.microsoft.com/en-us/securityengineering/ sdl/practices.[Accessed:07November2022].

[154] Microsoft.Applicationsforartificialintelligenceindepartmentofdefensecyber missions. https://blogs.microsoft.com/on-the-issues/2022/05/03/artificialintelligence-department-of-defense-cyber-missions/,052022.[Accessed:07 November2022].

[155] Microsoft.SpecialReport:Ukraine:AnoverviewofRussia’scyberattackactivityin Ukraine. https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4Vwwd, 2022.[Accessed:07November2022].

[156] MicrosoftCorporation.MicrosoftThreatModelingTool7. https://aka.ms/ threatmodelingtool,2022.[Accessed:07November2022].

[157] J.Miranda,N.Mäkitalo,J.Garcia-Alonso,J.Berrocal,T.Mikkonen,C.Canal,andJ.M. Murillo.Fromtheinternetofthingstotheinternetofpeople. IEEEInternetComputing, 19(2):40–47,2015.

[158] H.Modi.Netscoutthreatintelligencereport. https://www.netscout.com/ sites/default/files/2019-02/SECR_001_EN-1901%20-%20NETSCOUT%20Threat% 20Intelligence%20Report%202H%202018.pdf,2018.[Accessed:07November2022].

[159] Movie.Readyplayerone,2018.[Accessed:07November2022].

[160] P.Muncaster.Hotelguestslockedoutofroomsafterransomwareattack. https://www. infosecurity-magazine.com/news/hotel-guests-locked-out-rooms,Dec2021.[Accessed:07November2022].

[161] S.Muppidi,L.Fisher,andG.Parham.Aiandautomationforcybersecurity.Technical report,IBMCorporation,June2022.[Accessed07-November-2022].

[162] A.S.Namin,Z.Aguirre-Muñoz,andK.S.Jones.Teachingcybersecuritythroughcompetition:Anexperiencereportaboutaparticipatorytrainingworkshop.In InternationalConferenceonComputerScienceEducationInnovation&Technology(CSEIT).Proceedings,page98. GlobalScienceandTechnologyForum,2016.

[163] NationalHighwayTrafficSafetyAdministration.Part573safetyrecallreport. https: //static.nhtsa.gov/odi/rcl/2021/RCLRPT-21V035-4682.PDF.Accessed:2022-11-119.

[164] T.Ncubukezi.Humanerrors:Acybersecurityconcernandtheweakestlinktosmall businesses. InternationalConferenceonCyberWarfareandSecurity,17:395–403,032022.

[165] L.H.Newman.Millionsofwebcameraandbabymonitorfeedsareexposed. https: //www.wired.com/story/kalay-iot-bug-video-feeds/,2017.[Accessed:07November 2022].

[166] Z.Ning,F.Zhang,W.Shi,andW.Shi.Positionpaper:Challengestowardssecuring hardware-assistedexecutionenvironments.In ProceedingsoftheHardwareandArchitectural SupportforSecurityandPrivacy,HASP’17,NewYork,NY,USA,2017.Associationfor ComputingMachinery.

[167] NIST.Securesoftwaredevelopmentframework. https://nvlpubs.nist.gov/nistpubs/ SpecialPublications/NIST.SP.800-218.pdf.[Accessed:07November2022].

[168] NIST.Nistannouncesfirstfourquantum-resistantcryptographicalgorithms. https://www.nist.gov/news-events/news/2022/07/nist-announces-first-fourquantum-resistant-cryptographic-algorithms,July2022.[Accessed:07November 2022].

[169] C.Nobata,J.Tetreault,A.Thomas,Y.Mehdad,andY.Chang.Abusivelanguagedetection inonlineusercontent.In Proceedingsofthe25thInternationalConferenceonWorldWideWeb, WWW’16,page145–153,RepublicandCantonofGeneva,CHE,2016.InternationalWorld WideWebConferencesSteeringCommittee.

[170] C.Ntantogian,S.Malliaros,andC.Xenakis.Evaluationofpasswordhashingschemesin opensourcewebplatforms. Computers&Security,84:206–224,2019.

[171] M.Nunes,P.Burnap,P.Reinecke,andK.Lloyd.Baneorboon:Measuringtheeffectof evasivemalwareonsystemcallclassifiers. J.Inf.Secur.Appl.,67(C),jun2022.

[172] occlum.Occlum. https://occlum.io/.[Accessed:07November2022].

[173] N.I.ofStandardsandTechnology.Digitalidentityguidelines:Authenticationandlifecyclemanagement.Technicalreport,U.S.DepartmentofCommerce,Washington,D.C., 2017.

[174] L.O’Gorman.Comparingpasswords,tokens,andbiometricsforuserauthentication. ProceedingsoftheIEEE,91(12):2021–2040,2003.

[175] OMIGroup.Openmetaverseinteroperabilitygroup.Technicalreport,OMIGroup,Sept. 2022.[Accessed:07November2022].

[176] P.H.O’Neill.Ransomwaredidnotkillagermanhospitalpatient. https: //www.technologyreview.com/2020/11/12/1012015/ransomware-did-not-killa-german-hospital-patient,Nov2020.[Accessed:07November2022].

[177] opentitan.Opentitan. https://opentitan.org/.[Accessed:07November2022].

[178] C.Osborne.Miraisplinterbotnetsdominateiotattackscene. = https://www.zdnet.com/article/mirai-splinter-botnets-dominate-iot-attack-scene/, 12022.[Accessed:07November2022].

[179] M.Ovelgönne,T.Dumitra¸s,B.A.Prakash,V.S.Subrahmanian,andB.Wang.Understandingtherelationshipbetweenhumanbehaviorandsusceptibilitytocyberattacks:A data-drivenapproach. ACMTrans.Intell.Syst.Technol.,8(4),mar2017.

[180] OWASP.OWASPtop10-2021. https://owasp.org/Top10/,2021.[Accessed:07November2022].

[181] E.Papadogiannakis,P.Papadopoulos,N.Kourtellis,andE.P.Markatos.Usertrackingin thepost-cookieera:Howwebsitesbypassgdprconsenttotrackusers.In Proceedingsofthe WebConference2021,WWW’21,page2130–2141,NewYork,NY,USA,2021.Association forComputingMachinery.

[182] N.Papernot,P.McDaniel,A.Sinha,andM.P.Wellman.Sok:Securityandprivacyin machinelearning.In 2018IEEEEuropeanSymposiumonSecurityandPrivacy(EuroS&P), pages399–414,2018.

[183] M.Paquet-Clouston,M.Romiti,B.Haslhofer,andT.Charvat.Spamsmeetcryptocurrencies:Sextortioninthebitcoinecosystem.In Proceedingsofthe1stACMConferenceon AdvancesinFinancialTechnologies,AFT’19,page76–88,NewYork,NY,USA,2019.AssociationforComputingMachinery.

[184] C.PatsakisandF.Casino.Hydrasandipfs:adecentralisedplaygroundformalware. InternationalJournalofInformationSecurity,18(6):787–799,2019.

[185] C.PatsakisandA.Chrysanthou.Analysingthefall2020emotetcampaign. arXivpreprint arXiv:2011.06479,2020.

[186] S.Peldszus,K.Tuma,D.Strüber,J.Jürjens,andR.Scandariato.Securedata-flowcompliancechecksbetweenmodelsandcodebasedonautomatedmappings.In 2019ACM/IEEE 22ndInternationalConferenceonModelDrivenEngineeringLanguagesandSystems(MODELS),pages23–33,2019.

[187] R.P.Pires.Distributedsystemsandtrustedexecutionenvironments:Trade-offsand challenges. https://arxiv.org/pdf/2001.09670.pdf,December2019.[Accessed:20 November2022].

[188] S.Pletinckx,C.Trap,andC.Doerr.Malwarecoordinationusingtheblockchain:An analysisofthecerberransomware.In 2018IEEEConferenceonCommunicationsandNetwork Security(CNS),pages1–9,2018.

[189] N.Popper.Knightcapitalsaystradingglitchcostit$440million. https: //archive.nytimes.com/dealbook.nytimes.com/2012/08/02/knight-capitalsays-trading-mishap-cost-it-440-million/.Accessed:2022-11-119.

[190] W.PresthusandK.F.Sønslien.Ananalysisofviolationsandsanctionsfollowingthe gdpr. InternationalJournalofInformationSystemsandProjectManagement,9(1):38–53,Sep 2021.

[191] pwc.ConticyberattackontheHSE. https://www.hse.ie/eng/services/ publications/conti-cyber-attack-on-the-hse-full-report.pdf,2021.[Accessed: 07November2022].

[192] D.Reading.Populariotcamerasneedpatchingtofendoffcatastrophic attacks. https://www.darkreading.com/attacks-breaches/popular-iot-cameraspatching-catastrophic-attacks,92022.[Accessed:07November2022].

[193] D.Rehak,P.Senovsky,M.Hromada,andT.Lovecek.Complexapproachtoassessing resilienceofcriticalinfrastructureelements. Internationaljournalofcriticalinfrastructure protection,25:125–138,2019.

[194] Reuters.AXAdivisioninasiahitbyransomwarecyberattack. https://www.reuters. com/article/us-axa-cyber-idUSKCN2CX0B0,May2021.[Accessed:07November2022].

[195] Reuters.Danishtrainstandstillonsaturdaycausedbycyberattack. https: //www.reuters.com/technology/danish-train-standstill-saturday-caused-bycyber-attack-2022-11-03/,Nov2022.[Accessed:07November2022].

[196] S.Rinaldi,J.Peerenboom,andT.Kelly.Identifying,understanding,andanalyzingcritical infrastructureinterdependencies. IEEEControlSystemsMagazine,21(6):11–25,2001.

[197] riscure.Securitypitfallsinteedevelopment. https://www.riscure.com/publication/ security-pitfalls-in-tee-development/.[Accessed:07November2022].

[198] A.ROBERTSON.Mostkillersstalktheirvictimsonsocialmediabeforemurderingthem,saycriminologists. https://www.dailymail.co.uk/news/article-4439130/ Most-killers-stalk-victims-social-media-murder.html.Accessed:2022-11-119.

[199] A.Rowe.StudyRevealsAveragePersonHas100Passwords|Tech.co—tech.co. https: //tech.co/password-managers/how-many-passwords-average-person,2021.[Accessed:07November2022].

[200] P.RuggieroandJ.Foote.Cyberthreatstomobilephones. https://www.cisa.gov/ uscert/sites/default/files/publications/cyber_threats_to_mobile_phones.pdf, 2011.[Accessed:07November2022].

[201] O.SAMM.Softwareassurancematuritymodel. https://owaspsamm.org/model/.[Accessed:07November2022].

[202] Sanket.Theexponentialcostoffixingbugs. https://deepsource.io/blog/ exponential-cost-of-fixing-bugs/.[Accessed:07November2022].

[203] I.H.Sarker,A.I.Khan,Y.B.Abushark,andF.Alsolami.Internetofthings(iot)security intelligence:Acomprehensiveoverview,machinelearningsolutionsandresearchdirections. MobileNetworksandApplications,1:1–17,32022.

[204] T.Schaberreiter,K.Kittilä,K.Halunen,J.Röning,andD.Khadraoui.Riskassessmentin criticalinfrastructuresecuritymodellingbasedondependencyanalysis.In International WorkshoponCriticalInformationInfrastructuresSecurity,pages213–217.Springer,2011.

[205] P.M.Schwartz.Privacyanddemocracyincyberspace. Vand.L.Rev.,52:1607,1999.

[206] T.Seals.Mozibotnetaccountsformajorityofiottraffic. = https://threatpost.com/mozibotnet-majority-iot-traffic/159337/,92020.[Accessed:07November2022].

[207] F.T.Security.Boostengagementwithseriousgametraining. https:// terranovasecurity.com/serious-game/,102022.[Accessed:07November2022].

[208] R.Setola,S.DePorcellinis,andM.Sforna.Criticalinfrastructuredependencyassessment usingtheinput–outputinoperabilitymodel. InternationalJournalofCriticalInfrastructure Protection,2(4):170–178,2009.

[209] R.Setola,V.Rosato,E.Kyriakides,andE.Rome. Managingthecomplexityofcriticalinfrastructures:Amodellingandsimulationapproach.SpringerNature,2016.

[210] M.Shahraeini,P.Kotzanikolaou,andM.Nasrolahi.Communicationresilienceforsmart gridsbasedondependencegraphsandeigenspectralanalysis. IEEESystemsJournal,pages 1–11,2022.

[211] R.Shokri,M.Stronati,C.Song,andV.Shmatikov.Membershipinferenceattacksagainst machinelearningmodels.In 2017IEEESymposiumonSecurityandPrivacy(SP),pages 3–18,2017.

[212] A.Shostack. ThreatModeling:DesigningforSecurity.JohnWiley&Sons,Indianapolis, Indiana,2014.

[213] J.Sigholm,G.Falco,andA.Viswanathan.Enhancingcybersecurityeducationthrough high-fidelityliveexercises(hiflix).In HICSS,012019.

[214] L.Sion,D.VanLanduyt,K.Yskout,S.Verreydt,andW.Joosen.Automatedthreatanalysis andmanagementinacontinuousintegrationpipeline.In 2021IEEESecureDevelopment Conference(SecDev),pages30–37,2021.

[215] L.Sion,K.Yskout,D.VanLanduyt,andW.Joosen.Risk-basedDesignSecurityAnalysis. In Proceedings-2018IEEE/ACMFirstInternationalWorkshoponSecurityAwarenessfrom DesigntoDeployment,SEAD2018,page11–18,NewYork,NY,USA,2018.Associationfor ComputingMachinery.

[216] J.Smart,N.Cascio,andJ.Paffendorf.Metaverseroadmap–pathwaystothe3d web:Across-industrypublicforesightproject. https://metaverseroadmap.org/ MetaverseRoadmapOverview.pdf.[Accessed:07November2022].

[217] I.Sommerville.Softwareengineering10. Harlow:PearsonEducationLimited,2016.

[218] Sophos.Thestateofransomware2022. https://assets.sophos.com/X24WTUEQ/at/ 4zpw59pnkpxxnhfhgj9bxgj9/sophos-state-of-ransomware-2022-wp.pdf,2022.[Accessed:07November2022].

[219] M.Sporny,D.Longley,M.Sabadello,D.Reed,O.Steele,andC.Allen.Decentralized identifiers(dids).

[220] M.Sporny,G.Noble,D.Longley,D.C.Burnett,B.Zundel,andK.D.Hartog.Verifiable credentialsdatamodel.

[221] R.Steen.5reasonsautomationcan’ttakeovercybersecurity. https: //www.securitymagazine.com/articles/98396-5-reasons-automation-canttake-over-cybersecurity,92022.[Accessed:07November2022].

[222] I.Stellios,P.Kotzanikolaou,M.Psarakis,C.Alcaraz,andJ.Lopez.Asurveyofiotenabledcyberattacks:Assessingattackpathstocriticalinfrastructuresandservices. IEEE CommunicationsSurveys&Tutorials,20(4):3453–3495,2018.

[223] N.Stephenson. SnowCrash.BantamBooks,UnitedStatesofAmerica,1992.

[224] G.Stergiopoulos,P.Kotzanikolaou,M.Theocharidou,G.Lykou,andD.Gritzalis.Timebasedcriticalinfrastructuredependencyanalysisforlarge-scaleandcross-sectoralfailures. InternationalJournalofCriticalInfrastructureProtection,12:46–60,2016.

[225] G.E.Suh,D.Clarke,B.Gassend,M.vanDijk,andS.Devadas.Aegis:Architecturefor tamper-evidentandtamper-resistantprocessing.In Proceedingsofthe17thAnnualInternationalConferenceonSupercomputing,ICS’03,page160–171,NewYork,NY,USA,2003. AssociationforComputingMachinery.

[226] L.Sweeney.K-anonymity:Amodelforprotectingprivacy. Int.J.Uncertain.Fuzziness Knowl.-BasedSyst.,10(5):557–570,oct2002.

[227] F.SwiderskiandW.Snyder. Threatmodeling.MicrosoftPress,2004.

[228] J.Taylor.Facebookoutage:whatwentwrongandwhydidittakesolongtofixaftersocialplatformwentdown? https://www.theguardian.com/technology/2021/oct/ 05/facebook-outage-what-went-wrong-and-why-did-it-take-so-long-to-fix.Accessed:2022-11-119.

[229] W.C.D.Team.Metaverseinteroperabilitycommunitygroup.Technicalreport,w3c.org, Sept.2022.[Accessed:07November2022].

[230] W.Technologies.Newresearch:Filelessmalwareattackssurgeby900% andcryptominersmakeacomeback,whileransomwareattacksdecline. https://www.watchguard.com/wgrd-about/press-releases/new-researchfileless-malware-attacks-surge-900-and-cryptominers-make,2021.[Accessed:07 November2022].

[231] B.Tekinerdogan.Architecturaldriftanalysisusingarchitecturereflexionviewpointand designstructurereflexionmatrices.In SoftwareQualityAssurance,pages221–236.Elsevier, 2016.

[232] P.Torr.Demystifyingthethreatmodelingprocess. IEEESecurity&Privacy,3(5):66–70, 2005.

[233] TrailofBits.KubernetesThreatModel. https://github.com/kubernetes/community/ raw/683ec8f8a392522933b8950a052dfdce6da6a812/sig-security/security-audit2019/findings/Kubernetes%20Threat%20Model.pdf,2019.[Accessed:07November 2022].

[234] S.Truex,L.Liu,M.E.Gursoy,L.Yu,andW.Wei.Demystifyingmembershipinferenceattacksinmachinelearningasaservice. IEEETransactionsonServicesComputing, 14(6):2073–2089,2021.

[235] K.Tuma,L.Sion,R.Scandariato,andK.Yskout.Automatingtheearlydetectionof securitydesignflaws.In Proceedingsofthe23rdACM/IEEEInternationalConferenceonModel DrivenEngineeringLanguagesandSystems,MODELS’20,page332–342,NewYork,NY, USA,2020.AssociationforComputingMachinery.

[236] W.TurtonandK.Mehrotra.Colonialpipelinecyberattack:Hackersusedcompromisedpassword. https://www.bloomberg.com/news/articles/2021-06-04/hackersbreached-colonial-pipeline-using-compromised-password,Jun2021.[Accessed:07 November2022].

[237] U.S.DepartmentofJustice.JusticeDepartmentAnnouncesCourt-AuthorizedDisruptionofBotnetControlledbytheRussianFederation’sMainIntelligenceDirectorate (GRU). https://www.justice.gov/opa/pr/justice-department-announces-courtauthorized-disruption-botnet-controlled-russian-federation,2022.

[238] L.S.Vailshery.Numberofinternetofthings(iot)connecteddevicesworldwidefrom 2019to2021,withforecastsfrom2022to2030. https://www.statista.com/statistics/ 1183457/iot-connected-devices-worldwide/,82022.[Accessed:07November2022].

[239] J.vanRest,D.Boonstra,M.Everts,M.vanRijn,andR.vanPaassen.Designingprivacyby-design.InB.PreneelandD.Ikonomou,editors, PrivacyTechnologiesandPolicy,pages 55–72,Berlin,Heidelberg,2014.SpringerBerlinHeidelberg.

[240] Verizon.Databreachinvestigationsreport. https://www.verizon.com/business/ resources/reports/dbir/,062022.[Accessed:07November2022].

[241] Versen.Manifestoonsoftwareresearchandeducationinthenetherlands. https:// www.versen.nl/assets/manifesto/digitalfolder.pdf,2020.[Accessed:07November 2022].

[242] N.Virvilis,D.Gritzalis,andT.Apostolopoulos.Trustedcomputingvs.advancedpersistentthreats:Canadefenderwinthisgame?In 2013IEEE10thInternationalConferenceon UbiquitousIntelligenceandComputingand2013IEEE10thInternationalConferenceonAutonomicandTrustedComputing,pages396–403,2013.

[243] S.Wang,X.Gu,S.Luan,andM.Zhao.Resilienceanalysisofinterdependentcritical infrastructuresystemsconsideringdeeplearningandnetworktheory. InternationalJournal ofCriticalInfrastructureProtection,35:100459,2021.

[244] Y.Wang,Z.Su,N.Zhang,R.Xing,D.Liu,T.H.Luan,andX.Shen.Asurveyonmetaverse: Fundamentals,security,andprivacy. IEEECommunicationsSurveysandTutorials,pages1–1,2022.

[245] W.Wei.Casinogetshackedthroughitsinternet-connectedfishtankthermometer. https: //thehackernews.com/2018/04/iot-hacking-thermometer.html,42018.[Accessed:07 November2022].

[246] R.Weiss,X.Mountrouidou,S.Watson,J.Mache,E.Hawthorne,andA.Chattopadhyay. Cybersecurityacrossalldisciplinesin2020.In Proceedingsofthe51stACMTechnicalSymposiumonComputerScienceEducation,SIGCSE’20,page1404,NewYork,NY,USA,2020. AssociationforComputingMachinery.

[247] E.Weyuker.Testingcomponent-basedsoftware:acautionarytale. IEEESoftware,15(5):54–59,1998.

[248] Wikipedia.Self-sovereignidentity. https://en.wikipedia.org/wiki/Self-sovereign_ identity.[Accessed:07November2022].

[249] M.WilsonandJ.Hash.Sp800-50.buildinganinformationtechnologysecurityawareness andtrainingprogram.Technicalreport,NationalInstituteofStandardsandTechnology, Gaithersburg,MD,USA,2003.

[250] J.WolffandN.Atallah.Earlygdprpenalties:Analysisofimplementationandfines throughmay2020. JournalofInformationPolicy,11(1):63–103,2021.

[251] N.Woolf.DDoSattackthatdisruptedinternetwaslargestofitskindinhistory, expertssay. https://www.theguardian.com/technology/2016/oct/26/ddos-attackdyn-mirai-botnet,Oct2016.[Accessed:07November2022].

[252] O.Yoachimik. https://blog.cloudflare.com/mantis-botnet/,2022.

[253] K.Yskout,T.Heyman,D.VanLanduyt,L.Sion,K.Wuyts,andW.Joosen.Threatmodeling:frominfancytomaturity.In ProceedingsoftheACM/IEEE42ndInternationalConference onSoftwareEngineering:NewIdeasandEmergingResults,pages9–12.ACM,jun2020.

[254] K.Yu,L.Tan,S.Mumtaz,S.Al-Rubaye,A.Al-Dulaimi,A.K.Bashir,andF.A.Khan. Securingcriticalinfrastructures:Deep-learning-basedthreatdetectioniniiot. IEEECommunicationsMagazine,59(10):76–82,2021.

[255] F.Zhang,P.P.K.Chan,B.Biggio,D.S.Yeung,andF.Roli.Adversarialfeatureselection againstevasionattacks. IEEETransactionsonCybernetics,46(3):766–777,2016.

[256] E.Zio.Challengesinthevulnerabilityandriskanalysisofcriticalinfrastructures. ReliabilityEngineering&SystemSafety,152:137–150,2016.

CyberSec4Europe is a research and innovation pilot project for the European Cybersecurity Competence Centre in Bucharest and the network of National Coordination Centres.

As a research project, CyberSec4Europe is working towards harmonising the journey from the development of software components that fit the requirements identified by a set of short- and long-term roadmaps, leading to a series of consequent recommendations. These are tied to the project’s real-world demonstration use cases that address cybersecurity challenges within the vertical sectors of digital infrastructure, finance, government and smart cities, healthcare and transportation.

CyberSec4Europe’s main objective is piloting the cybersecurity capabilities required to secure and maintain European democracy and the integrity of the Digital Single Market. CyberSec4Europe has translated this broad objective into measurable, concrete steps through a set of policy, technical and innovation objectives.

CyberSec4Europe is funded by the European Union under the H2020 Programme Grant Agreement No. 830929