OWASP LLM_AI Security Governance Checklist

RevisionHistory

Revision Date

Author(s) Description

0.1 2023-11-01 SandyDunn initialdraft

0.5 2023-12-06 SandyDunn, OWASPLLM

AppsTeam publicdraft

Version:0.5

Published:December6,2023

Theinformationprovidedinthisdocumentdoesnot,andisnotintendedto,constitutelegaladvice. Allinformationisforgeneralinformationalpurposesonly.

Thisdocumentcontainslinkstootherthird-partywebsites.Suchlinksareonlyforconvenience andOWASPdoesnotrecommendorendorsethecontentsofthethird-partysites.

Overview

Everyinternetuserandbusinessshouldpreparefortheimpactofasurgeinpowerfulgenerative artificialintelligence(GenAI)applications.GenAIholdsenormouspromiseandopportunitiesfor discovery,efficiency,anddrivingcorporategrowthacrossmanyindustriesanddisciplines.However, aswithanystrongnewtechnology,itintroducesnewchallengestosecurityandprivacy.

ArtificialIntelligence,MachineLearning,LargeLanguageModels,andDiffusionModelshavebeen indevelopmentandthefocusofacademicresearchformanyyears.Recentimprovementsin trainingdataavailability,computerpower,GenAIcapacity,andthereleaseofsolutionssuchas ChatGPT,ElevenLabs,Midjourney,alongwiththeirbroaderavailabilityoutsideofwhatpreviously wasarelativelyisolatedandspecializedfield,haveledtoitseruptivegrowth.Theseadvancesin artificialintelligence(AI)emphasizetheimportanceoforganizationsdevelopingplanstomanage theirengagementanduseofAIwithintheirorganization.

• Artificialintelligence isabroadtermthatencompassesallfieldsofcomputersciencethat enablemachinestoaccomplishtasksthatwouldnormallyrequirehumanintelligence.Machine learningandgenerativeAIaretwosubcategoriesofAI.

• Machinelearning isasubsetofAIthatfocusesoncreatingalgorithmsthatcanlearnfrom data.Machinelearningalgorithmsaretrainedonasetofdata,andthentheycanusethatdata tomakepredictionsordecisionsaboutnewdata.

• GenerativeAI isatypeofmachinelearningthatfocusesoncreatingnewdata.Often,GenAI reliesontheuseoflargelanguagemodelstoperformthetasksneededtocreatethenewdata.

• A largelanguagemodel(LLM) isatypeofAIprogramthatusesmachinelearningtoperform naturallanguageprocessing(NLP)tasks.LLMsaretrainedonlargedatasetstounderstand, summarize,generate,andpredictnewcontent.

ThediagrambelowshowstherelationshipofLLMtothefieldofAIgenerally:

Figure1.1:ImageofLLMrelationshipwithinthefieldofArtificialIntelligence

OrganizationswillfacenewchallengesdefendingandmanagingGenAIsolutions.Additionally,there issignificantpotentialforacceleratedthreatsfromthreatactorswhowilluseGenAItoaugment attacktechniques.

Manyapplicationswithinabusinessemployartificialintelligenceapplications,suchashuman resourcehiring,SPAMdetectionforemail,behavioralanalyticsforSIEM,andMDRapps.Theprimary focusofthisdocumentisonLargeLanguageModelapplications,whichcanproducecontent.

ResponsibleandTrustworthyArtificialIntelligence

AschallengesandbenefitsofArtificialIntelligenceemerge-andregulationsandlawsarepassedtheprinciplesandpillarsofresponsibleandtrustworthyAIusageareevolvingfromidealisticobjects andconcernstoestablishedstandards.

TheOWASPAISecurityandPrivacyGuideworkinggroupismonitoringthesechangesandaddressing thebroaderandmorechallengingconsiderationsforallaspectsofartificialintelligence.

Figure1.2:ImagecreditMontrealAIEthicsInstitute

WhoisThisFor?

Executive,technology,cybersecurity,privacy,compliance,andlegalleadersmustpaycloseattention tothefastGenAItechnologicaltransformationanddeviseastrategytobenefitfromopportunities whilefightingagainstthreatsandmanagingrisks.

Thischecklistisdesignedtoassistthesetechnologyandbusinessleadersinquicklyunderstanding therisksandbenefitsofusingLLM,allowingthemtofocusondevelopingacomprehensivelistof essentialareasandtasksrequiredtodefendandprotecttheorganizationastheycreateaLarge LanguageModelstrategy.

Scenariospresentedhereincludethosethatpertaintointernaluseofmodelsreleasedcommercially orthosethatareopensourced,aswellasscenariosfororganizationsthatconsumeLLMservices providedbythird-parties.ResourcesfromMITREEngenuity,OWASP,andothersarereferenced.

Thediagrambelowshowshowtheseresourcescanbeusedtocreateathreatinformeddefense strategy.

Figure1.3:ImageofintegratingLLMSecuritywithOWASPandMITREresources

ItisthehopeoftheOWASPTop10forLLMApplicationsteamthatthislistwillhelporganizations improvetheirexistingdefensivetechniquesanddeveloptechniquestoaddressthenewthreatsthat comefromusingthisexcitingtechnology.

WhyaChecklist?

Checklistscanhelpwithstrategydevelopmentbyensuringthoroughness,clarifyinggoals,fostering consistency,andallowingforfocused,deliberateeffort,allofwhichmayresultinfeweroversights. Followingthelistcanbuildconfidenceinapathtosecureadoptionwhilesparkingideasforfuture businesscasesmovingforward.Itśaveryforwardandverypracticalwaytoachievecontinuous improvement.

NotComprehensive

Whilethisdocumentisintendedtosupportorganizationsindevelopingan initialLLMstrategyinarapidlychangingtechnical,legal,andregulatoryenvironment,itdoesnot covereveryusecaseorobligation.Organizationsshouldextendassessmentsandpracticesbeyond thescopeoftheprovidedchecklistasrequiredfortheirusecaseorjurisdiction.

LargeLanguageModelChallenges

LargeLanguagemodelsfaceanumberofseriousanduniqueissues.Oneofthemostimportantis thatwhileworkingwithLLMs,thecontrolanddataplanescannotbestrictlyisolatedorseparable. AnothersignificantchallengeisthatLLMsarenondeterministicbydesign,yieldingadifferent outcomewhenpromptedorrequested.Itisnotalwaysachallenge,butLLMsemploysemantic searchratherthankeywordsearch.Thekeydistinctionbetweenthetwoisthatthemodel’salgorithm prioritizesthetermsinitsresponse.Thisisasignificantdeparturefromhowconsumershave traditionallyusedtechnology,andithasanimpactontheconsistencyandreliabilityofthefindings. Hallucinations,emergingfromthegapsandtrainingflawsinthedatathemodelistrainedon,are theresultofthismethod.

Therearemethodstoimprovereliabilityandreducetheattacksurfaceforjailbreaking,model tricking,andhallucinations,butthereisatrade-offbetweenrestrictionsandutilityinbothcostand functionality.

LLMuseandapplicationsincreaseanorganization’sattacksurface.Somerisksassociatedwith LLMsareunique,butmanyarefamiliarissues,suchastheknownsoftwarebillofmaterials(SBOM), supplychain,datalossprotection(DLP),andauthorizedaccess.Therearealsoincreasedrisksnot directlyrelatedtoGenAI,butGenAIincreasestheefficiency,capability,andeffectivenessofattacks.

AdversariesareincreasinglyharnessingLLMandGenerativeAItoolstorefineandexpeditetraditional methods.Theseenhancedtechniquesallowthemtoeffortlesslycraftnewmalware,potentially embeddedwithnovelzero-dayvulnerabilitiesordesignedtoevadedetection.Theycanalsogenerate sophisticated,unique,ortailoredphishingschemes.Thecreationofconvincingdeepfakes,whether videooraudio,furtherfacilitatestheirsocialengineeringploys.Additionally,thesetoolsenablethem toexecuteintrusionsanddevelopinnovativehackingutilities.Itisverylikelythatinthefuture,more “tailored”andcompounduseofAItechnologybycriminalactorswilldemandspecificresponses anddedicatedsolutionsforappropriatedefenseschemas.

LLMThreatCategories

ArtificialIntelligenceSecurityandPrivacyTraining

Employeesthroughoutorganizationsbenefitfromtrainingtounderstandartificialintelligence, generativeartificialintelligence,andthefuturepotentialconsequencesofbuilding,buying,orutilizing LLMs.Trainingforpermissibleuseandsecurityawarenessshouldtargetallemployeesaswellas bemorespecializedforcertainpositionssuchashumanresources,legal,developers,datateams, andsecurityteams.

Fairusepoliciesandhealthyinteractionarekeyaspectsthat,ifincorporatedfromtheverystart, willbeacornerstonetothesuccessoffutureAIcybersecurityawarenesscampaigns.Thiswill necessarilyimplytheuser’sknowledgeofthebasicrulesforinteractionaswellastheabilityto separategoodbehaviorfrombadorunethicalbehavior.

IncorporateLLMSecurityandgovernancewithExisting,EstablishedPractices andControls

WhileAIandgeneratedAIaddanewdimensiontocybersecurity,resilience,privacy,andmeeting legalandregulatoryrequirements,thebestpracticesthathavebeenaroundforalongtimearestill thebestwaytofindrisks,testthem,fixthem,andlowerthem.

• Themanagementofartificialintelligencesystemsisintegratedwithexistingorganizational practices.

• Applyexistingprivacy,governance,andsecuritypractices.

FundamentalSecurityPrinciples

LLMcapabilitiesintroduceadifferenttypeofattackandattacksurface.LLMsarevulnerable tocomplexbusinesslogicbugs,suchaspromptinjection,insecureplugindesign,andremote codeexecution.Existingbestpracticesarethebestwaytosolvetheseissues.Aninternalproduct securityteamthatunderstandssecuresoftwarereview,architecture,datagovernance,andthird-party assessmentsThecybersecurityteamshouldalsocheckhowstrongthecurrentcontrolsaretofind problemsthatcouldbemadeworsebyLLM,likevoicecloning,impersonation,orgettingaround captchas.

Accountingforthespecificskillsandcompetencesdevelopedinthelastfewyearsaroundmachine learning,NLPandNLU,deepLearningandlately,LLMsandGenAI,itisadvisedtohaveskilled professionalswithpractice,knowledge,orexperienceinthesefieldstosidewithsecurityteamsin adopting,atbest,andevenshapingnewpotentialanalysesandresponsestothoseissues.

Risk

ReferencetoriskusestheISO31000definition:Risk="effectofuncertaintyonobjectives."LLM risksincludedinthechecklistincludeatargetedlistofLLMrisksthataddressadversarial,safety, legal,regulatory,reputation,financial,andcompetitiverisks.

VulnerabilityandMitigationTaxonomy

Establishedmethodsofvulnerabilityclassificationandthreatsharingareinearlydevelopment,such asOval,STIX,threatsharing,andvulnerabilityclassification.Thechecklistanticipatescalibrating withexisting,established,andacceptedstandards,suchasCVEclassification.

DeterminingLLMStrategy

TheaccelerationofLLMapplicationshasraisedthevisibilityofallartificialintelligenceapplications’ organizationaluse.Recommendationsforpolicy,governance,andaccountabilityshouldbeconsidered holistically.

TheimmediateLLMthreatsaretheuseofonlinetools,browserplugins,third-partyapplications,the extendedattacksurface,andwaysattackerscanleverageLLMtoolstofacilitateattacks.

Figure3.1:ImageofstepsofLLMimplementation

DeploymentStrategy

Thescopesrangefromleveragingpublicconsumerapplicationstotrainingproprietarymodelson privatedata.Factorslikeusecasesensitivity,capabilitiesneeded,andresourcesavailablehelp determinetherightbalanceofconveniencevs.control.Butunderstandingthesefivemodeltypes providesaframeworkforevaluatingoptions.

Figure3.2:Imageofoptionsfordeploymentstrategy

CheckList

AdversarialRisk

AdversarialRiskincludescompetitorsandattackers.

□ Scrutinizehowcompetitorsareinvestinginartificialintelligence.AlthoughtherearerisksinAI adoption,therearealsobusinessbenefitsthatmayimpactfuturemarketpositions.

□ ThreatModel:howattackersmayaccelerateexploitattacksagainsttheorganization, employees,executives,orusers.

□ ThreatmodelspotentialattacksoncustomersorclientsthroughspoofingandgenerativeAI.

□ Investigatetheimpactofcurrentcontrols,suchaspasswordresets,whichusevoice recognition.

□ UpdatetheIncidentResponsePlanandplaybooksforLLMincidents.

AIAssetInventory

AnAIassetinventoryshouldapplytobothinternallydevelopedandexternalorthird-partysolutions.

□ CatalogexistingAIservices,tools,andowners.Designateataginassetmanagementfor specificinventory.

□ IncludeAIcomponentsintheSoftwareBillofMaterial(SBOM),acomprehensivelistofallthe softwarecomponents,dependencies,andmetadataassociatedwithapplications.

□ CatalogAIdatasourcesandthesensitivityofthedata(protected,confidential,public)

□ EstablishifpentestingorredteamingofdeployedAIsolutionsisrequiredtodeterminethe currentattacksurfacerisk.

□ CreateanAIsolutiononboardingprocess.

□ EnsureskilledITadminstaffisavailableeitherinternallyorexternally,inaccordancetothe SBoM

AISecurityandPrivacyTraining

□ Trainallusersonethics,responsibility,andlegalissuessuchaswarranty,license,andcopyright.

□ UpdatesecurityawarenesstrainingtoincludeGenAIrelatedthreats.Voicecloningandimage cloning,aswellasinanticipationofincreasedspearphishingattacks

□ AnyadoptedGenAIsolutionsshouldincludetrainingforbothDevOpsandcybersecurityfor thedeploymentpipelinetoensureAIsafetyandsecurityassurances.

EstablishBusinessCases

SolidbusinesscasesareessentialtodeterminingthebusinessvalueofanyproposedAIsolution,balancing riskandbenefits,andevaluatingandtestingreturnoninvestment.Thereareanenormousnumber ofpotentialusecases;afewexamplesareprovided.

□ Enhancecustomerexperience

□ Betteroperationalefficiency

□ Betterknowledgemanagement

□ Enhancedinnovation

□ MarketResearchandCompetitorAnalysis

□ Documentcreation,translation,summarization,andanalysis

Governance

CorporategovernanceinLLMisneededtoprovideorganizationswithtransparencyandaccountability. IdentifyingAIplatformorprocessownerswhoarepotentiallyfamiliarwiththetechnologyorthe selectedusecasesforthebusinessisnotonlyadvisedbutalsonecessarytoensureadequate reactionspeedthatpreventscollateraldamagestowellestablishedenterprisedigitalprocesses.

□ EstablishtheorganizationśAIRACIchart(whoisresponsible,whoisaccountable,whoshould beconsulted,andwhoshouldbeinformed)

□ DocumentandassignAIrisk,riskassessments,andgovernanceresponsibilitywithinthe organization.

□ Establishdatamanagementpolicies,includingtechnicalenforcement,regardingdata classificationandusagelimitations.Modelsshouldonlyleveragedataclassifiedforthe minimumaccesslevelofanyuserofthesystem.Forexample,updatethedataprotection policytoemphasizenottoinputprotectedorconfidentialdataintononbusiness-managed tools.

□ CreateanAIPolicysupportedbyestablishedpolicy(e.g.,standardofgoodconduct,data protection,softwareuse)

□ PublishanacceptableusematrixforvariousgenerativeAItoolsforemployeestouse.

□ Documentthesourcesandmanagementofanydatathattheorganizationusesfromthe generativeLLMmodels.

Legal

ManyofthelegalimplicationsofAIareundefinedandpotentiallyverycostly.AnIT,security,and legalpartnershipiscriticaltoidentifyinggapsandaddressingobscuredecisions.

□ Confirmproductwarrantiesareclearintheproductdevelopmentstreamtoassignwhois responsibleforproductwarrantieswithAI.

□ ReviewandupdateexistingtermsandconditionsforanyGenAIconsiderations.

□ ReviewAIEULAagreements.End-userlicenseagreementsforGenAIplatformsarevery differentinhowtheyhandleuserprompts,outputrightsandownership,dataprivacy, complianceandliability,privacy,andlimitsonhowoutputcanbeused.

□ ReviewexistingAI-assistedtoolsusedforcodedevelopment.Achatbotśabilitytowritecode canthreatenacompanyśownershiprightstoitsownproductifachatbotisusedtogenerate codefortheproduct.Forexample,itcouldcallintoquestionthestatusandprotectionofthe generatedcontentandwhoholdstherighttousethegeneratedcontent.

□ Reviewanyriskstointellectualproperty.Intellectualpropertygeneratedbyachatbotcould beinjeopardyifimproperlyobtaineddatawasusedduringthegenerativeprocess,whichis subjecttocopyright,trademark,orpatentprotection.IfAIproductsuseinfringingmaterial,it createsariskfortheoutputsoftheAI,whichmayresultinintellectualpropertyinfringement.

□ Reviewanycontractswithindemnificationprovisions.Indemnificationclausestrytoputthe responsibilityforaneventthatleadstoliabilityonthepersonwhowasmoreatfaultforitor whohadthebestchanceofstoppingit.Establishguardrailstodeterminewhethertheprovider oftheAIoritsusercausedtheevent,givingrisetoliability.

□ ReviewliabilityforpotentialinjuryandpropertydamagecausedbyAIsystems.

□ Reviewinsurancecoverage.Traditional(D&O)liabilityandcommercialgeneralliability insurancepoliciesarelikelyinsufficienttofullyprotectAIuse.

□ Identifyanycopyrightissues.Humanauthorshipisrequiredforcopyright.Anorganization mayalsobeliableforplagiarism,propagationofbias,orintellectualpropertyinfringementif LLMtoolsaremisused.

□ EnsureagreementsareinplaceforcontractorsandappropriateuseofAIforanydevelopment orprovidedservices.

□ RestrictorprohibittheuseofgenerativeAItoolsforemployeesorcontractorswhere enforceablerightsmaybeanissueorwherethereareIPinfringementconcerns.

□ AssessandAIsolutionsusedforemployeemanagementorhiringcouldresultindisparate treatmentclaimsordisparateimpactclaims.

□ MakesuretheAIsolutionsdonotcollectorsharesensitiveinformationwithoutproperconsent orauthorization.

Regulatory

TheEUAIActisanticipatedtobethefirstcomprehensiveAIlawbutwillapplyin2025atthe earliest.TheEUśGeneralDataProtectionRegulation(GDPR)doesnotspecificallyaddressAIbut includesrulesfordatacollection,datasecurity,fairnessandtransparency,accuracyandreliability, andaccountability,whichcanimpactGenAIuse.IntheUnitedStates,AIregulationisincludedwithin broaderconsumerprivacylaws.TenUSstateshavepassedlawsorhavelawsthatwillgointoeffect bytheendof2023.

FederalorganizationssuchastheUSEqualEmploymentOpportunityCommission(EEOC),the ConsumerFinancialProtectionBureau(CFPB),theFederalTradeCommission(FTC),andtheUS DepartmentofJusticeśCivilRightsDivision(DOJ)arecloselymonitoringhiringfairness.

□ DetermineStatespecificcompliancerequirements.

□ Determinecompliancerequirementsforrestrictingelectronicmonitoringofemployeesand employment-relatedautomateddecisionsystems(Vermont)

□ DeterminecompliancerequirementsforconsentforfacialrecognitionandtheAIvideoanalysis required(Illinois,Maryland)

□ ReviewanyAItoolsinuseorbeingconsideredforemployeehiringormanagement.

□ ConfirmthevendorścompliancewithapplicableAIlawsandbestpractices.

□ AskanddocumentanyproductsusingAIduringthehiringprocess.Askhowthemodelwas trained,howitismonitored,andtrackanycorrectionsmadetoavoiddiscriminationandbias.

□ Askanddocumentwhataccommodationoptionsareincluded.

□ Askanddocumentwhetherthevendorcollectsconfidentialdata.

□ Askhowthevendorortoolstoresanddeletesdataandregulatestheuseoffacialrecognition andvideoanalysistoolsduringpre-employment.

□ Reviewotherorganization-specificregulatoryrequirementswithAIthatmayraisecompliance issues.TheEmployeeRetirementIncomeSecurityActof1974,forinstance,hasfiduciaryduty requirementsforretirementplansthatachatbotmightnotbeabletomeet.

UsingorImplementingLargeLanguageModelSolutions

□ ThreatModel:LLMcomponentsandarchitecturetrustboundaries.

□ DataSecurity:Verifyhowdataisclassifiedandprotectedbasedonsensitivity,including personalandproprietarybusinessdata.(Howareuserpermissionsmanaged,andwhat safeguardsareinplace?)

□ AccessControl:Implementleastprivilegeaccesscontrolsandimplementdefense-in-depth measures

□ TrainingPipelineSecurity:Requirerigorouscontrolaroundtrainingdatagovernance,pipelines, models,andalgorithms.

□ InputandOutputSecurity:Evaluateinputvalidationmethods,aswellashowoutputsare filtered,sanitized,andapproved.

□ MonitoringandResponse:Mapworkflows,monitoring,andresponsestounderstand automation,logging,andauditing.Confirmauditrecordsaresecure.

□ Includeapplicationtesting,sourcecodereview,vulnerabilityassessments,andredteamingin theproductionreleaseprocess.

□ ConsidervulnerabilitiesintheLLMmodelsolutions(RezilionOSFFScorecard).

□ LookintotheeffectsofthreatsandattacksonLLMsolutions,suchaspromptinjection,the releaseofsensitiveinformation,andprocessmanipulation.

□ InvestigatetheimpactofattacksandthreatstoLLMmodels,includingmodelpoisoning, improperdatahandling,supplychainattacks,andmodeltheft.

□ SupplyChainSecurity:Requestthird-partyaudits,penetrationtesting,andcodereviewsfor third-partyproviders.(bothinitiallyandonanongoingbasis)

□ InfrastructureSecurity:Howoftendoesthevendorperformresiliencetesting?Whataretheir SLAsintermsofavailability,scalability,andperformance?

□ UpdateincidentresponseplaybooksandincludeanLLMincidentintabletopexercises.

□ IdentifyorexpandmetricstobenchmarkgenerativecybersecurityAIagainstotherapproaches tomeasureexpectedproductivityimprovements.

Resources

OWASPResources UsingLLMsolutionsexpandsanorganization’sattacksurfaceandpresentsnew challenges,requiringspecialtacticsanddefenses.Italsoposesproblemsthataresimilartoknown issues,andtherearealreadyestablishedcybersecurityproceduresandmitigations.IntegratingLLM cybersecuritywithanorganization’sestablishedcybersecuritycontrols,processes,andprocedures allowsanorganizationtoreduceitsvulnerabilitytothreats.Howtheyintegratewitheachotheris availableattheOWASPIntegrationStandards.

OWASPResource Description WhyItIsRecommended&Where ToUseIt

OWASPSAMM

OWASPAISecurityand PrivacyGuide

SoftwareAssurance MaturityModel

OWASPAIExchange

OWASPProjectwitha goalofconnecting worldwideforan exchangeonAIsecurity, fosteringstandards alignment,anddriving collaboration.

OWASPAIExchangeis theintakemethodforthe OWASPAISecurityand PrivacyGuide.

Providesaneffectiveand measurablewaytoanalyzeand improveanorganization’ssecure developmentlifecycle.SAMM supportsthecompletesoftware lifecycle.Itisinterativeand risk-driven,enablingorganizations toidentifyandprioritizegapsin securesoftwaredevelopment soresourcesforimproving theprocesscanbededicated whereeffortshavethegreatest improvementimpact.

TheOWASPAISecurityandPrivacy Guideisacomprehensivelistof themostimportantAIsecurityand privacyconsiderations.Itismeant tobeacomprehensiveresourcefor developers,securityresearchers, andsecurityconsultantstoverify thesecurityandprivacyofAI systems.

TheAIExchangeistheprimary intakemethodusedbyOWASPto drivethedirectionoftheOWASPAI SecurityandPrivacyGuide.

OWASPMachine

LearningSecurity Top10

OpenCRE

OWASPThreatModeling

OWASPMachine LearningSecurity Top10securityissues ofmachinelearning systems.

OWASPCycloneDX

OpenCRE(Common Requirement Enumeration)is theinteractive content-linkingplatform forunitingsecurity standardsandguidelines intooneoverview.

Astructured,formal processforthreat modelingofan application

OWASPCycloneDX isafull-stackBill ofMaterials(BOM) standardthatprovides advancedsupplychain capabilitiesforcyberrisk reduction.

TheOWASPMachine LearningSecurityTop10isa community-drivenefforttocollect andpresentthemostimportant securityissuesofmachinelearning systemsinaformatthatiseasy tounderstandbybothasecurity expertandadatascientist.This projectincludestheMLTop10 andisaliveworkingdocument thatprovidesclearandactionable insightsondesigning,creating, testing,andprocuringsecureand privacy-preservingAIsystems.It isthebestOWASPresourcefor AIglobalregulatoryandprivacy information.

Usethissitetosearchfor standards.Youcansearchby standardnameorbycontroltype.

LearneverythingaboutThreat Modelingwhichisastructured representationofallthe informationthataffectsthe securityofanapplication.

Modernsoftwareisassembled usingthird-partyandopensource components.Theyareglued togetherincomplexandunique waysandintegratedwithoriginal codetoachievethedesired functionality.AnSBOMprovides anaccurateinventoryofall componentswhichenables organizationstoidentifyrisk, allowsforgreatertransparency, andenablesrapidimpactanalysis. EO14028providedminimum requirementsforSBOMforfederal systems.

OWASPSoftware ComponentVerification Standard(SCVS)

OWASPAPISecurity Project

Acommunity-driven efforttoestablisha frameworkforidentifying activities,controls,and bestpracticescanhelpin identifyingandreducing riskinasoftwaresupply chain.

APISecurityfocuses onstrategiesand solutionstounderstand andmitigatethe uniquevulnerabilities andsecurityrisks ofApplication ProgrammingInterfaces (APIs)

UseSCVStodevelopacommon setofactivities,controls,and best-practicesthatcanreduce riskinasoftwaresupplychain andidentifyabaselineandpath tomaturesoftwaresupplychain vigilance.

APIsareafoundationalelement ofconnectingapplications,and mitigatingmisconfigurationsor vulnerabilitiesismandatoryto protectusersandorganizations. Useforsecuritytestingandred teamingthebuildandproduction environments.

OWASPApplication SecurityVerification StandardASVS

OWASPThreatand SafeguardMatrix (TaSM)

ApplicationSecurity VerificationStandard (ASVS)Projectprovides abasisfortestingweb applicationtechnical securitycontrols andalsoprovides developerswithalistof requirementsforsecure development.

Anactionorientedview tosafeguardandenable thebusiness

Cookbookforwebapplication securityrequirements,security testing,andmetrics.Useto establishsecurityuserstoriesand securityusecasereleasetesting.

DefectDojo

Anopensource vulnerability managementtoolthat streamlinesthetesting processbyoffering templating,report generation,metrics,and baselineself-service tools.

Thismatrixallowsacompanyto overlayitsmajorthreatswiththe NISTCyberSecurityFramework Functions(Identify,Protect,Detect, Respond,&Recover)tobuilda robustsecurityplan.Useitasa dashboardtotrackandreporton securityacrosstheorganization.

UseDefectDojotoreducethe timeforloggingvulnerabilities withtemplatesforvulnerabilities, importsforcommonvulnerability scanners,reportgeneration,and metrics.

Table5.1:OWASPResources

Figure5.1:ImageofOWASPTop10forLargeLanguageModelApplications

Figure5.2:ImageofOWASPTop10forLargeLanguageModelApplicationsVisualized

MITREResources TheincreasedfrequencyofLLMthreatsemphasizesthevalueofaresilience-first approachtodefendinganorganization’sattacksurface.ExistingTTPSarecombinedwithnew attacksurfacesandcapabilitiesinLLMAdversarythreatsandmitigations.MITREmaintainsa well-establishedandwidelyacceptedmechanismforcoordinatingopponenttacticsandprocedures basedonreal-worldobservations.

Coordinationandmappingofanorganization’sLLMSecurityStrategytoMITREATT&CKandMITRE ATLASallowsanorganizationtodeterminewhereLLMSecurityiscoveredbycurrentprocesses suchasAPISecurityStandardsorwheresecurityholesexists.

MITREATT&CK(AdversarialTactics,Techniques,andCommonKnowledge)isaframework,collection ofdatamatrices,andassessmenttoolthatwasmadebytheMITRECorporationtohelporganizations figureouthowwelltheircybersecurityworksacrosstheirentiredigitalattacksurfaceandfindholes thathadnotbeenfoundbefore.Itisaknowledgerepositorythatisusedallovertheworld.The MITREATT&CKmatrixcontainsacollectionofstrategiesusedbyadversariestoachieveacertain goal.IntheATT&CKMatrix,theseobjectivesareclassifiedastactics.Theobjectivesareoutlinedin attackorder,beginningwithreconnaissanceandprogressingtotheeventualgoalofexfiltrationor impact.

MITREATLAS,whichstandsfor"AdversarialThreatLandscapeforArtificialIntelligenceSystems,"is aknowledgebasethatisbasedonreal-lifeexamplesofattacksonmachinelearning(ML)systems bybadactors.ATLASisbasedontheMITREATT&CKarchitecture,anditstacticsandprocedures complementthosefoundinATT&CK.

MITREResource Description WhyItIsRecommended&Where ToUseIt

MITREATT&CK

Knowledgebaseof adversarytacticsand techniquesbasedon real-worldobservations

MITREAT&CK Workbench

CreateorextendATT&CK datainalocalknowledge base

TheATT&CKknowledgebase isusedasafoundationforthe developmentofspecificthreat modelsandmethodologies. Mapexistingcontrolswithinthe organizationtoadversarytactics andtechniquestoidentifygapsor areastotest.

Hostandmanageacustomized copyoftheATT&CKknowledge base.Thislocalcopyofthe ATT&CKknowledgebasecanbe extendedwithneworupdated techniques,tactics,mitigation groups,andsoftwarethatis specifictoyourorganization.

MITREATLAS

MITREATLAS

(AdversarialThreat Landscapefor Artificial-Intelligence Systems)isaknowledge baseofadversary tactics,techniques, andcasestudiesfor machinelearning(ML) systemsbasedon real-worldobservations, demonstrationsfromML redteamsandsecurity groups,andthestate ofthepossiblefrom academicresearch

UseittomapknownML vulnerabilitiesandmapchecks andcontrolsforproposedprojects orexistingsystems.

MITREATT&CKPowered Suit

TheThreatReport ATT&CKMapper(TRAM)

ATT&CKPoweredSuitis abrowserextensionthat putstheMITREATT&CK knowledgebaseatyour fingertips.

AutomatesTTP IdentificationinCTI Reports

Addtoyourbrowsertoquickly searchfortactics,techniques, andmorewithoutdisruptingyour workflow.

MappingTTPsfoundinCTIreports toMITREATT&CKisdifficult, errorprone,andtime-consuming. TRAMusesLLMstoautomatethis processforthe50mostcommon techniques.SupportsJuypter notebooks.

AttackFlowv2.1.0

MITRECaldera

CALDERAplugin: Arsenal

AttackFlowisa languagefordescribing howcyberadversaries combineandsequence variousoffensive techniquestoachieve theirgoals.

Acybersecurityplatform (framework)designed toeasilyautomate adversaryemulation, assistmanualred-teams, andautomateincident response.

Aplugindevelopedfor adversaryemulationof AI-enabledsystems.

AttackFlowhelpsvisualizehow anattackerusesatechnique,so defendersandleadersunderstand howadversariesoperateand improvetheirowndefensive posture.

PluginsareavailableforCaldera thathelptoexpandthecore capabilitiesoftheframeworkand provideadditionalfunctionality, includingagents,reporting, collectionsofTTPsandothers

ThispluginprovidesTTPsdefined inMITREATLAStointerfacewith CALDERA.

MITREResource Description WhyItIsRecommended&Where

AtomicRedTeam Libraryoftestsmapped totheMITREATT&CK framework.

MITRECTIBlueprints AutomatesCyberThreat Intelligencereporting.

Usetovalidateandtestcontrols inanenvironment.Securityteams canuseAtomicRedTeamto quickly,portably,andreproducibly testtheirenvironments.Youcan executeatomictestsdirectlyfrom thecommandline;noinstallation isrequired.

CTIBlueprintshelpsCyberThreat Intelligence(CTI)analystscreate high-quality,actionablereports moreconsistentlyandefficiently.

Table5.2:MITREResources

AIVulnerabilityRepositories

Name Description

AIIncidentDatabase

OECDAIIncidentsMonitor(AIM)

ArepositoryofarticlesaboutdifferenttimesAIhas failedinreal-worldapplicationsandismaintainedbya collegeresearchgroupandcrowdssourced.

Offersanaccessiblestartingpointforcomprehending thelandscapeofAI-relatedchallenges.

ThreeoftheleadingcompaniestrackingAIModelvulnerabilities

HuntrBugBounty:ProtectAI BugbountyplatformforAI/ML

AIVulnerabilityDatabase(AVID):Garak Databaseofmodelvulnerabilities

AIRiskDatabase:RobustIntelligence Databaseofmodelvulnerabilities

Table5.3:AIVulnerabilityRepositories

AIProcurementGuidance

Name Description

WorldEconomicForum:AdoptingAI Responsibly:GuidelinesforProcurementof AISolutionsbythePrivateSector:Insight ReportJune2023

Thestandardbenchmarksandassessmentcriteriafor procuringArtificialsystemsareinearlydevelopment. Theprocurementguidelinesprovideorganizations withabaselineofconsiderationsfortheend-to-end procurementprocess.

Usethisguidancetoaugmentanorganization’sexisting ThirdPartyRiskSupplierandVendorprocurement process.

Table5.4:AIProcurementGuidance

Team

ThankyoutotheOWASPTop10forLLMApplicationsCybersecurityandGovernanceChecklist Contributors.

ChecklistContributors

SandyDunn

SteveWilson

BobSimonoff

EmmanualGuilhermeJunior

HeatherLinn

FabrizioCilli

DavidRowe

AndreaSucci

JohnSotiropoulos

AubreyKing

RobVanderveer

JasonRoss

TableA.1:OWASPLLMAISecurity&GovernanceChecklist v.0.5Team