SUPPORTPOOL

OFEXPERTSPROGRAMME

AIAuditing

ProposalforAIleaflets

byDr.GemmaGALDONCLAVELL

AIAuditing-ProposalforAIleaflets

2

AspartoftheSPEprogramme,theEDPBmaycommissioncontractorstoprovidereportsandtoolsonspecifictopics.

TheviewsexpressedinthedeliverablesarethoseoftheirauthorsandtheydonotnecessarilyreflecttheofficialpositionoftheEDPB.TheEDPBdoesnotguaranteetheaccuracyoftheinformation

includedinthedeliverables.NeithertheEDPBnoranypersonactingontheEDPB’sbehalfmaybeheldresponsibleforanyusethatmaybemadeoftheinformationcontainedinthedeliverables.

Someexcerptsmayberedactedorremovedfromthedeliverablesastheirpublicationwould

underminetheprotectionoflegitimateinterests,including,interalia,theprivacyandintegrityofanindividualregardingtheprotectionofpersonaldatainaccordancewithRegulation(EU)2018/1725and/orthecommercialinterestsofanaturalorlegalperson.

AIAuditing-ProposalforAIleaflets

3

TableofContents

Background 4

1.Basicdefinitions 4

2.Whyalgorithmicleaflets 5

3.FrommodelcardstoAIleaflets 7

4.AIleaflettemplate 9

References 12

DocumentinitiallysubmittedinJanuary2023,updatedinJune2024

AIAuditing-ProposalforAIleaflets

4

Background

Since2016,GDPRhaslaidoutprinciplesandproceduresthathaveshapedhowissuesrelatedtodataprotectionandsocialimpactareaddressedindata-intensivetechnologies.Thenotionsoftransparency(articles13and14GDPR),“humanintervention”(article22.3GDPR),informationaboutthelogicoftheprocessing(article14.2.gGDPR),accountability(article5.2GDPR),dataprotectionbydesignandbydefault(article25GDPR)andauditability(includingthenotionofconformityassessmentintheAIact)haveshapedashared,globalunderstandingofwhatdataprotectionmeansinpractice.

Tobeaccountablemeans,amongothers,acompletetraceabilityofallthedesigndecisions,takenbydesign,properlydocumented,analysedinadvance,andbackedwithproofandevidence.Butwhiletheaccountabilityprincipleshavebeenlaidout,itisstillunclearhowtheseprinciplescanbeimplementedandcheckedinpracticeinwaysthatcoverallrelevantmomentsinthesupplychainandfacilitateenforcementbythesupervisoryauthorities.

ThisisparticularlyrelevantatatimewhenweseetheaccountabilitychaingettingincreasinglycomplexinAI,withcompaniesoftenbuyingAI(foundational)modelsandservicesfromthirdpartiesandretrainingthemwithadditionaldataorusingthemontheirowndecision-makingprocesses.

WehavedevelopedAIleafletsasakeytoolofeffectiveAItransparencyforAIusersandimplementors,butalsoasamechanismtoprotectSMEsandprovidealevel-playingfieldforallindustryactors.AIimplementorsandthoseusingAI,bothend-usersandAI“clients”,currentlylackstandardizedtoolstoexercisefree,Informedchoice.Intheabsenceofthesetools,entitiesareforcedtorelyonmarketingclaimsandunverifiedinformationwhichmaycreaterisksfortheirusersandexposeorganizationsto“inheritedliability”.

InthissecondreportfortheEuropeanDataProtectionBoard(EDPB),wedevelopaproposalfor“AIleaflets”,aconceptexportedfromthemedicaldomaintoenforceaprioritransparencyforAIsystemsandproducts,andwhichdrawsonpreviousworkdevelopedfortheSpanishDPAandtheSpanishMinistryofLabor.AIleafletscomplementexistingtoolslikeModelCards,impactassessments,AIauditsandalgo-scores.Duetotheirtechnicalnature,AileafletsareclosetoModelCards.AstheinformationinanAIleafletisintendedforatech-savvyaudience,Aiimplementorsshouldimplementthealgo-scoresweproposedinourfirstreporttofacilitateand-userunderstandingandchoice.

1.Basicdefinitions

Objectivesofthealgorithmicleaflet:toprovideaccessibleinformationthatpromotestransparency,auditabilityandrecoursetothosebuying,implementingorbeingimpactedbyAIsystems.TheleafletfacilitatescompliancewithrequirementsincludedinGDPRandAIAct.

Definitionofalgorithmicsystem:softwarethatisdevelopedwithoneormoretechniquesandMachineLearningapproaches,includingsupervised,unsupervisedandreinforcementlearning,usingawidevarietyofmethodsincludingdeeplearning;Logic-andknowledge-basedapproaches,includingknowledgerepresentation,inductive(logic)programming,knowledgebases,inferenceanddeductiveengines,(symbolic)reasoningandexpertsystems;andstatisticalapproaches,Bayesianestimation,searchandoptimizationmethodsthatcan,foragivensetofhuman-definedobjectives,generateoutputssuchascontent,predictions,recommendations,ordecisionsinfluencingtheenvironmentstheyinteractwith(adaptedfromthedefinitionofartificialintelligencecontainedinAIActart.3.1).

Examplesofaffectedsystems:thosewhereoneormorealgorithmsareatthecenterofadecision-makingprocessthathasimplicationsforfundamentalrightsorindividual/collectivelife-chances,includingsocialmediacontentrecommenders,price/retributionmodelsinconsumerservices,hiring

AIAuditing-ProposalforAIleaflets

5

decisions,individual/groupriskassessmentindifferentsettings(facialrecognitionasproofoflife/identity,benefitallocation,recidivism,etc)andLargeandSmallLanguageandImageModels(GenerativeAI)usedtointeractwithcomplexorunstructureddatawhichproducenewcontentusersrelyontounderstandanissueormakedecisions.

Inheritedliability:WhenoneentitybuysAIproductsfromanotherandusesthemintheirowndecision-makingprocessesorproductdesign,itcanbeheldlegallyresponsibleforanyissuesthatleadtoharmful,inefficientordiscriminatorydecisionsorassessments.LeafletsprovidekeyinformationtoAIclientsanduserssotheycanmakebetterdecisionswhenchoosinganAIsystemorprovider.

2.Whyalgorithmicleaflets

Inthelastfewyears,atleast170setsofethicalorhuman-rightsbasedAIprinciples,frameworks,andguidelineshavebeendevelopedtosupportresponsibleAIdevelopmentanddeploymentinthepublicandprivatesectors.

1

Researchhasshownthatagrowingconsensushasemergedaroundcoreprinciples,suchastheneedforaccountability,privacyandsecurity,transparencyandexplainability,fairnessandnon-discrimination,professionalresponsibility,humancontrol,andthepromotionofhumanvalues.

2

TheseprinciplesandvalueshavemadeitintodiscussionsaroundhowtoregulateAI-relatedtechnologies,andbothexistingEUregulationssuchastheGeneralDataProtectionRegulation(GDPR)andtheDigitalServicesAct(DSA)andnewregulatoryproposalsbeingdiscussedrightnow,suchastheAIACT,echothisemergentconsensus.

Butwhilesignificantstepshavebeentakentoalignhigh-levelapproachesandprinciples,animportantlessonfromtheGDPR,passedin2016,isthatenforcementcanbeachallenge.AsAIprinciplesgainacceptancewithinthepublicandprivatesectors,thefocusisshiftingtothedevelopmentofappropriatestrategiestooperationalizethemintoresponsiblepractices.Yet,asNonneckeandDawsonhighlight,“thisprocessisnotstraightforward”.

3

Onewaytoacceleratetheadoptionofenforcementpracticesisbydrawingonthelonghistoryofhowmodernsocietieshavedealtwiththenegativeexternalitiesofinnovation,howcomplexscientificinsighthasbeencommunicatedtousersandcitizensinrecenthistory,andthetoolsthathaveemergedtoprotectpeopleandrightsinhighlyinnovativeprocesses.

Lookingatthehistoryoftheregulationofinnovation,arelevantprecedentandexamplefortheeffectiveregulationofAIsystemsandproductsisthemedicalsector.Inthelate18thCenturyandearly19thcentury,manycompaniesdevelopingdrugsandmedicinewouldmarkettheirproductsunderfalse,untestedpremises.In1902,oneadvertisementforamedicalproductclaimed,“Nootherpreparationhashaditstherapeuticvaluemorethoroughlydefinedorbetterestablished...[as]aremedyinthetreatmentofcoughs,bronchitis...asthma,laryngitis,pneumonia,andwhoopingcough.”Thedrugwasheroin.

4

1AIEthicsGuidelinesGlobalInventory,”AlgorithmWatch,

/

2JessicaFjeldetal.(2020),PrincipledArtificialIntelligence:MappingConsensusinEthicalandRights-basedApproachestoPrinciplesforAI,BerkmanKleinCenterforInternet&Society,

/urn-3:HUL.InstRepos:42160420

3Nonnecke,B.andDawson,B.(2021)HumanRightsImplicationsofAlgorithmicImpact

Assessments:PriorityConsiderationstoGuideEffectiveDevelopmentandUse.CarrCenterDiscussionPaperSeries,

/files/cchr/files/nonneckeanddawsonhumanrightsimplications.

pdf

4Hamburg,M.A.(2010),Innovation,Regulation,andtheFDA.NEnglJMed;363:2228-2232.

/doi/full/10.1056/NEJMsa1007467

AIAuditing-ProposalforAIleaflets

6

Andwhilethe20thCenturysawenormousandhugelybeneficialadvancesinmedicine,initsearlydecadesmanycompaniesmarketedtheirproductswithavarietyofunprovenclaims.Itwas,aspharmacologistLouisGoodmancalledit’,a“therapeuticjungle”,notmuchdifferentfromatechandAIindustrythatmanyhavedescribedasthe“WildWest”.Ittookseveralpublichealthcrisestopullmedicineintothemodernerabytriggeringnewregulatoryauthoritiesandstandards.ThishappenedearlierintheUS,wheretheElixirSulfanilamidecaseandits107victimspromptedthepassingoftheFood,Drug,andCosmeticActin1938.Thelawestablishedthatdrugsintendedtopreventortreatdiseasehadtoprovetheyweresafeforuseaslabeledandreceiveaprioriauthorizationbyprovidingkeydatatotheregulator.“Forthefirsttime,beforepharmaceuticalcompaniescouldmarketadrug,theyhadtoshowatleastthattheproductwassafe.”

5

Itwasunclearatfirstwhatdatahadtobesharedtoprovecompliance,butovertimestandardizedassessmentsemergedandbecamestandardpracticeacrossthepharmaceuticalindustry.

ThisearlydevelopmentofaregulatoryframeworkfordrugsmeantthattheUSmanagedtoprotectitscitizensfromthehealthcrisisthatpromptedthedevelopmentofsimilarprotectionsinEurope.TheUSregulatordeniedapprovaltothalidomide,adrugwidelymarketedinEuropeasasedativeandantiemeticagentandrecommendedforusebywomenintheirfirsttrimesterofpregnancy,becauseitsmanufacturerfailedtoshowbasicaspectsoftheproduct'spharmacologicandtoxicologiccharacteristics.IntheEU,manybabiesdiedandthousandswerebornwithseverehealthproblems.ThethalidomidetragedyservedasthecatalystforharmonizedEuropeanpharmaceuticalregulation,whichisnowcentralisedundertheEuropeanMedicinesAgency(EMA).

OneofthekeycompetenciesoftheEMAisto“provideguidanceandtemplates[…]withpracticaladviceonhowtodrawuptheproductinformationforhumanmedicines,whichincludes[…]apackageleaflet”,definedas“Theleafletineverypackofmedicinethatcontainsinformationonthemedicineforend-users,suchaspatientsandanimalowners.”

6

Thisleafletisthemainpieceofwritteninformationthatcitizensreceivewhenusingdrugsthathavebeendesignedtohelpthembutmayharmthem.Togetherwiththemedicalprescriptionandtheassistanceofpharmacystaff,packageleafletsareawaytoprotectandenforcerights,guideproperuseandprovideinformationthatempowerscitizenstounderstandthecharacteristicsandusesofmedicalproducts,aswellaswaystoseekrecourseshouldanythinggowrong.

7

5?bid.

6EMAwebsite

https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/product-

information-requirements

7Directive2001/83/ECoftheEuropeanParliamentandoftheCouncilestablishedthatsuchpackageleafletsmustincludeinformationon:(a)thenameofthemedicinalproductfollowedbyitsstrength

andpharmaceuticalform,and,ifappropriate,whetheritisintendedforbabies,childrenoradults;

wheretheproductcontainsuptothreeactivesubstances,theinternationalnonproprietaryname

(INN)shallbeincluded,or,ifonedoesnotexist,thecommonname;(b)astatementoftheactive

substancesexpressedqualitativelyandquantitativelyperdosageunitoraccordingtotheformof

administrationforagivenvolumeorweight,usingtheircommonnames;(c)thepharmaceuticalformandthecontentsbyweight,byvolumeorbynumberofdosesoftheproduct;(d)alistofthose

excipientsknowntohavearecognizedactionoreffectandincludedinthedetailedguidance

publishedpursuanttoArticle65.However,iftheproductisinjectable,oratopicaloreyepreparation,allexcipientsmustbestated;(e)themethodofadministrationand,ifnecessary,therouteof

administration.Spaceshallbeprovidedfortheprescribeddosetobeindicated;(f)aspecialwarningthatthemedicinalproductmustbestoredoutofthereachandsightofchildren;(g)aspecialwarning,

ifthisisnecessaryforthemedicinalproduct;(h)theexpirydateinclearterms(month/year);(i)specialstorageprecautions,ifany;(j)specificprecautionsrelatingtothedisposalofunusedmedicinal

productsorwastederivedfrommedicinalproducts,whereappropriate,aswellasreferencetoanyappropriatecollectionsysteminplace;(k)thenameandaddressofthemarketingauthorisation

holderand,whereapplicable,thenameoftherepresentativeappointedbytheholdertorepresent

AIAuditing-ProposalforAIleaflets

7

3.FrommodelcardstoAIleaflets

TheadaptationofthemedicalleafletmodeltotheAIandtechnicalinnovationspaceholdssignificantpromise,butalsochallenges.Thefirstmainchallengeisdefiningwhatneedstobesharedinthisexerciseof“upfront”transparency.ThecomplexitiesofdoingtransparencyinpracticehavebeenacknowledgedbytheEuropeanParliament,asevidencedbythereleasein2019ofareporton“Agovernanceframeworkforalgorithmicaccountabilityandtransparency”

8

andtheEuropeanCommission’screationoftheEuropeanCentreforAlgorithmicTransparency(ECAT)in2023.

9

Inthisproposalwemoveawayfromanotionofabsolutetransparency,whichmayimplysharingcodeorhighlytechnicaldatathatlaycitizensmaynotbeequippedtounderstandandusetoprotecttheirrights,andfavoranotionof“meaningfultransparency”,drawingonAnnanyandCrawford,

10

Kaminski

11

andtheexcellentworkofSafakandParkerfortheAdaLovelaceInstitute.

12

Bymeaningfultransparencywemeaninformationthat“isrealisticallyaccessibletoamemberofthegeneralpublicatthetimeoftherequest.Itmustbeavailableinpractice,notjustintheory”,astheICOputit.

13

Here,weseektomakeinformationaccessibleforthegeneralpublic,butalsoregulators,civilsocietyorganizationsandallrelevantparties.Thisrequiressomelevelof“translation”ofhighlytechnicalterms,butalsotheincorporationofnon-technicalinformationrelatedtogovernanceandimpacts.

Inordertoengageintherequiredtranslationexercise,wealsodrawoneffortstofosterthedocumentationofthedecisionsmadeduringthedevelopmentandtestingoftechnologyproducts,andspecificallyonthe“ModelCardsforModelReporting”proposaldevelopedbyMitchelletal.whileworkingatGoogle,whichhavebecomeawidespreadtool.

14

Intheirpaper,theydefinemodelcardsas“shortdocumentsaccompanyingtrainedmachinelearningmodelsthatprovidebenchmarkedevaluationinavarietyofconditions,suchasacrossdifferentcultural,demographic,orphenotypicgroups(e.g.,race,geographiclocation,sex,Fitzpatrickskintype)andintersectionalgroups(e.g.,ageandrace,orsexandFitzpatrickskintype)thatarerelevanttotheintendedapplicationdomains.Modelcardsalsodisclosethecontextinwhichmodelsareintendedtobeused,detailsoftheperformanceevaluationprocedures,andotherrelevantinformation.”

15

him;(l)thenumberoftheauthorizationforplacingthemedicinalproductonthemarket;(m)the

manufacturer'sbatchnumber;(n)inthecaseofnon-prescriptionmedicinalproducts,instructionsforuse.

8Availableat

https://www.europarl.europa.eu/RegData/etudes/STUD/2019/624262/EPRSSTU(2019)624262EN.p

df

9See

https://algorithmic-transparency.ec.europa.eu/indexen

10Ananny,M.,&Crawford,K.(2018).Seeingwithoutknowing:Limitationsofthetransparencyidealanditsapplicationtoalgorithmicaccountability.NewMedia&Society,20(3),973–989.

/10.1177/1461444816676645

11Kaminski,MargotE.,UnderstandingTransparencyinAlgorithmicAccountability(June8,2020).ForthcominginCambridgeHandbookoftheLawofAlgorithms,ed.WoodrowBarfield,CambridgeUniversityPress(2020).,UofColoradoLawLegalStudiesResearchPaperNo.20-34,AvailableatSSRN:

/abstract=3622657

12CansuSafakandImogenParker(2020)Meaningfultransparencyand(in)visiblealgorithms.Ada

LovelaceInstitute

/blog/meaningful-transparency-and-invisible-

algorithms/

13SeeICO“Informationinthepublicdomain”

.uk/for-organisations/guidance-

index/freedom-of-information-and-environmental-information-regulations/information-in-the-public-

domain/

14AmazonWebServices,forinstance,usesaversionofModelCardscalled“serviecards”.

15MitchellM,WuS,ZaldivarA,BarnesP,VassermanL,HutchinsonB,SpitzerE,RajiID,GebruT.Modelcardsformodelreporting.InProceedingsoftheconferenceonfairness,accountability,andtransparency2019Jan29(pp.220-229).

/abs/1810.03993

AIAuditing-ProposalforAIleaflets

8

OnelimitationofModelCardswhichweseektoovercomewithAIleafletsistheirfocusonthemodelalone.AsthesupplychainsofAIbecomemorecomplex,thereisaneedtodevelopmechanismsthatcaptureboththeexistenceofdifferentdevelopersandactors,thecombinationofdifferentdatasourcesandthelikelihoodoffinalAIimplementorsusingAImodelsinwaysthatwerenotforeseenorwithnon-testeddata.

Thecombinationofthehistoricalexampleandsuccessesofthemedicalsectorinprotectingpeopleandrightsthroughmeaningful,concretepractices,someindustryeffortstopromotegreatertransparencyinengineeringdecisions,themanyproposalsthathaveemergedfromcivilsocietyandpublicandprivateactorsdemandingactioninthespaceofalgorithmicexplainability,accountabilityandenforcement,aswellastheneedfromregulatorstohavesharedstandardstoassesscompliance,resultsinaproposalforanexerciseofupfrontmeaningfultransparencywhichwehavecalled“algorithmicleaflet”andisdescribedindetailinthenextsection

Therearethreenotesworthhighlightingbeforeengaginginthedescriptionofthealgorithmicleafletfields.First,thattheformatandfieldsproposedareanattempttoovercomesomeoftheissuesthathavemadeotherpolicytoolsdifficulttoimplementinpractice.Specifically,withalgorithmicleafletswesuggestaformatthatistransparentbydesigninthatleafletsaremadeavailabletoendusers,regulatorsandpotentialbuyerstofacilitatedecision-makingandunderstandingofhowalgorithmicsystemswork.Also,itisourexperiencethatDataProtectionImpactAssessmentsareoftendevelopedbylegalteamswhomaynothaveaccesstoortheskillsrequiredtoassesstechnicalprocesses.Theleafletweproposeishighlytechnicalinitsconceptiontoensurethattherelevantinformationiscollectedbythetechnicalteamsmakingtherelevantdecisions,andthattheinformationisreleasedpubliclyatthesametimeasthetechnology.

Second,thatthealgorithmicleafletisnottheonlytoolthatcanorshouldpromotebettertransparency,accountabilityandtrustworthinessaroundAIandtechnologicalsystemsandprocesses.Thedynamicnatureofmanyalgorithmicsystemsmeansthatanyattempttocapturetheirfunctioningandimpactsmaybeshort-livedorincomplete,andsoleafletsneedtobecomplementedbydynamicexerciseslikeaudits,andclearinstructionsonhowoftenandwhentoupdatethem.AIleafletsarenotaperfecttooleither.ButtheyareagoodenoughtoolthattranslatesandstandardizesveryrealconcernsaroundtheneedtobetterunderstandhowAIsystemswork,toempowercitizensandcivilsocietytoengagewithtechnicalsystemsandtoprovidetheAIindustrywithclearinstructionsastowhatconstitutescompliance.

Third,wewanttohighlightthatinordertopromotetheeffectiveincorporationofAIleaflets,itisrecommendedthataprocessofexpertconsultationandindustrypilotingisdesignedandimplementedbeforeitstermsarefinalized.Standardsbecomemeaningfulwhentheyareeitherimposedthroughlawsandregulationsortheresultofcollaborativeprocessesthatallowthemtoconsolidate.Duetotherapidlychangingnatureofthetechnicalfield,andtheimplementationchallengesobservedinothertechnology-relatedregulation,itisdesirablethatthepracticeandimplementationtoolsthatwillneedtoemergetomakelegalprotectionseffectiveandmeaningfulareembracedbyasmanyactorsaspossible.

AIAuditing-ProposalforAIleaflets

9

4.AIleaflettemplate

Thissectionstartsbyprovidinganoverviewoftheleafletcategories.Specificdefinitionsforeachitemareprovidedbelow.

Leafletcategories:

Generalinformation

oSystemname/codeandversion(5.2GDPR)

oLeafletversionandversionhistory(5.2GDPR)

oSystemownerandsuppliersdata

oSuppliers’role

oRisklevel(AIAct)

oGovernanceroles(ChapterIVGDPR)

oDistributiondate(5.2GDPR)

oExistingdocumentation

Informationonprocess

oDescriptionofintendedpurposes,uses,contextandrole/serviceprovided(Article5.1.b,5.2and24.1GDPR)

oStakeholderinvolvement

oOrganizationalcontext

oHumanrole/s(Article22GDPR)

Informationontraining/validationdata

oDatasources/collectionmethodology(Articles5and9GDPR)

oDatatypesandcharacteristics(Article5.1.a,bGDPR)

oPrivacybyDesign(Article25GDPR)

oDatasheetsforDatasets(Article5.1.a,bGDPR)

Informationonthemodel

oMethod/susedandjustification

oSimplifiedoutput/s

oDecisionvariables

oObjectivefunction/s(Article5.1.dGDPR)

Informationonbiasandimpacts(inlab/operationalsettings)

oMetrics(Articles5.1.aand5.1.bGDPR)

oProtectedcategories(Articles13.1.e,14.1.eand35.9GDPR)

oImpactratespercategoryandprofilebeforeandaftereachtechnicalintervention(Article5.1.dGDPR)

oAuditabilityandauditscore(Articles5,22,24and25GDPR)

Informationonredress,ifrelevant:

oExplainabilityprofiling(Recital71GDPR)

oRedressorreview(Articles13.2.f,14.2.gand15GDPR)

oRedressmetrics

AIAuditing-ProposalforAIleaflets

10

Definitions

Systemnameandversion:ifany

Leafletversion:specifyifitisthefirstinstance.Leafletsshouldberevisitedwithaanymajorsystemchange,orearlierifunsupervisedmachinelearningisused.

Systemownerandsupplier/sdata:includingcontactdetailsandnameoftheteaminchargeofproduct

development,andanyexternalorganisationorpersonthathasbeencontractedtodevelopthewholeorpartsoforthealgorithmictool.

Suppliers’role:descriptionoftheroletheexternalsupplierhadinthedevelopmentofthealgorithmictool.Ifmultipleorganisationshavebeencontractedortherearemultiplecompaniesinvolvedinthedeliveryofthetool,theserelationshipsshouldbedescribedclearlyandconcisely.

Risklevel:asdefinedinAIActorotherrelevantlegislation.Ifasystemhasdifferentrisklevelsindifferentregulations,thisshouldbespecified.

Governanceroles:identificationofcontroller/s,processor/s,DPO/s,auditor/sDistributiondate:thedatethesystemstartedtooperate

Existingdocumentation:forinstancedatareusepermissions/authorizations,datasharingagreements,ethics/IRBapproval,DPAapproval,algorithmicaudit,proportionalityassessment,impactassessment,transparencyreport,academicpaper/s,GitHub/publicrepositories,etc.Informationshouldbeprovidedonwhetherthesedocumentsexist,wheretheycanbefound(iftheyarepublic)andwhois/wasresponsiblefordevelopingthem.

Descriptionofthepurposeandrole/serviceprovidedbythealgorithm,including,

-Organizationalcontext(howthealgorithmictoolisintegratedintothedecision-makingprocessandwhatinfluencethealgorithmictoolhasonit)

-Whetheritisanewrole/serviceortheautomationofanexistingrole/service

-Purposeofthealgorithmictool

-Descriptionofitsuse

-Excludeduses(potentialusesthatthetoolwasnotdesignedfortohelpavoidmisconceptionsaboutthescopeandpurposeofthetool)

-Benefits

Stakeholderinvolvement:descriptionofanystakeholderconsultationprocessesperformed,includingUXstudies

Humanrole/s:descriptionofhowsystemoutputsarehandled.Ifhumansareinvolved,descriptionoftheirroleandproceduretoapprove/rejectalgorithmicdecisions,statisticsonimpactofhumaninvolvement

Datasources/collectionmethodology:including,

-Legalbasisforaccess

-ListofsourcesandlinktoGDPRcompliancepolicies

-Timeframeandgeographicalcoverageofalldataused,includingAPIs

-Ifthedatasetsarepublic,linktotheirlocation/repositoryandsharingpolicy

-Informationonpreprocessing

-InformationonprohibitionsstatedinArticle9,GDPR

AIAuditing-ProposalforAIleaflets

11

Datatypesandcharacteristics:foreachdatasource,describedatatype(number,string,image,etc.),whetherdataispersonaland/orsensitive,andwhatinformationisincludedinthedata(age,gender,location,etc.)

PrivacybyDesign:descriptionofmeasurestakentominimize,anonymizeorotherwiseprotectpersonaldata

Datasets:name,content,formatanduseofalldatase