Agents

Authors:JuliaWiesinger,PatrickMarlowandVladimirVuskovic

Agents

Acknowledgements

ReviewersandContributors

EvanHuangEmilyXue

OlcanSercinogluSebastianRiedelSatinderBavejaAntonioGulliAnantNawalgaria

CuratorsandEditors

AntonioGulliAnantNawalgariaGraceMollison

TechnicalWriter

JoeyHaymaker

Designer

MichaelLanning

September2024 2

Endnotes

Tableofcontents

Introduction 4

Whatisanagent? 5

Themodel 6

Thetools 7

Theorchestrationlayer 7

Agentsvs.models 8

Cognitivearchitectures:Howagentsoperate 8

Tools:Ourkeystotheoutsideworld 12

Extensions 13

SampleExtensions 15

Functions 18

Usecases 21

Functionsamplecode 24

Datastores 27

Implementationandapplication 28

Toolsrecap 32

Enhancingmodelperformancewithtargetedlearning 33

AgentquickstafiwithLangChain 35

ProductionapplicationswithVefiexAIagents 38

Summary 40

42

Agents

September2024

PAGE

20

Thiscombinationofreasoning,logic,andaccesstoexternalinformationthatareallconnectedtoaGenerativeAImodelinvokestheconceptofanagent.

Introduction

Humansarefantasticatmessypatternrecognitiontasks.However,theyoftenrelyontools

-likebooks,GoogleSearch,oracalculator-tosupplementtheirpriorknowledgebeforearrivingataconclusion.Justlikehumans,GenerativeAImodelscanbetrainedtousetoolstoaccessreal-timeinformationorsuggestareal-worldaction.Forexample,amodelcanleverageadatabaseretrievaltooltoaccessspecificinformation,likeacustomer'spurchasehistory,soitcangeneratetailoredshoppingrecommendations.Alternatively,basedonauser'squery,amodelcanmakevariousAPIcallstosendanemailresponsetoacolleagueorcompleteafinancialtransactiononyourbehalf.Todoso,themodelmustnotonlyhaveaccesstoasetofexternaltools,itneedstheabilitytoplanandexecuteanytaskinaself-directedfashion.Thiscombinationofreasoning,logic,andaccesstoexternalinformationthatareallconnectedtoaGenerativeAImodelinvokestheconceptofanagent,oraprogramthatextendsbeyondthestandalonecapabilitiesofaGenerativeAImodel.Thiswhitepaperdivesintoalltheseandassociatedaspectsinmoredetail.

Whatisanagent?

Initsmostfundamentalform,aGenerativeAIagentcanbedefinedasanapplicationthatattemptstoachieveagoalbyobservingtheworldandactinguponitusingthetoolsthatithasatitsdisposal.Agentsareautonomousandcanactindependentlyofhumanintervention,especiallywhenprovidedwithpropergoalsorobjectivestheyaremeanttoachieve.Agentscanalsobeproactiveintheirapproachtoreachingtheirgoals.Evenintheabsenceofexplicitinstructionsetsfromahuman,anagentcanreasonaboutwhatitshoulddonexttoachieveitsultimategoal.WhilethenotionofagentsinAIisquitegeneralandpowerful,thiswhitepaperfocusesonthespecifictypesofagentsthatGenerativeAImodelsarecapableofbuildingatthetimeofpublication.

Inordertounderstandtheinnerworkingsofanagent,let’sfirstintroducethefoundationalcomponentsthatdrivetheagent’sbehavior,actions,anddecisionmaking.Thecombinationofthesecomponentscanbedescribedasacognitivearchitecture,andtherearemanysucharchitecturesthatcanbeachievedbythemixingandmatchingofthesecomponents.Focusingonthecorefunctionalities,therearethreeessentialcomponentsinanagent’scognitivearchitectureasshowninFigure1.

Figure1.Generalagentarchitectureandcomponents

Themodel

Inthescopeofanagent,amodelreferstothelanguagemodel(LM)thatwillbeutilizedasthecentralizeddecisionmakerforagentprocesses.ThemodelusedbyanagentcanbeoneormultipleLM’sofanysize(small/large)thatarecapableoffollowinginstructionbasedreasoningandlogicframeworks,likeReAct,Chain-of-Thought,orTree-of-Thoughts.Modelscanbegeneralpurpose,multimodalorfine-tunedbasedontheneedsofyourspecificagentarchitecture.Forbestproductionresults,youshouldleverageamodelthatbestfitsyourdesiredendapplicationand,ideally,hasbeentrainedondatasignaturesassociatedwiththetoolsthatyouplantouseinthecognitivearchitecture.It’simpofianttonotethatthemodelistypicallynottrainedwiththespecificconfigurationsettings(i.e.toolchoices,orchestration/reasoningsetup)oftheagent.However,it’spossibletofufiherrefinethemodelfortheagent’stasksbyprovidingitwithexamplesthatshowcasetheagent’scapabilities,includinginstancesoftheagentusingspecifictoolsorreasoningstepsinvariouscontexts.

Thetools

Foundationalmodels,despitetheirimpressivetextandimagegeneration,remainconstrainedbytheirinabilitytointeractwiththeoutsideworld.Toolsbridgethisgap,empoweringagentstointeractwithexternaldataandserviceswhileunlockingawiderrangeofactionsbeyondthatoftheunderlyingmodelalone.Toolscantakeavarietyofformsandhavevarying

depthsofcomplexity,buttypicallyalignwithcommonwebAPImethodslikeGET,POST,PATCH,andDELETE.Forexample,atoolcouldupdatecustomerinformationinadatabaseorfetchweatherdatatoinfluenceatravelrecommendationthattheagentisprovidingtotheuser.Withtools,agentscanaccessandprocessreal-worldinformation.Thisempowersthemtosuppofimorespecializedsystemslikeretrievalaugmentedgeneration(RAG),whichsignificantlyextendsanagent’scapabilitiesbeyondwhatthefoundationalmodelcanachieveonitsown.We’lldiscusstoolsinmoredetailbelow,butthemostimpofiantthing

tounderstandisthattoolsbridgethegapbetweentheagent’sinternalcapabilitiesandtheexternalworld,unlockingabroaderrangeofpossibilities.

Theorchestrationlayer

Theorchestrationlayerdescribesacyclicalprocessthatgovernshowtheagenttakesininformation,performssomeinternalreasoning,andusesthatreasoningtoinformitsnextactionordecision.Ingeneral,thisloopwillcontinueuntilanagenthasreacheditsgoalorastoppingpoint.Thecomplexityoftheorchestrationlayercanvarygreatlydependingontheagentandtaskit’sperforming.Someloopscanbesimplecalculationswithdecisionrules,whileothersmaycontainchainedlogic,involveadditionalmachinelearningalgorithms,orimplementotherprobabilisticreasoningtechniques.We’lldiscussmoreaboutthedetailedimplementationoftheagentorchestrationlayersinthecognitivearchitecturesection.

Agentsvs.models

Togainaclearerunderstandingofthedistinctionbetweenagentsandmodels,considerthefollowingchafi:

Models

Agents

Knowledgeislimitedtowhatisavailableintheirtrainingdata.

Knowledgeisextendedthroughtheconnectionwithexternalsystemsviatools

Singleinference/predictionbasedontheuserquery.Unlessexplicitlyimplementedforthemodel,thereisnomanagementofsession

historyorcontinuouscontext.(i.e.chathistory)

Managedsessionhistory(i.e.chathistory)toallowformultiturninference/predictionbasedonuserqueriesanddecisionsmadeintheorchestrationlayer.Inthiscontext,a‘turn’isdefinedasaninteractionbetweentheinteractingsystemandtheagent.(i.e.1incomingevent/queryand1agentresponse)

Nonativetoolimplementation.

Toolsarenativelyimplementedinagentarchitecture.

Nonativelogiclayerimplemented.Userscanformpromptsassimplequestionsorusereasoningframeworks(CoT,ReAct,etc.)toformcomplexpromptstoguidethemodelinprediction.

NativecognitivearchitecturethatusesreasoningframeworkslikeCoT,ReAct,orotherpre-builtagentframeworkslikeLangChain.

Cognitivearchitectures:Howagentsoperate

Imagineachefinabusykitchen.Theirgoalistocreatedeliciousdishesforrestaurantpatronswhichinvolvessomecycleofplanning,execution,andadjustment.

Theygatherinformation,likethepatron’sorderandwhatingredientsareinthepantryandrefrigerator.

Theyperformsomeinternalreasoningaboutwhatdishesandflavorprofilestheycancreatebasedontheinformationtheyhavejustgathered.

Theytakeactiontocreatethedish:choppingvegetables,blendingspices,searingmeat.

Ateachstageintheprocessthechefmakesadjustmentsasneeded,refiningtheirplanasingredientsaredepletedorcustomerfeedbackisreceived,andusesthesetofpreviousoutcomestodeterminethenextplanofaction.Thiscycleofinformationintake,planning,executing,andadjustingdescribesauniquecognitivearchitecturethatthechefemploystoreachtheirgoal.

Justlikethechef,agentscanusecognitivearchitecturestoreachtheirendgoalsbyiterativelyprocessinginformation,makinginformeddecisions,andrefiningnextactionsbasedonpreviousoutputs.Atthecoreofagentcognitivearchitecturesliestheorchestrationlayer,responsibleformaintainingmemory,state,reasoningandplanning.Itusestherapidlyevolvingfieldofpromptengineeringandassociatedframeworkstoguidereasoningandplanning,enablingtheagenttointeractmoreeffectivelywithitsenvironmentandcompletetasks.Researchintheareaofpromptengineeringframeworksandtaskplanningforlanguagemodelsisrapidlyevolving,yieldingavarietyofpromisingapproaches.Whilenotanexhaustivelist,theseareafewofthemostpopularframeworksandreasoningtechniquesavailableatthetimeofthispublication:

ReAct,apromptengineeringframeworkthatprovidesathoughtprocessstrategyforlanguagemodelstoReasonandtakeactiononauserquery,withorwithoutin-contextexamples.ReActpromptinghasshowntooutperformseveralSOTAbaselinesandimprovehumaninteroperabilityandtrustwofihinessofLLMs.

Chain-of-Thought(CoT),apromptengineeringframeworkthatenablesreasoningcapabilitiesthroughintermediatesteps.Therearevarioussub-techniquesofCoTincludingself-consistency,active-prompt,andmultimodalCoTthateachhavestrengthsandweaknessesdependingonthespecificapplication.

Tree-of-thoughts(ToT),,apromptengineeringframeworkthatiswellsuitedforexplorationorstrategiclookaheadtasks.Itgeneralizesoverchain-of-thoughtpromptingandallowsthemodeltoexplorevariousthoughtchainsthatserveasintermediatestepsforgeneralproblemsolvingwithlanguagemodels.

Agentscanutilizeoneoftheabovereasoningtechniques,ormanyothertechniques,tochoosethenextbestactionforthegivenuserrequest.Forexample,let’sconsideranagentthatisprogrammedtousetheReActframeworktochoosethecorrectactionsandtoolsfortheuserquery.Thesequenceofeventsmightgosomethinglikethis:

Usersendsquerytotheagent

AgentbeginstheReActsequence

Theagentprovidesaprompttothemodel,askingittogenerateoneofthenextReActstepsanditscorrespondingoutput:

Question:Theinputquestionfromtheuserquery,providedwiththeprompt

Thought:Themodel’sthoughtsaboutwhatitshoulddonext

Action:Themodel’sdecisiononwhatactiontotakenext

Thisiswheretoolchoicecanoccur

Forexample,anactioncouldbeoneof[Flights,Search,Code,None],wherethefirst3representaknowntoolthatthemodelcanchoose,andthelastrepresents“notoolchoice”

Actioninput:Themodel’sdecisiononwhatinputstoprovidetothetool(ifany)

Observation:Theresultoftheaction/actioninputsequence

Thisthought/action/actioninput/observationcouldrepeatN-timesasneeded

Finalanswer:Themodel’sfinalanswertoprovidetotheoriginaluserquery

TheReActloopconcludesandafinalanswerisprovidedbacktotheuser

Figure2.ExampleagentwithReActreasoningintheorchestrationlayer

AsshowninFigure2,themodel,tools,andagentconfigurationworktogethertoprovideagrounded,conciseresponsebacktotheuserbasedontheuser’soriginalquery.Whilethemodelcouldhaveguessedatananswer(hallucinated)basedonitspriorknowledge,itinsteadusedatool(Flights)tosearchforreal-timeexternalinformation.Thisadditional

informationwasprovidedtothemodel,allowingittomakeamoreinformeddecisionbasedonrealfactualdataandtosummarizethisinformationbacktotheuser.

Insummary,thequalityofagentresponsescanbetieddirectlytothemodel’sabilitytoreasonandactaboutthesevarioustasks,includingtheabilitytoselecttherighttools,andhowwellthattoolshasbeendefined.Likeachefcraftingadishwithfreshingredientsandattentivetocustomerfeedback,agentsrelyonsoundreasoningandreliableinformationtodeliveroptimalresults.Inthenextsection,we’lldiveintothevariouswaysagentsconnectwithfreshdata.

Tools:Ourkeystotheoutsideworld

Whilelanguagemodelsexcelatprocessinginformation,theylacktheabilitytodirectlyperceiveandinfluencetherealworld.Thislimitstheirusefulnessinsituationsrequiringinteractionwithexternalsystemsordata.Thismeansthat,inasense,alanguagemodelisonlyasgoodaswhatithaslearnedfromitstrainingdata.Butregardlessofhowmuch

datawethrowatamodel,theystilllackthefundamentalabilitytointeractwiththeoutsideworld.Sohowcanweempowerourmodelstohavereal-time,context-awareinteractionwithexternalsystems?Functions,Extensions,DataStoresandPluginsareallwaystoprovidethiscriticalcapabilitytothemodel.

Whiletheygobymanynames,toolsarewhatcreatealinkbetweenourfoundationalmodelsandtheoutsideworld.Thislinktoexternalsystemsanddataallowsouragenttoperformawidervarietyoftasksanddosowithmoreaccuracyandreliability.Forinstance,toolscanenableagentstoadjustsmafihomesettings,updatecalendars,fetchuserinformationfromadatabase,orsendemailsbasedonaspecificsetofinstructions.

Asofthedateofthispublication,therearethreeprimarytooltypesthatGooglemodelsareabletointeractwith:Extensions,Functions,andDataStores.Byequippingagentswithtools,weunlockavastpotentialforthemtonotonlyunderstandtheworldbutalsoactuponit,openingdoorstoamyriadofnewapplicationsandpossibilities.

Extensions

TheeasiestwaytounderstandExtensionsistothinkofthemasbridgingthegapbetweenanAPIandanagentinastandardizedway,allowingagentstoseamlesslyexecuteAPIs

regardlessoftheirunderlyingimplementation.Let’ssaythatyou’vebuiltanagentwithagoalofhelpingusersbookflights.YouknowthatyouwanttousetheGoogleFlightsAPItoretrieveflightinformation,butyou’renotsurehowyou’regoingtogetyouragenttomakecallstothisAPIendpoint.

Figure3.HowdoAgentsinteractwithExternalAPIs?

Oneapproachcouldbetoimplementcustomcodethatwouldtaketheincominguserquery,parsethequeryforrelevantinformation,thenmaketheAPIcall.Forexample,inaflightbookingusecaseausermightstate“IwanttobookaflightfromAustintoZurich.”Inthisscenario,ourcustomcodesolutionwouldneedtoextract“Austin”and“Zurich”asrelevantentitiesfromtheuserquerybeforeattemptingtomaketheAPIcall.Butwhathappensiftheusersays“IwanttobookaflighttoZurich”andneverprovidesadepafiurecity?TheAPIcallwouldfailwithouttherequireddataandmorecodewouldneedtobeimplementedinordertocatchedgeandcornercaseslikethis.Thisapproachisnotscalableandcouldeasilybreakinanyscenariothatfallsoutsideoftheimplementedcustomcode.

AmoreresilientapproachwouldbetouseanExtension.AnExtensionbridgesthegapbetweenanagentandanAPIby:

TeachingtheagenthowtousetheAPIendpointusingexamples.

TeachingtheagentwhatargumentsorparametersareneededtosuccessfullycalltheAPIendpoint.

Figure4.ExtensionsconnectAgentstoExternalAPIs

Extensionscanbecraftedindependentlyoftheagent,butshouldbeprovidedaspafioftheagent’sconfiguration.TheagentusesthemodelandexamplesatruntimetodecidewhichExtension,ifany,wouldbesuitableforsolvingtheuser’squery.ThishighlightsakeystrengthofExtensions,theirbuilt-inexampletypes,thatallowtheagenttodynamicallyselectthemostappropriateExtensionforthetask.

Figure5.1-to-manyrelationshipbetweenAgents,ExtensionsandAPIs

ThinkofthisthesamewaythatasoftwaredeveloperdecideswhichAPIendpointstousewhilesolvingandsolutioningforauser’sproblem.Iftheuserwantstobookaflight,thedevelopermightusetheGoogleFlightsAPI.Iftheuserwantstoknowwherethenearestcoffeeshopisrelativetotheirlocation,thedevelopermightusetheGoogleMapsAPI.Inthissameway,theagent/modelstackusesasetofknownExtensionstodecidewhichonewillbethebestfitfortheuser’squery.Ifyou’dliketoseeExtensionsinaction,youcantrythemoutontheGeminiapplicationbygoingtoSettings>Extensionsandthenenablinganyyouwouldliketotest.Forexample,youcouldenabletheGoogleFlightsextensionthenaskGemini“ShowmeflightsfromAustintoZurichleavingnextFriday.”

SampleExtensions

TosimplifytheusageofExtensions,Googleprovidessomeoutoftheboxextensionsthatcanbequicklyimpofiedintoyourprojectandusedwithminimalconfigurations.Forexample,theCodeInterpreterextensioninSnippet1allowsyoutogenerateandrunPythoncodefromanaturallanguagedescription.

Python

importvertexaiimportpprint

PROJECT_ID="YOUR_PROJECT_ID"

REGION="us-central1"vertexai.init(project=PROJECT_ID,location=REGION)

fromvertexai.preview.extensionsimportExtension

extension_code_interpreter=Extension.from_hub("code_interpreter")CODE_QUERY="""WriteapythonmethodtoinvertabinarytreeinO(n)time."""

response=extension_code_interpreter.execute(operation_id="generate_and_execute",operation_params={"query":CODE_QUERY}

)

print("GeneratedCode:")pprint.pprint({response['generated_code']})

#Theabovesnippetwillgeneratethefollowingcode.

```

GeneratedCode:

classTreeNode:

definit(self,val=0,left=None,right=None):self.val=val

self.left=leftself.right=right

Continuesnextpage...

Python

definvert_binary_tree(root):"""

Invertsabinarytree.Args:

root:Therootofthebinarytree.

Returns:

Therootoftheinvertedbinarytree.

"""

ifnotroot:

returnNone

#Swaptheleftandrightchildrenrecursivelyroot.left,root.right=

invert_binary_tree(root.right),invert_binary_tree(root.left)

returnroot

#Exampleusage:

#Constructasamplebinarytreeroot=TreeNode(4)

root.left=TreeNode(2)root.right=TreeNode(7)root.left.left=TreeNode(1)root.left.right=TreeNode(3)root.right.left=TreeNode(6)root.right.right=TreeNode(9)

#Invertthebinarytree

inverted_root=invert_binary_tree(root)

```

Snippet1.CodeInterpreterExtensioncangenerateandrunPythoncode

Tosummarize,Extensionsprovideawayforagentstoperceive,interact,andinfluencetheoutsideworldinamyriadofways.TheselectionandinvocationoftheseExtensionsisguidedbytheuseofExamples,allofwhicharedefinedaspafioftheExtensionconfiguration.

Functions

Intheworldofsoftwareengineering,functionsaredefinedasself-containedmodulesofcodethataccomplishaspecifictaskandcanbereusedasneeded.Whenasoftwaredeveloperiswritingaprogram,theywilloftencreatemanyfunctionstodovarioustasks.

Theywillalsodefinethelogicforwhentocallfunction_aversusfunction_b,aswellastheexpectedinputsandoutputs.

Functionsworkverysimilarlyintheworldofagents,butwecanreplacethesoftwaredeveloperwithamodel.AmodelcantakeasetofknownfunctionsanddecidewhentouseeachFunctionandwhatargumentstheFunctionneedsbasedonitsspecification.FunctionsdifferfromExtensionsinafewways,mostnotably:

AmodeloutputsaFunctionanditsarguments,butdoesn’tmakealiveAPIcall.

Functionsareexecutedontheclient-side,whileExtensionsareexecutedontheagent-side.

UsingourGoogleFlightsexampleagain,asimplesetupforfunctionsmightlookliketheexampleinFigure7.

Figure7.HowdofunctionsinteractwithexternalAPIs?

NotethatthemaindifferencehereisthatneithertheFunctionnortheagentinteractdirectlywiththeGoogleFlightsAPI.SohowdoestheAPIcallactuallyhappen?

Withfunctions,thelogicandexecutionofcallingtheactualAPIendpointisoffloadedawayfromtheagentandbacktotheclient-sideapplicationasseeninFigure8andFigure9below.Thisoffersthedevelopermoregranularcontrolovertheflowofdataintheapplication.TherearemanyreasonswhyaDevelopermightchoosetousefunctionsoverExtensions,butafewcommonusecasesare:

APIcallsneedtobemadeatanotherlayeroftheapplicationstack,outsideofthedirectagentarchitectureflow(e.g.amiddlewaresystem,afrontendframework,etc.)

SecurityorAuthenticationrestrictionsthatpreventtheagentfromcallinganAPIdirectly(e.gAPIisnotexposedtotheinternet,ornon-accessiblebyagentinfrastructure)

Timingororder-of-operationsconstraintsthatpreventtheagentfrommakingAPIcallsinreal-time.(i.e.batchoperations,human-in-the-loopreview,etc.)

AdditionaldatatransformationlogicneedstobeappliedtotheAPIResponsethattheagentcannotperform.Forexample,consideranAPIendpointthatdoesn’tprovideafilteringmechanismforlimitingthenumberofresultsreturned.UsingFunctionsontheclient-sideprovidesthedeveloperadditionaloppofiunitiestomakethesetransformations.

ThedeveloperwantstoiterateonagentdevelopmentwithoutdeployingadditionalinfrastructurefortheAPIendpoints(i.e.FunctionCallingcanactlike“stubbing”ofAPIs)

WhilethedifferenceininternalarchitecturebetweenthetwoapproachesissubtleasseeninFigure8,theadditionalcontrolanddecoupleddependencyonexternalinfrastructuremakesFunctionCallinganappealingoptionfortheDeveloper.

Figure8.Delineatingclientvs.agentsidecontrolforextensionsandfunctioncalling

Usecases

Amodelcanbeusedtoinvokefunctionsinordertohandlecomplex,client-sideexecutionflowsfortheenduser,wheretheagentDevelopermightnotwantthelanguagemodeltomanagetheAPIexecution(asisthecasewithExtensions).Let’sconsiderthefollowingexamplewhereanagentisbeingtrainedasatravelconciergetointeractwithusersthatwanttobookvacationtrips.Thegoalistogettheagenttoproducealistofcitiesthatwecanuseinourmiddlewareapplicationtodownloadimages,data,etc.fortheuser’stripplanning.Ausermightsaysomethinglike:

I’dliketotakeaskitripwithmyfamilybutI’mnotsurewheretogo.

Inatypicalprompttothemodel,theoutputmightlooklikethefollowing:Sure,here’salistofcitiesthatyoucanconsiderforfamilyskitrips:

CrestedButte,Colorado,USA

Whistler,BC,Canada

Zermatt,Switzerland

Whiletheaboveoutputcontainsthedatathatweneed(citynames),theformatisn’tidealforparsing.WithFunctionCalling,wecanteachamodeltoformatthisoutputinastructuredstyle(likeJSON)that’smoreconvenientforanothersystemtoparse.Giventhesameinputpromptfromtheuser,anexampleJSONoutputfromaFunctionmightlooklikeSnippet

5instead.

Unset

function_call{

name:"display_cities"args:{

"cities":["CrestedButte","Whistler","Zermatt"],"preferences":"skiing"

}

}

Snippet5.SampleFunctionCallpayloadfordisplayingalistofcitiesanduserpreferences

ThisJSONpayloadisgeneratedbythemodel,andthensenttoourClient-sideservertodowhateverwewouldliketodowithit.Inthisspecificcase,we’llcalltheGooglePlacesAPItotakethecitiesprovidedbythemodelandlookupImages,thenprovidethemasformattedrichcontentbacktoourUser.ConsiderthissequencediagraminFigure9showingtheaboveinteractioninstepbystepdetail.

Figure9.SequencediagramshowingthelifecycleofaFunctionCall

TheresultoftheexampleinFigure9isthatthemodelisleveragedto“fillintheblanks”withtheparametersrequiredfortheClientsideUItomakethecalltotheGooglePlacesAPI.TheClientsideUImanagestheactualAPIcallusingtheparametersprovidedbythemodelinthereturnedFunction.ThisisjustoneusecaseforFunctionCalling,buttherearemanyotherscenariostoconsiderlike:

Youwantalanguagemodeltosuggestafunctionthatyoucanuseinyourcode,butyoudon'twanttoincludecredentialsinyourcode.Becausefunctioncallingdoesn'trunthefunction,youdon'tneedtoincludecredentialsinyourcodewiththefunctioninformation.

Youarerunningasynchronousoperationsthatcantakemorethanafewseconds.Thesescenariosworkwellwithfunctioncallingbecauseit'sanasynchronousoperation.

Youwanttorunfunctionsonadevicethat'sdifferentfromthesystemproducingthefunctioncallsandtheirarguments.

OnekeythingtorememberaboutfunctionsisthattheyaremeanttoofferthedevelopermuchmorecontrolovernotonlytheexecutionofAPIcalls,butalsotheentireflowofdataintheapplicationasawhole.IntheexampleinFigure9,thedeveloperchosetonotreturnAPIinformationbacktotheagentasitwasnotpefiinentforfutureactionstheagentmighttake.However,basedonthearchitectureoftheapplication,itmaymakesensetoreturntheexternalAPIcalldatatotheagentinordertoinfluencefuturereasoning,logic,andactionchoices.Ultimately,itisuptotheapplicationdevelopertochoosewhatisrightforthespecificapplication.

Functionsamplecode

Toachievetheaboveoutputfromourskivacationscenario,let’sbuildouteachofthecomponentstomakethisworkwithourgemini-1.5-flash-001model.

First,we’lldefineourdisplay_citiesfunctionasasimplePythonmethod.

Python

defdisplay_cities(cities:list[str],preferences:Optional[str]=None):"""Providesalistofcitiesbasedontheuser'ssearchqueryandpreferences.

Args:

preferences(str):Theuser'spreferencesforthesearch,likeskiing,beach,restaurants,bbq,etc.

cities(list[str]):Thelistofcitiesbeingrecommendedtotheuser.

Returns:

list[str]:Thelistofcitiesbeingrecommendedtotheuser.

"""

returncities

Snippet6.Samplepythonmethodforafunctionthatwilldisplayalistofcities.

Next,we’llinstantiateourmodel,buildtheTool,thenpassinouruser’squeryandtoolstothemodel.Executingthecodebelowwouldresultintheoutputasseenatthebottomofthecodesni