Wide layout

Table of contents

Agentic AI

Introducing Agentic AI Workflows

  • An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.

  • The output of exciting models improve significantly in agent mode.

Degrees of Autonomy

Less autonomous agent

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;

    prompt[Prompt] -->|Write an essay about\nblack holes| llm1["LLM"]:::blue
    llm1 --> websearch["Web Search"]:::green
    websearch --> webfetch["Web Fetch"]:::green
    webfetch --> llm2["LLM"]:::blue
    llm2 --> output[Output]
  • All steps predetermined

  • All tool use hard coded

  • Autonomy is in text generation

Semi autonomous agent

  • Can make some decisions

  • Can decide to use a predefined tool

More autonomous agent

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    subgraph searchTools [Tools]
        toolWebSearch["web search"]:::green
        toolNews["news"]:::green
        toolArxiv["arXiv"]:::green
    end

    subgraph fetchTools [Tools]
        toolWebFetch["web fetch"]:::green
        toolPdfToText["pdf to text"]:::green
    end

    llm1["LLM"]:::blue
    websearch["Web Search"]:::green
    llm2["LLM"]:::blue
    webfetch["Web Fetch"]:::green
    llm3["LLM"]:::blue
    llm4["LLM"]:::blue
    output[Output]

    llm1Note["Search web"]:::note
    llm2Note["Fetch 5 best sources"]:::note
    llm3Note["Write essay draft"]:::note
    llm4Note["Reflect and improve draft"]:::note

    prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output

    searchTools --> llm1
    fetchTools --> llm2
    llm1 -.- llm1Note
    llm2 -.- llm2Note
    llm3 -.- llm3Note
    llm4 -.- llm4Note
  • Can make many decisions autonomously, including when to execute a step or order or execution of steps.

  • Can create tools on the fly

Example:

Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"

  1. Check inventory for black jeans

  2. Check inventory for blue jeans

  3. Respond to customer

The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.

Parallelism and Modularity

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    subgraph searches [3 parallel searches]
        search1["LLM + web search"]:::green
        search2["LLM + news search"]:::red
        search3["LLM + web search"]:::green
    end
        search2Note["Modularity:\nMight swap to another web search provider / news search engine"]:::note

    subgraph fetches [9 parallel web page downloads]
        fetch1["LLM + web fetch"]:::green
        fetch2["LLM + web fetch"]:::green
        fetch3["LLM + web fetch"]:::green
        fetch4["LLM + web fetch"]:::green
        fetch5["LLM + web fetch"]:::green
        fetch6["LLM + web fetch"]:::green
        fetch7["LLM + web fetch"]:::green
        fetch8["LLM + web fetch"]:::green
        fetch9["LLM + web fetch"]:::green
    end

    prompt -->|Write an essay about<br/>black holes| LLM:::blue --> search1 & search2 & search3
    search1 --> fetch1 & fetch2 & fetch3
    search2 --> fetch4 & fetch5 & fetch6
    search3 --> fetch7 & fetch8 & fetch9

    fetches --> |Fetch top 3 from each run| LLM2[LLM]:::blue --> output

    search2 -.- search2Note
  • Parallelisation, lets us execute steps faster.

  • Modularisation, lets us swap out models or tools to improve output.

Task decomposition

  • Start by asking, "How would a person solve this problem step by step?"

  • Break down your problem into smaller, more manageable steps.

  • Can this step be implemented with an LLM or and API call that we have access to?

  • If its not, then I would ask myself "How would a Human solve this", can decompose it further?

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;

    subgraph direct [Direct generation]
        direction TB
        direct1["1. Write an essay on topic X"]:::red
    end

    subgraph threeStep [3-step workflow]
        direction TB
        three1["1. Write an essay outline on topic X"]:::orange
        three2["2. Search web"]:::orange
        three3["3. Write the essay"]:::orange
        three1 --> three2 --> three3
    end

    subgraph fiveStep [5-step workflow]
        direction TB
        five1["1. Write an essay outline on topic X"]:::green
        five2["2. Search web"]:::green
        five3["3. Write the first draft"]:::green
        five4["4. Consider what parts need revision"]:::green
        five5["5. Revise your draft"]:::green
        five1 --> five2 --> five3 --> five4 --> five5
    end

    direct -->|Not good enough...| threeStep -->|Still not good enough...| fiveStep

Building blocks for Agentic workflows

Building block
Examples
Use cases

Models

LLMs

Text generation, tool use, information extraction

Other AI models

PDF-to-text, text-to-speech, image analysis

Tools

API

Web search, get real-time data, send email, check calendar

Information retrieval

Databases, Retrieval Augmented Generation (RAG)

Code execution

Basic calculator, data analysis

Evaluating agentic AI (evals)

  1. Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).

  2. Do an End-to-End and Component Level evals.

  3. Examine traces to perform error analysis.

Design patterns for building agents

Reflection

mermaid
sequenceDiagram
    actor User
    participant Coder as Coder Agent (LLM)
    participant Critic as Critic Agent (LLM)

    User->>Coder: Please write code for {task}
    Coder->>Critic: def do_task(x): ...
    Note over Coder,Critic: Here's code intended for {task}.<br/>Check it carefully for correctness, style and efficiency,<br/>and give constructive criticism for how to improve it.
    Critic-->>Coder: There's a bug on line 5. Fix it by ...
    Coder->>Critic: def do_task_v2(x): ...
    Critic-->>Coder: It failed Unit Test 3. Try changing ...
    Coder->>User: def do_task_v3(x): ...

Tool use

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;

    subgraph webSearchExample [Web search tool]
        searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
        searchLlm["LLM"]:::blue
        searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
        searchAnswer[Answer]
        searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
    end

    subgraph codeExecExample [Code execution tool]
        codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
        codeLlm["LLM"]:::blue
        codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
        codeAnswer["Answer: ~$225.22"]
        codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
    end

    subgraph toolCategories [Tool categories]
        direction TB
        analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
        infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
        productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
        images["Images<br/>Image generation, Image captioning, OCR"]:::orange
    end

Planning

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
    planner["LLM:<br/>plans the steps"]:::blue

    subgraph plan [Plan the LLM executes]
        inputImage[/example.jpg/]
        poseStep["Pose determination"]:::green
        pose[/Pose skeleton/]
        imageStep["Pose-to-image"]:::green
        newImage[/Image of girl reading/]
        captionStep["Image-to-text"]:::green
        caption[/"A girl is sitting on a bed reading a book."/]
        speechStep["Text-to-speech"]:::green
        audio[/Audio/]

        poseStepNote["openpose model"]:::note
        imageStepNote["google/vit model"]:::note
        captionStepNote["vit-gpt2 model"]:::note
        speechStepNote["fastspeech model"]:::note
    end

    request --> planner --> inputImage
    inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio

    poseStep -.- poseStepNote
    imageStep -.- imageStepNote
    captionStep -.- captionStepNote
    speechStep -.- speechStepNote

Multi-agent collaboration

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;

    request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
    manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
    output[Brochure]

    subgraph team [Specialist agents]
        researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
        designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
        writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
    end

    subgraph tools [Tools]
        webSearch["Web search"]:::green
        imageGen["Image generation"]:::green
    end

    request --> manager <--> team --> tools
    manager --> output

Reflection Design Pattern

mermaid
flowchart LR
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    prompt["Write code for task X"]:::red
    llm1["LLM:<br/>write first draft"]:::blue
    codeV1["code V1:<br/>def do_task(args): ..."]

    subgraph reflection [Reflection]
        llm2["LLM 2:<br/>check for bugs and write<br/>improved second draft"]:::blue
        llm2Note["Can be a different LLM,<br/>e.g. a reasoning model"]:::note
    end

    codeV2["code V2:<br/>def do_task_v2(args): ..."]

    prompt --> llm1 --> codeV1 --> llm2 --> codeV2

    llm2 -.- llm2Note

    style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5

A reasoning model aka thinking model, is better at finding bugs.

Reflection with external feedback

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    prompt["Write code for task X"]:::red
    llm1["LLM:<br/>write first draft"]:::blue
    codeV1["code V1:<br/>def do_task(args): ..."]

    subgraph reflection [Reflection with external feedback]
        execute["Execute code"]:::green
        feedback["Code output, errors"]
        feedbackNote["SyntaxError: unterminated string<br/>literal (detected at line 1)"]:::note
        llm2["LLM:<br/>reflect on output or errors and<br/>write improved second draft"]:::blue
        llm2Note["You may also want to pass in the<br/>first draft and the conversation so far"]:::note
    end

    codeV2["code V2:<br/>def do_task_v2(args): ..."]

    prompt --> llm1 --> codeV1 --> execute --> feedback --> llm2 --> codeV2

    feedback -.- feedbackNote
    llm2 -.- llm2Note

    style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5

Why not direct generation?

No title

Research shows that Reflection consistently out performs direct generation.

Zero-shot (no examples)

text
Convert to MM/DD/YYYY format

Input: {input_date}

One-shot (single example)

text
Convert to MM/DD/YYYY format

Input: Jan 1st, 2025
Output: 01/01/2025

Input: {input_date}

Two-shot (two examples), or few-shot (multiple examples)

text
Convert to MM/DD/YYYY format

Input: Jan 1st, 2025
Output: 01/01/2025

Input: 21st June, 2025
Output: 06/21/2025

Input: {input_date}

Examples of reflection prompt

Brainstorming domain names

text
Review the domain names you suggested.

Check if each name is easy to pronounce and thus easy to spread via word of mouth.

Consider whether each name might mean something negative in other languages.

Then output a shortlist of only the names that meet these criteria.

Improving email

text
Review the email first draft.

Check that the tone is professional and look for phrases that could be considered rude or insensitive.

Verify all facts, dates, and promises are accurate.

Then write the next draft of the email.

Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.

Evaluating the impact of Reflection

Create a dataset of prompts and answers (Objective Reflection)

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;

    prompt["Which color of product has<br/>the highest total sales?"]:::red
    llm1["LLM:<br/>generate SQL query"]:::blue

    subgraph reflection [Reflection]
        llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
    end

    execute["Execute V2 query"]:::green
    results["V2 query results"]
    llm3["LLM:<br/>answer question"]:::blue

    prompt --> llm1 --> llm2 --> execute --> results --> llm3

    subgraph evals [Evals]
        dataset[("Dataset:<br/>prompts + ground truth answers")]
        compare["Compare answer with<br/>ground truth (code)"]:::green
        accuracy["Accuracy<br/>no reflection vs with reflection"]
        dataset --> compare --> accuracy
    end

    dataset -.->|prompts| prompt
    llm3 --> compare

    style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
    style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5

Run the dataset each time you change the reflection prompt, and compare against the ground truth:

Prompt
Ground truth answer
No reflection
With reflection

Number of items sold in May 2025?

1201

980

1201

Most expensive item?

Airflow sneaker

Airflow sneaker

Airflow sneaker

How many styles carried?

14

14

14

Accuracy

87% correct

95% correct

LLM as judge (Subjective Reflection)

No title

When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.

Comparing two outputs

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    plotV1[/plot.png/]
    plotV2[/plot_v2.png/]
    judge["LLM:<br/>which image is better?"]:::blue
    judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note

    plotV1 & plotV2 --> judge
    judge -.- judgeNote

Known issues with asking an LLM to compare outputs:

  • Answers are often not very good.

  • Position bias: the LLM tends to pick whichever option is shown first (A over B).

Grading with a rubric

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;

    plot[/plot.png or plot_v2.png/]
    judge["LLM:<br/>score against rubric"]:::blue
    scores["JSON scores<br/>e.g. 6 / 8"]

    plot --> judge --> scores
text
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.

1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...
Input
No reflection
With reflection

User query 1

4

6

User query 2

5

8

...

...

...

User query 10

5

7

Why a rubric is better than asking "which is better?"

  • Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.

  • No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.

  • Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.

  • Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.

  • Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.

Using external feedback

Challenge
Example
Source of feedback

Mentioning competitors

Our company's shoes are better than RivalCo

Pattern matching for competitor names

Fact checking an essay

The Taj Mahal was built in 1648

Web search results

LLM won't follow output length guidelines

Essay is over word limit

Word count tool

Tool Use

LLM can choose tools when appropriate

mermaid
flowchart TD
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    subgraph Tools
        get_current_time["fn_get_current_time"]:::green
        interest_calc["fn_interest_calc"]:::green
        check_calendar["fn_check_calendar"]:::green
        eval["fn_eval"]:::green
    end

    Tools -.-> LLM

    prompt:::blue --> |What time is it?| LLM --> |Its 3:20pm| response

How to build tools for LLM use

Primitive function keyword

py
from datetime import datetime

def get_current_time(): -> str:
    return datetime.now().strftime('%I:%M%p')
mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    subgraph Tools
        get_current_time:::green
    end


    tool_prompt["You have access to a tool called\nget_current_time. To use it, return the following exactly:\nFunction:\nget_current_time"]:::note -.- LLM

    prompt[Prompt] --> |What time is it?| LLM --> |Function:\nget_current_time| parser --> |conversation history\ntime| LLM1[LLM] --> |Its 8am| output

    Tools -.-> parser

AI suite

py
from datetime import datetime

def get_current_time(): -> str:
    """Returns the current time as a string"""
    return datetime.now().strftime('%I:%M%p')
py
import aisuite as ai
client = ai.Client()

response = client.chat.completions.create(
    model="openai:gpt-4o",
    messages=messages,
    tools=[get_current_time]
    max_turns=5
)
mermaid
sequenceDiagram
    autonumber
    actor User as User Script
    participant Suite as aisuite.Client
    participant Tool as Local Function (get_current_time)
    participant LLM as OpenAI (gpt-4o)

    User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
    activate Suite

    Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema

    loop Up to max_turns
        Suite->>LLM: Send messages + tool schemas
        activate LLM
        LLM-->>Suite: Return response (Tool Call requested)
        deactivate LLM

        alt Tool Call requested
            Suite->>Tool: Execute get_current_time()
            activate Tool
            Tool-->>Suite: Return "10:32PM"
            deactivate Tool
            Suite->>Suite: Append Tool Call & Tool Result to messages
        else Normal text response (stop)
            Suite-->>User: Return final completion response
        end
    end

    deactivate Suite
  • Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.

  • Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.

  • Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.

Code execution

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;


    sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/><br/>Return your answer as python code delimited<br/>with &lt;execute_python&gt; and &lt;/execute_python&gt; tags."]:::note
    query["What's the<br/>square root of 2?"]:::red
    llm1["LLM"]
    output["output<br/><br/>&lt;execute_python&gt;<br/>import math<br/>print(math.sqrt(2))<br/>&lt;/execute_python&gt;"]
    exec["Execute code<br/><br/>exec(output)"]:::green
    result["result<br/><br/>1.4142135623730951"]
    llm2["LLM<br/><br/>The square root of 2<br/>is approximately 1.4142."]

    sysPrompt -.-> llm1
    query --> llm1 --> output --> exec --> result --> llm2
  • Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).

  • Use sandbox tools like docker, E2B to safegaurd your environemnt.

MCP

  • MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.

  • MCP clients, allows you to fetch data from MCP servers.

  • MCP servers, allows access to data.

Practical tips for builiding Agentic AI

Evaluations

Examples

Invoice processor evals

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
    invoice["Invoice PDF"]:::red
    pdf["PDF to text"]:::green
    llm["LLM"]
    subgraph tools["Tools"]
        toolDb["update<br/>database"]:::green
    end
    update["update<br/>database"]:::green
    record["Record<br/>created!"]

    test_data[(Test Data)]
    fields -.-> llm

    subgraph metrics["Eval"]
        date_eval["Evaluate date format"]
        update_metric["Update Metric"]
        metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
    end

    test_data -->|20 examples| invoice
    invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
    llm -.-> tools
    update_metric -.- metric_note

Research agent evals

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    query["User query"]:::red
    subgraph tools["Tools"]
        t_search["web search"]:::green
        t_fetch["web fetch"]:::green
        t_pdf["PDF to text"]:::green
    end
    agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]

    query --> agent --> llm_as_judge
    agent -.-> tools

    test_data[(Test Data)]
    test_data -->|20 topic examples| query

    subgraph metrics["Eval"]
        llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
        rubric_score["Rubric Score<br/>0-5"]:::green
        judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
        rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
    end

    gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note

    llm_as_judge --> rubric_score
    gold -..-> llm_as_judge
    judge_prompt -.- llm_as_judge
    rubric_score -.- rubric_note
    rubric_score --> essay["Essay draft"]

Two "axes" of evaluation

Evaluate with code (objective)
LLM-as-judge (subjective)

Per example ground truth

Checking invoice date extraction
if (extracted_date == actual_date): num_correct += 1

Counting gold-standard talking points
"Count the number of gold standard points in the following text..."

No per example ground truth

Checking marketing copy length
if len(text) <= 10: num_correct += 1

Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."

Tips and Trics

  • Quick and dirty evals are okay to start.

  • Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.

  • Look for places where performance is worst than humans

Error Analysis Process

Setting up Traces

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef ErrorNote fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000,stroke-dasharray: 5 5,color:#333;

    subgraph searchTools [Tools]
        toolWebSearch["web search"]:::green
        toolNews["news"]:::green
        toolArxiv["arXiv"]:::green
    end

    subgraph fetchTools [Tools]
        toolWebFetch["web fetch"]:::green
        toolPdfToText["pdf to text"]:::green
    end

    llm1["LLM"]:::blue
    websearch["Web Search"]:::green
    llm2["LLM"]:::blue
    webfetch["Web Fetch"]:::green
    llm3["LLM"]:::blue
    llm4["LLM"]:::blue
    output[Output]

    llm1Note["<b>Bad search terms?</b><br/>Are the queries too vague,<br/>too narrow or off-topic?"]:::ErrorNote
    llm2Note["<b>Low quality search results?</b><br/>Do the results actually cover<br/>the topic, or are they noisy?"]:::ErrorNote
    llm3Note["<b>Poor selection of URLs?</b><br/>Did it pick the most relevant<br/>and credible sources?"]:::ErrorNote
    llm4Note["<b>Bad reasoning over texts?</b><br/>Does the essay miss, distort or<br/>invent points from the sources?"]:::ErrorNote

    prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output

    searchTools --> llm1
    fetchTools --> llm2
    llm1 -.- llm1Note
    llm2 -.- llm2Note
    llm3 -.- llm3Note
    llm4 -.- llm4Note
  • Try to identify the points of failure in the pipeline.

  • See if a human expert would do it differently.

  • See intermidate outputs for issues.

Counting up the errors

Prompt
Search terms
Search results
Picking 5 best sources
...
...

Recent developments in black hole science

Too many blog posts, not enough papers

Renting vs buying a home in Seattle

Missed well-known blog

Robotics for harvesting fruit

Terms too generic

Website for elementary school students

Batteries for electric vehicles

Only selected US-based companies

Missed magazine

5% dissatisfaction

45% dissatisfaction

10% dissatisfaction

  • Carry out error analysis to figure out which component performed poorly.

  • Focus on those areas to improve over-all output.

Component Level Evals

  • Re-running the entire pipeline every time you tweak one component is slow and expensive.

  • Instead, build evals that test each component on its own.

For example, to test the web search component:

  • Create a list of gold-standard web resources for each query.

  • Compute a rubric score for how many of the returned results match those gold-standard resources.

Benefits of component level evaluations

  • Can provide cleaner signals for specific errors.

    • Avoids the noise from end to end system runs.

  • Lets each team own and track the metric for its own component.

How to address problems you identify?

Non LLM components

  • Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.

  • Replace the component, try a different web search provider, etc.

LLM components

  • Improve your prompts, (improve instructions, add examples, with few-shot prompting)

  • Try a new model

  • Decompose a task into multiple steps

  • Fine-tune a model. (Should be the last option you consider because its expensive)

How to develop intution for model intelligence?

  • Play with models often

    • Have a personal set of evals

    • Read other peoples prompt for ideas of how to best use models

  • Use different models in your agentic workflows

    • See traces and component level evals

    • ai-suite makes it easy to quickly swap out models

Latency and Cost optimisation

Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.

Latency

  • Benchmark each component runs

  • Consider parallelism

  • Try smaller models or a faster LLM provider if a step takes longer.

Costing your workflow

  • Consider token usage for each component.

Patterns for Highly Autonomous Agents

Planning system design

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    subgraph tools["Tools"]
        t1["get_item_<br/>descriptions"]:::green
        t2["check_<br/>inventory"]:::green
        t3["process_<br/>item_return"]:::green
        t4["get_item_<br/>price"]:::green
        t5["check_past_<br/>transactions"]:::green
        t6["process_<br/>item_sale"]:::green
    end

    sysPrompt["<b>System prompt</b><br/>You have access to the following tools:<br/>{description of tools}<br/><br/>Return a step-by-step plan to carry<br/>out the user's request."]:::note
    query["Do you have any round<br/>sunglasses in stock<br/>that are under $100?"]:::red
    planner["LLM<br/>(planner)"]:::blue
    plan["<b>Plan</b><br/>1. Use <b>get_item_descriptions</b> tool<br/>to find round sunglasses<br/>2. Use <b>check_inventory</b> to see<br/>if results are in stock<br/>3. Use <b>get_item_price</b> to see if<br/>in-stock results are &lt;$100"]

    sysPrompt -.-> planner
    tools -.-> planner
    query --> planner --> plan

    step1["Step 1 text"]
    llm1["LLM"]:::blue
    tool1["get_item_<br/>descriptions"]:::green
    step2["Step 1 output,<br/>Step 2 text"]
    llm2["LLM"]:::blue
    tool2["check_<br/>inventory"]:::green
    step3["Step 2 output,<br/>Step 3 text"]
    llm3["LLM"]:::blue
    tool3["get_item_<br/>price"]:::green
    answer["Answer for the user"]:::red

    plan --> step1
    step1 --> llm1 --> tool1 --> step2
    step2 --> llm2 --> tool2 --> step3
    step3 --> llm3 --> tool3 --> answer

Because the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.

How to get LLM's to create a good plan

Ask LLM to format the plan as a JSON / XML formats.

json
{
	"plan": [
		{
			"step": 1,
			"description": "Find round sunglasses",
			"tool": "get_item descriptions",
			"args": { "query": "round Sunglasses" }
		},
		{
			"step": 2,
			"description": "Check available stock",
			"tool": "check_inventory",
			"args": {
				"items": "results from step 1"
			}
		}
	]
}

Planning with code executions

The challenge of planning with tools

  • Every new kind of question needs a new tool, so the tool list keeps growing.

  • Even a simple question needs a long plan that chains many tool calls.

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    data[("coffee_sales.csv<br/>date, price, coffee_name, size")]:::note
    query["Which month had the<br/>highest sales of hot chocolate?"]:::red
    llm["LLM"]:::blue

    subgraph tools["Tools (one per kind of query)"]
        t1["get_column_max"]:::green
        t2["get_column_min"]:::green
        t3["get_column_mean"]:::green
        t4["get_column_median"]:::green
        t5["filter_rows"]:::green
        t6["sum_rows"]:::green
        t7["get_unique_entries"]:::green
        t8["get_last_N_values"]:::green
        t9["... one more tool<br/>for every new question"]:::green
    end

    plan["<b>Plan</b><br/>1. filter_rows: January, Hot Chocolate<br/>2. get_column_mean: average amount<br/>3. filter_rows: February, Hot Chocolate<br/>4. get_column_mean: average amount<br/>5. Repeat for March ... December<br/>6. Pick the month with the highest average"]:::note

    q2["How many unique<br/>transactions last week?"]:::red
    q3["What were the amounts of<br/>the last 5 transactions?"]:::red
    q2 -->|needs new tool| t7
    q3 -->|needs new tool| t8

    data -.-> query
    query --> llm
    tools -.-> llm
    llm --> plan

Planning with code execution

  • Instead of one tool per query, let the LLM write code that solves the query.

  • Run the code in a sandbox, so a mistake in the generated code can't harm your environment.

  • One "execute code" capability replaces the whole growing list of tools.

mermaid
flowchart LR
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;


    sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with &lt;execute_python&gt; and &lt;/execute_python&gt; tags."]:::note
    query["How many unique<br/>transactions last week?"]:::red
    answer["LLM answers<br/>the user"]:::blue

    subgraph reflection [Reflection with external feedback]
        llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
        code["<b>Generated code</b>\n<br/>&lt;execute_python&gt;<br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/>&lt;/execute_python&gt;"]
        subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
            exec["Execute code"]:::green
        end
        result["Result or error"]
    end

    sysPrompt -.-> llm
    query --> llm --> code --> exec --> result
    result -->|feedback| llm
    result -->|result is correct| answer

    style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5

Multi agentic workflows

  • Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.

mermaid
flowchart LR
    classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;

    sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
    query["Create a summer marketing<br/>campaign for sunglasses"]:::red
    answer["Final report"]:::blue

    subgraph planner [Planner]
        llm1["LLM"]
        llm2["LLM"]
        llm3["LLM"]
        llm4["LLM"]
    end

    subgraph agents [Agents]
        researcher["researcher"]:::blue
        designer["graphic designer"]:::blue
        writer["writer"]:::blue
    end

    sysPrompt -.-> llm1
    query --> llm1
    llm1 -->|"1. Research current sunglasses trends"| researcher
    researcher --> llm2
    llm2 -->|"2. Create ad images"| designer
    designer --> llm3
    llm3 -->|"3. Create report"| writer
    writer --> llm4
    llm4 -->|"4. Review report"| answer

Multi Agent Communication Patterns

Hierarical Communications

  • A manager agent coordinates a flat team of agents and talks to each one directly.

mermaid
flowchart TD
    classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;

    manager["marketing manager"]
    researcher["researcher"]:::blue
    designer["graphic designer"]:::blue
    writer["writer"]:::blue

    manager <--> researcher
    manager <--> designer
    manager <--> writer

Deeper Hierarichal Communications

  • Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.

mermaid
flowchart TD
    classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;

    manager["marketing manager"]
    researcher["researcher"]:::blue
    designer["graphic designer"]:::blue
    writer["writer"]:::blue
    webResearcher["web researcher"]:::blue
    factChecker["fact checker"]:::blue
    styleWriter["style writer"]:::blue
    citationChecker["citation checker"]:::blue

    manager <--> researcher
    manager <--> designer
    manager <--> writer
    researcher <--> webResearcher
    researcher <--> factChecker
    writer <--> styleWriter
    writer <--> citationChecker

All to All

  • Every agent can talk to every other agent, with no fixed manager.

  • Hard to control

  • Caohs and unpredictable

mermaid
flowchart LR
    classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;

    manager["marketing manager"]:::blue
    researcher["researcher"]:::blue
    designer["graphic designer"]:::blue
    writer["writer"]:::blue

    manager <--> researcher
    manager <--> designer
    manager <--> writer
    researcher <--> designer
    researcher <--> writer
    designer <--> writer