Introducing Agentic AI Workflows
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt[Prompt] -->|Write an essay about\nblack holes| llm1["LLM"]:::blue
llm1 --> websearch["Web Search"]:::green
websearch --> webfetch["Web Fetch"]:::green
webfetch --> llm2["LLM"]:::blue
llm2 --> output[Output]All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["Search web"]:::note
llm2Note["Fetch 5 best sources"]:::note
llm3Note["Write essay draft"]:::note
llm4Note["Reflect and improve draft"]:::note
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteCan make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Parallelism and Modularity
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searches [3 parallel searches]
search1["LLM + web search"]:::green
search2["LLM + news search"]:::red
search3["LLM + web search"]:::green
end
search2Note["Modularity:\nMight swap to another web search provider / news search engine"]:::note
subgraph fetches [9 parallel web page downloads]
fetch1["LLM + web fetch"]:::green
fetch2["LLM + web fetch"]:::green
fetch3["LLM + web fetch"]:::green
fetch4["LLM + web fetch"]:::green
fetch5["LLM + web fetch"]:::green
fetch6["LLM + web fetch"]:::green
fetch7["LLM + web fetch"]:::green
fetch8["LLM + web fetch"]:::green
fetch9["LLM + web fetch"]:::green
end
prompt -->|Write an essay about<br/>black holes| LLM:::blue --> search1 & search2 & search3
search1 --> fetch1 & fetch2 & fetch3
search2 --> fetch4 & fetch5 & fetch6
search3 --> fetch7 & fetch8 & fetch9
fetches --> |Fetch top 3 from each run| LLM2[LLM]:::blue --> output
search2 -.- search2NoteParallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Task decomposition
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph direct [Direct generation]
direction TB
direct1["1. Write an essay on topic X"]:::red
end
subgraph threeStep [3-step workflow]
direction TB
three1["1. Write an essay outline on topic X"]:::orange
three2["2. Search web"]:::orange
three3["3. Write the essay"]:::orange
three1 --> three2 --> three3
end
subgraph fiveStep [5-step workflow]
direction TB
five1["1. Write an essay outline on topic X"]:::green
five2["2. Search web"]:::green
five3["3. Write the first draft"]:::green
five4["4. Consider what parts need revision"]:::green
five5["5. Revise your draft"]:::green
five1 --> five2 --> five3 --> five4 --> five5
end
direct -->|Not good enough...| threeStep -->|Still not good enough...| fiveStepBuilding blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
sequenceDiagram
actor User
participant Coder as Coder Agent (LLM)
participant Critic as Critic Agent (LLM)
User->>Coder: Please write code for {task}
Coder->>Critic: def do_task(x): ...
Note over Coder,Critic: Here's code intended for {task}.<br/>Check it carefully for correctness, style and efficiency,<br/>and give constructive criticism for how to improve it.
Critic-->>Coder: There's a bug on line 5. Fix it by ...
Coder->>Critic: def do_task_v2(x): ...
Critic-->>Coder: It failed Unit Test 3. Try changing ...
Coder->>User: def do_task_v3(x): ...Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection Design Pattern
flowchart LR
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection]
llm2["LLM 2:<br/>check for bugs and write<br/>improved second draft"]:::blue
llm2Note["Can be a different LLM,<br/>e.g. a reasoning model"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> llm2 --> codeV2
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5A reasoning model aka thinking model, is better at finding bugs.
Reflection with external feedback
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection with external feedback]
execute["Execute code"]:::green
feedback["Code output, errors"]
feedbackNote["SyntaxError: unterminated string<br/>literal (detected at line 1)"]:::note
llm2["LLM:<br/>reflect on output or errors and<br/>write improved second draft"]:::blue
llm2Note["You may also want to pass in the<br/>first draft and the conversation so far"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> execute --> feedback --> llm2 --> codeV2
feedback -.- feedbackNote
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
Convert to MM/DD/YYYY format
Input: {input_date}One-shot (single example)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Tool Use
LLM can choose tools when appropriate
flowchart TD
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph Tools
get_current_time["fn_get_current_time"]:::green
interest_calc["fn_interest_calc"]:::green
check_calendar["fn_check_calendar"]:::green
eval["fn_eval"]:::green
end
Tools -.-> LLM
prompt:::blue --> |What time is it?| LLM --> |Its 3:20pm| responseHow to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph Tools
get_current_time:::green
end
tool_prompt["You have access to a tool called\nget_current_time. To use it, return the following exactly:\nFunction:\nget_current_time"]:::note -.- LLM
prompt[Prompt] --> |What time is it?| LLM --> |Function:\nget_current_time| parser --> |conversation history\ntime| LLM1[LLM] --> |Its 8am| output
Tools -.-> parserAI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/><br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["What's the<br/>square root of 2?"]:::red
llm1["LLM"]
output["output<br/><br/><execute_python><br/>import math<br/>print(math.sqrt(2))<br/></execute_python>"]
exec["Execute code<br/><br/>exec(output)"]:::green
result["result<br/><br/>1.4142135623730951"]
llm2["LLM<br/><br/>The square root of 2<br/>is approximately 1.4142."]
sysPrompt -.-> llm1
query --> llm1 --> output --> exec --> result --> llm2Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Practical tips for builiding Agentic AI
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef ErrorNote fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["<b>Bad search terms?</b><br/>Are the queries too vague,<br/>too narrow or off-topic?"]:::ErrorNote
llm2Note["<b>Low quality search results?</b><br/>Do the results actually cover<br/>the topic, or are they noisy?"]:::ErrorNote
llm3Note["<b>Poor selection of URLs?</b><br/>Did it pick the most relevant<br/>and credible sources?"]:::ErrorNote
llm4Note["<b>Bad reasoning over texts?</b><br/>Does the essay miss, distort or<br/>invent points from the sources?"]:::ErrorNote
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteTry to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Component Level Evals
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
Patterns for Highly Autonomous Agents
Planning system design
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph tools["Tools"]
t1["get_item_<br/>descriptions"]:::green
t2["check_<br/>inventory"]:::green
t3["process_<br/>item_return"]:::green
t4["get_item_<br/>price"]:::green
t5["check_past_<br/>transactions"]:::green
t6["process_<br/>item_sale"]:::green
end
sysPrompt["<b>System prompt</b><br/>You have access to the following tools:<br/>{description of tools}<br/><br/>Return a step-by-step plan to carry<br/>out the user's request."]:::note
query["Do you have any round<br/>sunglasses in stock<br/>that are under $100?"]:::red
planner["LLM<br/>(planner)"]:::blue
plan["<b>Plan</b><br/>1. Use <b>get_item_descriptions</b> tool<br/>to find round sunglasses<br/>2. Use <b>check_inventory</b> to see<br/>if results are in stock<br/>3. Use <b>get_item_price</b> to see if<br/>in-stock results are <$100"]
sysPrompt -.-> planner
tools -.-> planner
query --> planner --> plan
step1["Step 1 text"]
llm1["LLM"]:::blue
tool1["get_item_<br/>descriptions"]:::green
step2["Step 1 output,<br/>Step 2 text"]
llm2["LLM"]:::blue
tool2["check_<br/>inventory"]:::green
step3["Step 2 output,<br/>Step 3 text"]
llm3["LLM"]:::blue
tool3["get_item_<br/>price"]:::green
answer["Answer for the user"]:::red
plan --> step1
step1 --> llm1 --> tool1 --> step2
step2 --> llm2 --> tool2 --> step3
step3 --> llm3 --> tool3 --> answerBecause the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
{
"plan": [
{
"step": 1,
"description": "Find round sunglasses",
"tool": "get_item descriptions",
"args": { "query": "round Sunglasses" }
},
{
"step": 2,
"description": "Check available stock",
"tool": "check_inventory",
"args": {
"items": "results from step 1"
}
}
]
}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
data[("coffee_sales.csv<br/>date, price, coffee_name, size")]:::note
query["Which month had the<br/>highest sales of hot chocolate?"]:::red
llm["LLM"]:::blue
subgraph tools["Tools (one per kind of query)"]
t1["get_column_max"]:::green
t2["get_column_min"]:::green
t3["get_column_mean"]:::green
t4["get_column_median"]:::green
t5["filter_rows"]:::green
t6["sum_rows"]:::green
t7["get_unique_entries"]:::green
t8["get_last_N_values"]:::green
t9["... one more tool<br/>for every new question"]:::green
end
plan["<b>Plan</b><br/>1. filter_rows: January, Hot Chocolate<br/>2. get_column_mean: average amount<br/>3. filter_rows: February, Hot Chocolate<br/>4. get_column_mean: average amount<br/>5. Repeat for March ... December<br/>6. Pick the month with the highest average"]:::note
q2["How many unique<br/>transactions last week?"]:::red
q3["What were the amounts of<br/>the last 5 transactions?"]:::red
q2 -->|needs new tool| t7
q3 -->|needs new tool| t8
data -.-> query
query --> llm
tools -.-> llm
llm --> planPlanning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerIntroducing Agentic AI Workflows
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt[Prompt] -->|Write an essay about\nblack holes| llm1["LLM"]:::blue
llm1 --> websearch["Web Search"]:::green
websearch --> webfetch["Web Fetch"]:::green
webfetch --> llm2["LLM"]:::blue
llm2 --> output[Output]All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["Search web"]:::note
llm2Note["Fetch 5 best sources"]:::note
llm3Note["Write essay draft"]:::note
llm4Note["Reflect and improve draft"]:::note
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteCan make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Parallelism and Modularity
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searches [3 parallel searches]
search1["LLM + web search"]:::green
search2["LLM + news search"]:::red
search3["LLM + web search"]:::green
end
search2Note["Modularity:\nMight swap to another web search provider / news search engine"]:::note
subgraph fetches [9 parallel web page downloads]
fetch1["LLM + web fetch"]:::green
fetch2["LLM + web fetch"]:::green
fetch3["LLM + web fetch"]:::green
fetch4["LLM + web fetch"]:::green
fetch5["LLM + web fetch"]:::green
fetch6["LLM + web fetch"]:::green
fetch7["LLM + web fetch"]:::green
fetch8["LLM + web fetch"]:::green
fetch9["LLM + web fetch"]:::green
end
prompt -->|Write an essay about<br/>black holes| LLM:::blue --> search1 & search2 & search3
search1 --> fetch1 & fetch2 & fetch3
search2 --> fetch4 & fetch5 & fetch6
search3 --> fetch7 & fetch8 & fetch9
fetches --> |Fetch top 3 from each run| LLM2[LLM]:::blue --> output
search2 -.- search2NoteParallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Task decomposition
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph direct [Direct generation]
direction TB
direct1["1. Write an essay on topic X"]:::red
end
subgraph threeStep [3-step workflow]
direction TB
three1["1. Write an essay outline on topic X"]:::orange
three2["2. Search web"]:::orange
three3["3. Write the essay"]:::orange
three1 --> three2 --> three3
end
subgraph fiveStep [5-step workflow]
direction TB
five1["1. Write an essay outline on topic X"]:::green
five2["2. Search web"]:::green
five3["3. Write the first draft"]:::green
five4["4. Consider what parts need revision"]:::green
five5["5. Revise your draft"]:::green
five1 --> five2 --> five3 --> five4 --> five5
end
direct -->|Not good enough...| threeStep -->|Still not good enough...| fiveStepBuilding blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
sequenceDiagram
actor User
participant Coder as Coder Agent (LLM)
participant Critic as Critic Agent (LLM)
User->>Coder: Please write code for {task}
Coder->>Critic: def do_task(x): ...
Note over Coder,Critic: Here's code intended for {task}.<br/>Check it carefully for correctness, style and efficiency,<br/>and give constructive criticism for how to improve it.
Critic-->>Coder: There's a bug on line 5. Fix it by ...
Coder->>Critic: def do_task_v2(x): ...
Critic-->>Coder: It failed Unit Test 3. Try changing ...
Coder->>User: def do_task_v3(x): ...Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection Design Pattern
flowchart LR
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection]
llm2["LLM 2:<br/>check for bugs and write<br/>improved second draft"]:::blue
llm2Note["Can be a different LLM,<br/>e.g. a reasoning model"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> llm2 --> codeV2
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5A reasoning model aka thinking model, is better at finding bugs.
Reflection with external feedback
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection with external feedback]
execute["Execute code"]:::green
feedback["Code output, errors"]
feedbackNote["SyntaxError: unterminated string<br/>literal (detected at line 1)"]:::note
llm2["LLM:<br/>reflect on output or errors and<br/>write improved second draft"]:::blue
llm2Note["You may also want to pass in the<br/>first draft and the conversation so far"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> execute --> feedback --> llm2 --> codeV2
feedback -.- feedbackNote
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
Convert to MM/DD/YYYY format
Input: {input_date}One-shot (single example)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Tool Use
LLM can choose tools when appropriate
flowchart TD
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph Tools
get_current_time["fn_get_current_time"]:::green
interest_calc["fn_interest_calc"]:::green
check_calendar["fn_check_calendar"]:::green
eval["fn_eval"]:::green
end
Tools -.-> LLM
prompt:::blue --> |What time is it?| LLM --> |Its 3:20pm| responseHow to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph Tools
get_current_time:::green
end
tool_prompt["You have access to a tool called\nget_current_time. To use it, return the following exactly:\nFunction:\nget_current_time"]:::note -.- LLM
prompt[Prompt] --> |What time is it?| LLM --> |Function:\nget_current_time| parser --> |conversation history\ntime| LLM1[LLM] --> |Its 8am| output
Tools -.-> parserAI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/><br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["What's the<br/>square root of 2?"]:::red
llm1["LLM"]
output["output<br/><br/><execute_python><br/>import math<br/>print(math.sqrt(2))<br/></execute_python>"]
exec["Execute code<br/><br/>exec(output)"]:::green
result["result<br/><br/>1.4142135623730951"]
llm2["LLM<br/><br/>The square root of 2<br/>is approximately 1.4142."]
sysPrompt -.-> llm1
query --> llm1 --> output --> exec --> result --> llm2Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Practical tips for builiding Agentic AI
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef ErrorNote fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["<b>Bad search terms?</b><br/>Are the queries too vague,<br/>too narrow or off-topic?"]:::ErrorNote
llm2Note["<b>Low quality search results?</b><br/>Do the results actually cover<br/>the topic, or are they noisy?"]:::ErrorNote
llm3Note["<b>Poor selection of URLs?</b><br/>Did it pick the most relevant<br/>and credible sources?"]:::ErrorNote
llm4Note["<b>Bad reasoning over texts?</b><br/>Does the essay miss, distort or<br/>invent points from the sources?"]:::ErrorNote
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteTry to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Component Level Evals
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
Patterns for Highly Autonomous Agents
Planning system design
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph tools["Tools"]
t1["get_item_<br/>descriptions"]:::green
t2["check_<br/>inventory"]:::green
t3["process_<br/>item_return"]:::green
t4["get_item_<br/>price"]:::green
t5["check_past_<br/>transactions"]:::green
t6["process_<br/>item_sale"]:::green
end
sysPrompt["<b>System prompt</b><br/>You have access to the following tools:<br/>{description of tools}<br/><br/>Return a step-by-step plan to carry<br/>out the user's request."]:::note
query["Do you have any round<br/>sunglasses in stock<br/>that are under $100?"]:::red
planner["LLM<br/>(planner)"]:::blue
plan["<b>Plan</b><br/>1. Use <b>get_item_descriptions</b> tool<br/>to find round sunglasses<br/>2. Use <b>check_inventory</b> to see<br/>if results are in stock<br/>3. Use <b>get_item_price</b> to see if<br/>in-stock results are <$100"]
sysPrompt -.-> planner
tools -.-> planner
query --> planner --> plan
step1["Step 1 text"]
llm1["LLM"]:::blue
tool1["get_item_<br/>descriptions"]:::green
step2["Step 1 output,<br/>Step 2 text"]
llm2["LLM"]:::blue
tool2["check_<br/>inventory"]:::green
step3["Step 2 output,<br/>Step 3 text"]
llm3["LLM"]:::blue
tool3["get_item_<br/>price"]:::green
answer["Answer for the user"]:::red
plan --> step1
step1 --> llm1 --> tool1 --> step2
step2 --> llm2 --> tool2 --> step3
step3 --> llm3 --> tool3 --> answerBecause the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
{
"plan": [
{
"step": 1,
"description": "Find round sunglasses",
"tool": "get_item descriptions",
"args": { "query": "round Sunglasses" }
},
{
"step": 2,
"description": "Check available stock",
"tool": "check_inventory",
"args": {
"items": "results from step 1"
}
}
]
}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
data[("coffee_sales.csv<br/>date, price, coffee_name, size")]:::note
query["Which month had the<br/>highest sales of hot chocolate?"]:::red
llm["LLM"]:::blue
subgraph tools["Tools (one per kind of query)"]
t1["get_column_max"]:::green
t2["get_column_min"]:::green
t3["get_column_mean"]:::green
t4["get_column_median"]:::green
t5["filter_rows"]:::green
t6["sum_rows"]:::green
t7["get_unique_entries"]:::green
t8["get_last_N_values"]:::green
t9["... one more tool<br/>for every new question"]:::green
end
plan["<b>Plan</b><br/>1. filter_rows: January, Hot Chocolate<br/>2. get_column_mean: average amount<br/>3. filter_rows: February, Hot Chocolate<br/>4. get_column_mean: average amount<br/>5. Repeat for March ... December<br/>6. Pick the month with the highest average"]:::note
q2["How many unique<br/>transactions last week?"]:::red
q3["What were the amounts of<br/>the last 5 transactions?"]:::red
q2 -->|needs new tool| t7
q3 -->|needs new tool| t8
data -.-> query
query --> llm
tools -.-> llm
llm --> planPlanning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerIntroducing Agentic AI Workflows
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt[Prompt] -->|Write an essay about\nblack holes| llm1["LLM"]:::blue
llm1 --> websearch["Web Search"]:::green
websearch --> webfetch["Web Fetch"]:::green
webfetch --> llm2["LLM"]:::blue
llm2 --> output[Output]All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["Search web"]:::note
llm2Note["Fetch 5 best sources"]:::note
llm3Note["Write essay draft"]:::note
llm4Note["Reflect and improve draft"]:::note
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteCan make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Parallelism and Modularity
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searches [3 parallel searches]
search1["LLM + web search"]:::green
search2["LLM + news search"]:::red
search3["LLM + web search"]:::green
end
search2Note["Modularity:\nMight swap to another web search provider / news search engine"]:::note
subgraph fetches [9 parallel web page downloads]
fetch1["LLM + web fetch"]:::green
fetch2["LLM + web fetch"]:::green
fetch3["LLM + web fetch"]:::green
fetch4["LLM + web fetch"]:::green
fetch5["LLM + web fetch"]:::green
fetch6["LLM + web fetch"]:::green
fetch7["LLM + web fetch"]:::green
fetch8["LLM + web fetch"]:::green
fetch9["LLM + web fetch"]:::green
end
prompt -->|Write an essay about<br/>black holes| LLM:::blue --> search1 & search2 & search3
search1 --> fetch1 & fetch2 & fetch3
search2 --> fetch4 & fetch5 & fetch6
search3 --> fetch7 & fetch8 & fetch9
fetches --> |Fetch top 3 from each run| LLM2[LLM]:::blue --> output
search2 -.- search2NoteParallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Task decomposition
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph direct [Direct generation]
direction TB
direct1["1. Write an essay on topic X"]:::red
end
subgraph threeStep [3-step workflow]
direction TB
three1["1. Write an essay outline on topic X"]:::orange
three2["2. Search web"]:::orange
three3["3. Write the essay"]:::orange
three1 --> three2 --> three3
end
subgraph fiveStep [5-step workflow]
direction TB
five1["1. Write an essay outline on topic X"]:::green
five2["2. Search web"]:::green
five3["3. Write the first draft"]:::green
five4["4. Consider what parts need revision"]:::green
five5["5. Revise your draft"]:::green
five1 --> five2 --> five3 --> five4 --> five5
end
direct -->|Not good enough...| threeStep -->|Still not good enough...| fiveStepBuilding blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
sequenceDiagram
actor User
participant Coder as Coder Agent (LLM)
participant Critic as Critic Agent (LLM)
User->>Coder: Please write code for {task}
Coder->>Critic: def do_task(x): ...
Note over Coder,Critic: Here's code intended for {task}.<br/>Check it carefully for correctness, style and efficiency,<br/>and give constructive criticism for how to improve it.
Critic-->>Coder: There's a bug on line 5. Fix it by ...
Coder->>Critic: def do_task_v2(x): ...
Critic-->>Coder: It failed Unit Test 3. Try changing ...
Coder->>User: def do_task_v3(x): ...Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
No title
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Rendering diagram...
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Rendering diagram...
Parallelism and Modularity
Rendering diagram...
Parallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Rendering diagram...
Task decomposition
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
Rendering diagram...
Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection
Rendering diagram...
Rendering diagram...
Tool use
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Reflection Design Pattern
flowchart LR
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection]
llm2["LLM 2:<br/>check for bugs and write<br/>improved second draft"]:::blue
llm2Note["Can be a different LLM,<br/>e.g. a reasoning model"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> llm2 --> codeV2
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5A reasoning model aka thinking model, is better at finding bugs.
Reflection with external feedback
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection with external feedback]
execute["Execute code"]:::green
feedback["Code output, errors"]
feedbackNote["SyntaxError: unterminated string<br/>literal (detected at line 1)"]:::note
llm2["LLM:<br/>reflect on output or errors and<br/>write improved second draft"]:::blue
llm2Note["You may also want to pass in the<br/>first draft and the conversation so far"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> execute --> feedback --> llm2 --> codeV2
feedback -.- feedbackNote
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
Convert to MM/DD/YYYY format
Input: {input_date}One-shot (single example)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
No title
Rendering diagram...
A reasoning model aka thinking model, is better at finding bugs.
Rendering diagram...
Reflection with external feedback
Rendering diagram...
Rendering diagram...
Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Tool Use
LLM can choose tools when appropriate
Rendering diagram...
How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/><br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["What's the<br/>square root of 2?"]:::red
llm1["LLM"]
output["output<br/><br/><execute_python><br/>import math<br/>print(math.sqrt(2))<br/></execute_python>"]
exec["Execute code<br/><br/>exec(output)"]:::green
result["result<br/><br/>1.4142135623730951"]
llm2["LLM<br/><br/>The square root of 2<br/>is approximately 1.4142."]
sysPrompt -.-> llm1
query --> llm1 --> output --> exec --> result --> llm2Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
LLM can choose tools when appropriate
Rendering diagram...
Rendering diagram...
How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Primitive function keyword
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Rendering diagram...
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Practical tips for builiding Agentic AI
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef ErrorNote fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["<b>Bad search terms?</b><br/>Are the queries too vague,<br/>too narrow or off-topic?"]:::ErrorNote
llm2Note["<b>Low quality search results?</b><br/>Do the results actually cover<br/>the topic, or are they noisy?"]:::ErrorNote
llm3Note["<b>Poor selection of URLs?</b><br/>Did it pick the most relevant<br/>and credible sources?"]:::ErrorNote
llm4Note["<b>Bad reasoning over texts?</b><br/>Does the essay miss, distort or<br/>invent points from the sources?"]:::ErrorNote
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteTry to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Component Level Evals
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Component Level Evals
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Patterns for Highly Autonomous Agents
Planning system design
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph tools["Tools"]
t1["get_item_<br/>descriptions"]:::green
t2["check_<br/>inventory"]:::green
t3["process_<br/>item_return"]:::green
t4["get_item_<br/>price"]:::green
t5["check_past_<br/>transactions"]:::green
t6["process_<br/>item_sale"]:::green
end
sysPrompt["<b>System prompt</b><br/>You have access to the following tools:<br/>{description of tools}<br/><br/>Return a step-by-step plan to carry<br/>out the user's request."]:::note
query["Do you have any round<br/>sunglasses in stock<br/>that are under $100?"]:::red
planner["LLM<br/>(planner)"]:::blue
plan["<b>Plan</b><br/>1. Use <b>get_item_descriptions</b> tool<br/>to find round sunglasses<br/>2. Use <b>check_inventory</b> to see<br/>if results are in stock<br/>3. Use <b>get_item_price</b> to see if<br/>in-stock results are <$100"]
sysPrompt -.-> planner
tools -.-> planner
query --> planner --> plan
step1["Step 1 text"]
llm1["LLM"]:::blue
tool1["get_item_<br/>descriptions"]:::green
step2["Step 1 output,<br/>Step 2 text"]
llm2["LLM"]:::blue
tool2["check_<br/>inventory"]:::green
step3["Step 2 output,<br/>Step 3 text"]
llm3["LLM"]:::blue
tool3["get_item_<br/>price"]:::green
answer["Answer for the user"]:::red
plan --> step1
step1 --> llm1 --> tool1 --> step2
step2 --> llm2 --> tool2 --> step3
step3 --> llm3 --> tool3 --> answerBecause the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
{
"plan": [
{
"step": 1,
"description": "Find round sunglasses",
"tool": "get_item descriptions",
"args": { "query": "round Sunglasses" }
},
{
"step": 2,
"description": "Check available stock",
"tool": "check_inventory",
"args": {
"items": "results from step 1"
}
}
]
}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
data[("coffee_sales.csv<br/>date, price, coffee_name, size")]:::note
query["Which month had the<br/>highest sales of hot chocolate?"]:::red
llm["LLM"]:::blue
subgraph tools["Tools (one per kind of query)"]
t1["get_column_max"]:::green
t2["get_column_min"]:::green
t3["get_column_mean"]:::green
t4["get_column_median"]:::green
t5["filter_rows"]:::green
t6["sum_rows"]:::green
t7["get_unique_entries"]:::green
t8["get_last_N_values"]:::green
t9["... one more tool<br/>for every new question"]:::green
end
plan["<b>Plan</b><br/>1. filter_rows: January, Hot Chocolate<br/>2. get_column_mean: average amount<br/>3. filter_rows: February, Hot Chocolate<br/>4. get_column_mean: average amount<br/>5. Repeat for March ... December<br/>6. Pick the month with the highest average"]:::note
q2["How many unique<br/>transactions last week?"]:::red
q3["What were the amounts of<br/>the last 5 transactions?"]:::red
q2 -->|needs new tool| t7
q3 -->|needs new tool| t8
data -.-> query
query --> llm
tools -.-> llm
llm --> planPlanning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerPlanning system design
Rendering diagram...
Because the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
Rendering diagram...
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerThe challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
Introducing Agentic AI Workflows
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt[Prompt] -->|Write an essay about\nblack holes| llm1["LLM"]:::blue
llm1 --> websearch["Web Search"]:::green
websearch --> webfetch["Web Fetch"]:::green
webfetch --> llm2["LLM"]:::blue
llm2 --> output[Output]All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["Search web"]:::note
llm2Note["Fetch 5 best sources"]:::note
llm3Note["Write essay draft"]:::note
llm4Note["Reflect and improve draft"]:::note
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteCan make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Parallelism and Modularity
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph searches [3 parallel searches]
search1["LLM + web search"]:::green
search2["LLM + news search"]:::red
search3["LLM + web search"]:::green
end
search2Note["Modularity:\nMight swap to another web search provider / news search engine"]:::note
subgraph fetches [9 parallel web page downloads]
fetch1["LLM + web fetch"]:::green
fetch2["LLM + web fetch"]:::green
fetch3["LLM + web fetch"]:::green
fetch4["LLM + web fetch"]:::green
fetch5["LLM + web fetch"]:::green
fetch6["LLM + web fetch"]:::green
fetch7["LLM + web fetch"]:::green
fetch8["LLM + web fetch"]:::green
fetch9["LLM + web fetch"]:::green
end
prompt -->|Write an essay about<br/>black holes| LLM:::blue --> search1 & search2 & search3
search1 --> fetch1 & fetch2 & fetch3
search2 --> fetch4 & fetch5 & fetch6
search3 --> fetch7 & fetch8 & fetch9
fetches --> |Fetch top 3 from each run| LLM2[LLM]:::blue --> output
search2 -.- search2NoteParallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Task decomposition
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph direct [Direct generation]
direction TB
direct1["1. Write an essay on topic X"]:::red
end
subgraph threeStep [3-step workflow]
direction TB
three1["1. Write an essay outline on topic X"]:::orange
three2["2. Search web"]:::orange
three3["3. Write the essay"]:::orange
three1 --> three2 --> three3
end
subgraph fiveStep [5-step workflow]
direction TB
five1["1. Write an essay outline on topic X"]:::green
five2["2. Search web"]:::green
five3["3. Write the first draft"]:::green
five4["4. Consider what parts need revision"]:::green
five5["5. Revise your draft"]:::green
five1 --> five2 --> five3 --> five4 --> five5
end
direct -->|Not good enough...| threeStep -->|Still not good enough...| fiveStepBuilding blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
sequenceDiagram
actor User
participant Coder as Coder Agent (LLM)
participant Critic as Critic Agent (LLM)
User->>Coder: Please write code for {task}
Coder->>Critic: def do_task(x): ...
Note over Coder,Critic: Here's code intended for {task}.<br/>Check it carefully for correctness, style and efficiency,<br/>and give constructive criticism for how to improve it.
Critic-->>Coder: There's a bug on line 5. Fix it by ...
Coder->>Critic: def do_task_v2(x): ...
Critic-->>Coder: It failed Unit Test 3. Try changing ...
Coder->>User: def do_task_v3(x): ...Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
No title
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Rendering diagram...
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Rendering diagram...
Parallelism and Modularity
Rendering diagram...
Parallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Rendering diagram...
Task decomposition
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
Rendering diagram...
Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection
Rendering diagram...
Rendering diagram...
Tool use
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Reflection Design Pattern
flowchart LR
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection]
llm2["LLM 2:<br/>check for bugs and write<br/>improved second draft"]:::blue
llm2Note["Can be a different LLM,<br/>e.g. a reasoning model"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> llm2 --> codeV2
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5A reasoning model aka thinking model, is better at finding bugs.
Reflection with external feedback
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
prompt["Write code for task X"]:::red
llm1["LLM:<br/>write first draft"]:::blue
codeV1["code V1:<br/>def do_task(args): ..."]
subgraph reflection [Reflection with external feedback]
execute["Execute code"]:::green
feedback["Code output, errors"]
feedbackNote["SyntaxError: unterminated string<br/>literal (detected at line 1)"]:::note
llm2["LLM:<br/>reflect on output or errors and<br/>write improved second draft"]:::blue
llm2Note["You may also want to pass in the<br/>first draft and the conversation so far"]:::note
end
codeV2["code V2:<br/>def do_task_v2(args): ..."]
prompt --> llm1 --> codeV1 --> execute --> feedback --> llm2 --> codeV2
feedback -.- feedbackNote
llm2 -.- llm2Note
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
Convert to MM/DD/YYYY format
Input: {input_date}One-shot (single example)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
No title
Rendering diagram...
A reasoning model aka thinking model, is better at finding bugs.
Rendering diagram...
Reflection with external feedback
Rendering diagram...
Rendering diagram...
Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Tool Use
LLM can choose tools when appropriate
Rendering diagram...
How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/><br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["What's the<br/>square root of 2?"]:::red
llm1["LLM"]
output["output<br/><br/><execute_python><br/>import math<br/>print(math.sqrt(2))<br/></execute_python>"]
exec["Execute code<br/><br/>exec(output)"]:::green
result["result<br/><br/>1.4142135623730951"]
llm2["LLM<br/><br/>The square root of 2<br/>is approximately 1.4142."]
sysPrompt -.-> llm1
query --> llm1 --> output --> exec --> result --> llm2Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
LLM can choose tools when appropriate
Rendering diagram...
Rendering diagram...
How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Primitive function keyword
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Rendering diagram...
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Practical tips for builiding Agentic AI
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef ErrorNote fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000,stroke-dasharray: 5 5,color:#333;
subgraph searchTools [Tools]
toolWebSearch["web search"]:::green
toolNews["news"]:::green
toolArxiv["arXiv"]:::green
end
subgraph fetchTools [Tools]
toolWebFetch["web fetch"]:::green
toolPdfToText["pdf to text"]:::green
end
llm1["LLM"]:::blue
websearch["Web Search"]:::green
llm2["LLM"]:::blue
webfetch["Web Fetch"]:::green
llm3["LLM"]:::blue
llm4["LLM"]:::blue
output[Output]
llm1Note["<b>Bad search terms?</b><br/>Are the queries too vague,<br/>too narrow or off-topic?"]:::ErrorNote
llm2Note["<b>Low quality search results?</b><br/>Do the results actually cover<br/>the topic, or are they noisy?"]:::ErrorNote
llm3Note["<b>Poor selection of URLs?</b><br/>Did it pick the most relevant<br/>and credible sources?"]:::ErrorNote
llm4Note["<b>Bad reasoning over texts?</b><br/>Does the essay miss, distort or<br/>invent points from the sources?"]:::ErrorNote
prompt -->|Write an essay about\nblack holes| llm1 --> websearch --> llm2 --> webfetch --> llm3 --> llm4 --> output
searchTools --> llm1
fetchTools --> llm2
llm1 -.- llm1Note
llm2 -.- llm2Note
llm3 -.- llm3Note
llm4 -.- llm4NoteTry to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Component Level Evals
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Component Level Evals
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Patterns for Highly Autonomous Agents
Planning system design
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
subgraph tools["Tools"]
t1["get_item_<br/>descriptions"]:::green
t2["check_<br/>inventory"]:::green
t3["process_<br/>item_return"]:::green
t4["get_item_<br/>price"]:::green
t5["check_past_<br/>transactions"]:::green
t6["process_<br/>item_sale"]:::green
end
sysPrompt["<b>System prompt</b><br/>You have access to the following tools:<br/>{description of tools}<br/><br/>Return a step-by-step plan to carry<br/>out the user's request."]:::note
query["Do you have any round<br/>sunglasses in stock<br/>that are under $100?"]:::red
planner["LLM<br/>(planner)"]:::blue
plan["<b>Plan</b><br/>1. Use <b>get_item_descriptions</b> tool<br/>to find round sunglasses<br/>2. Use <b>check_inventory</b> to see<br/>if results are in stock<br/>3. Use <b>get_item_price</b> to see if<br/>in-stock results are <$100"]
sysPrompt -.-> planner
tools -.-> planner
query --> planner --> plan
step1["Step 1 text"]
llm1["LLM"]:::blue
tool1["get_item_<br/>descriptions"]:::green
step2["Step 1 output,<br/>Step 2 text"]
llm2["LLM"]:::blue
tool2["check_<br/>inventory"]:::green
step3["Step 2 output,<br/>Step 3 text"]
llm3["LLM"]:::blue
tool3["get_item_<br/>price"]:::green
answer["Answer for the user"]:::red
plan --> step1
step1 --> llm1 --> tool1 --> step2
step2 --> llm2 --> tool2 --> step3
step3 --> llm3 --> tool3 --> answerBecause the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
{
"plan": [
{
"step": 1,
"description": "Find round sunglasses",
"tool": "get_item descriptions",
"args": { "query": "round Sunglasses" }
},
{
"step": 2,
"description": "Check available stock",
"tool": "check_inventory",
"args": {
"items": "results from step 1"
}
}
]
}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
data[("coffee_sales.csv<br/>date, price, coffee_name, size")]:::note
query["Which month had the<br/>highest sales of hot chocolate?"]:::red
llm["LLM"]:::blue
subgraph tools["Tools (one per kind of query)"]
t1["get_column_max"]:::green
t2["get_column_min"]:::green
t3["get_column_mean"]:::green
t4["get_column_median"]:::green
t5["filter_rows"]:::green
t6["sum_rows"]:::green
t7["get_unique_entries"]:::green
t8["get_last_N_values"]:::green
t9["... one more tool<br/>for every new question"]:::green
end
plan["<b>Plan</b><br/>1. filter_rows: January, Hot Chocolate<br/>2. get_column_mean: average amount<br/>3. filter_rows: February, Hot Chocolate<br/>4. get_column_mean: average amount<br/>5. Repeat for March ... December<br/>6. Pick the month with the highest average"]:::note
q2["How many unique<br/>transactions last week?"]:::red
q3["What were the amounts of<br/>the last 5 transactions?"]:::red
q2 -->|needs new tool| t7
q3 -->|needs new tool| t8
data -.-> query
query --> llm
tools -.-> llm
llm --> planPlanning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerPlanning system design
Rendering diagram...
Because the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
Rendering diagram...
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerThe challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
No title
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Rendering diagram...
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Rendering diagram...
Parallelism and Modularity
Rendering diagram...
Parallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Rendering diagram...
Task decomposition
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
Rendering diagram...
Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection
Rendering diagram...
Rendering diagram...
Tool use
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
No title
Rendering diagram...
A reasoning model aka thinking model, is better at finding bugs.
Rendering diagram...
Reflection with external feedback
Rendering diagram...
Rendering diagram...
Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Primitive function keyword
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Component Level Evals
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Planning system design
Rendering diagram...
Because the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
Rendering diagram...
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerThe challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
No title
An agentic AI workflow is a process where an LLM based app executes multiple steps to complete a task.
The output of exciting models improve significantly in agent mode.
Degrees of Autonomy
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Semi autonomous agent
Can make some decisions
Can decide to use a predefined tool
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Less autonomous agent
Rendering diagram...
All steps predetermined
All tool use hard coded
Autonomy is in text generation
Rendering diagram...
More autonomous agent
Rendering diagram...
Can make many decisions autonomously, including when to execute a step or order or execution of steps.
Can create tools on the fly
Example:
Customer service agent, gets the following query,
"Do you have any back jeans or blue jeans?"
Check inventory for black jeans
Check inventory for blue jeans
Respond to customer
The agent doesn't know the required steps, ahead of time. The agent needs to come up with the steps to execute on the fly.
Rendering diagram...
Parallelism and Modularity
Rendering diagram...
Parallelisation, lets us execute steps faster.
Modularisation, lets us swap out models or tools to improve output.
Rendering diagram...
Task decomposition
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
No title
Start by asking, "How would a person solve this problem step by step?"
Break down your problem into smaller, more manageable steps.
Can this step be implemented with an LLM or and API call that we have access to?
If its not, then I would ask myself "How would a Human solve this", can decompose it further?
Rendering diagram...
Rendering diagram...
Building blocks for Agentic workflows
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Models
LLMs
Text generation, tool use, information extraction
Other AI models
PDF-to-text, text-to-speech, image analysis
Tools
API
Web search, get real-time data, send email, check calendar
Information retrieval
Databases, Retrieval Augmented Generation (RAG)
Code execution
Basic calculator, data analysis
Evaluating agentic AI (evals)
Measure output quality with code when there's a clear right answer (objective evals), and use an LLM as a judge when quality is a matter of judgement (subjective evals).
Do an End-to-End and Component Level evals.
Examine traces to perform error analysis.
Design patterns for building agents
Reflection
Rendering diagram...
Tool use
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
subgraph webSearchExample [Web search tool]
searchPrompt["What is the best coffee maker<br/>according to reviewers?"]:::red
searchLlm["LLM"]:::blue
searchTool["Web search:<br/>best coffee maker according to reviewers"]:::green
searchAnswer[Answer]
searchPrompt --> searchLlm --> searchTool --> searchLlm --> searchAnswer
end
subgraph codeExecExample [Code execution tool]
codePrompt["If I invest $100 at compound 7% interest<br/>for 12 years, what do I have at the end?"]:::red
codeLlm["LLM"]:::blue
codeTool["Code execution:<br/>principal = 100<br/>interest_rate = 0.07<br/>years = 12<br/>value = principal * (1 + interest_rate) ** years"]:::green
codeAnswer["Answer: ~$225.22"]
codePrompt --> codeLlm --> codeTool --> codeLlm --> codeAnswer
end
subgraph toolCategories [Tool categories]
direction TB
analysis["Analysis<br/>Code execution, Wolfram Alpha,<br/>Bearly Code Interpreter"]:::orange
infoGathering["Information gathering<br/>Web search, Wikipedia,<br/>Database access"]:::orange
productivity["Productivity<br/>Email, Calendar, Messaging"]:::orange
images["Images<br/>Image generation, Image captioning, OCR"]:::orange
endPlanning
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
request["Generate an image of a girl reading a book,<br/>in the same pose as the boy in example.jpg,<br/>then describe the new image with your voice"]:::red
planner["LLM:<br/>plans the steps"]:::blue
subgraph plan [Plan the LLM executes]
inputImage[/example.jpg/]
poseStep["Pose determination"]:::green
pose[/Pose skeleton/]
imageStep["Pose-to-image"]:::green
newImage[/Image of girl reading/]
captionStep["Image-to-text"]:::green
caption[/"A girl is sitting on a bed reading a book."/]
speechStep["Text-to-speech"]:::green
audio[/Audio/]
poseStepNote["openpose model"]:::note
imageStepNote["google/vit model"]:::note
captionStepNote["vit-gpt2 model"]:::note
speechStepNote["fastspeech model"]:::note
end
request --> planner --> inputImage
inputImage --> poseStep --> pose --> imageStep --> newImage --> captionStep --> caption --> speechStep --> audio
poseStep -.- poseStepNote
imageStep -.- imageStepNote
captionStep -.- captionStepNote
speechStep -.- speechStepNoteMulti-agent collaboration
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
request["Create a marketing brochure<br/>for a new pair of sunglasses"]:::red
manager["Manager agent (LLM):<br/>plans and delegates"]:::blue
output[Brochure]
subgraph team [Specialist agents]
researcher["Researcher agent (LLM):<br/>analyses market trends and competitors"]:::orange
designer["Graphic designer agent (LLM):<br/>creates visuals for the brochure"]:::orange
writer["Writer agent (LLM):<br/>writes the brochure copy"]:::orange
end
subgraph tools [Tools]
webSearch["Web search"]:::green
imageGen["Image generation"]:::green
end
request --> manager <--> team --> tools
manager --> output
Reflection
Rendering diagram...
Rendering diagram...
Tool use
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
No title
Rendering diagram...
A reasoning model aka thinking model, is better at finding bugs.
Rendering diagram...
Reflection with external feedback
Rendering diagram...
Rendering diagram...
Why not direct generation?
No title
Research shows that Reflection consistently out performs direct generation.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
Convert to MM/DD/YYYY format
Input: Jan 1st, 2025
Output: 01/01/2025
Input: 21st June, 2025
Output: 06/21/2025
Input: {input_date}Examples of reflection prompt
Brainstorming domain names
Review the domain names you suggested.
Check if each name is easy to pronounce and thus easy to spread via word of mouth.
Consider whether each name might mean something negative in other languages.
Then output a shortlist of only the names that meet these criteria.Improving email
Review the email first draft.
Check that the tone is professional and look for phrases that could be considered rude or insensitive.
Verify all facts, dates, and promises are accurate.
Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
Zero-shot (no examples)
1Convert to MM/DD/YYYY format
2
3Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: {input_date}One-shot (single example)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: {input_date}Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Evaluating the impact of Reflection
Create a dataset of prompts and answers (Objective Reflection)
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
prompt["Which color of product has<br/>the highest total sales?"]:::red
llm1["LLM:<br/>generate SQL query"]:::blue
subgraph reflection [Reflection]
llm2["LLM:<br/>reflect on V1 SQL,<br/>write V2 query"]:::blue
end
execute["Execute V2 query"]:::green
results["V2 query results"]
llm3["LLM:<br/>answer question"]:::blue
prompt --> llm1 --> llm2 --> execute --> results --> llm3
subgraph evals [Evals]
dataset[("Dataset:<br/>prompts + ground truth answers")]
compare["Compare answer with<br/>ground truth (code)"]:::green
accuracy["Accuracy<br/>no reflection vs with reflection"]
dataset --> compare --> accuracy
end
dataset -.->|prompts| prompt
llm3 --> compare
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5
style evals stroke:#16a34a,stroke-width:2px,stroke-dasharray: 5 5Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
plotV1[/plot.png/]
plotV2[/plot_v2.png/]
judge["LLM:<br/>which image is better?"]:::blue
judgeNote["Answers often not very good<br/>Position bias: LLM picks A more often"]:::note
plotV1 & plotV2 --> judge
judge -.- judgeNoteKnown issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
plot[/plot.png or plot_v2.png/]
judge["LLM:<br/>score against rubric"]:::blue
scores["JSON scores<br/>e.g. 6 / 8"]
plot --> judge --> scoresAssess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...How to build tools for LLM use
Primitive function keyword
from datetime import datetime
def get_current_time(): -> str:
return datetime.now().strftime('%I:%M%p')Rendering diagram...
AI suite
from datetime import datetime
def get_current_time(): -> str:
"""Returns the current time as a string"""
return datetime.now().strftime('%I:%M%p')import aisuite as ai
client = ai.Client()
response = client.chat.completions.create(
model="openai:gpt-4o",
messages=messages,
tools=[get_current_time]
max_turns=5
)Rendering diagram...
Modern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
Primitive function keyword
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')Rendering diagram...
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
MCP
MCP (Model Context Protocol) is an open protocol that lets LLM applications connect to external tools, data sources, and services through a single standard interface, instead of custom integrations for each one.
MCP clients, allows you to fetch data from MCP servers.
MCP servers, allows access to data.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
Evaluations
Examples
Invoice processor evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
fields["<b>4 required fields</b><br/>Biller<br/>Biller address<br/>Amount due<br/>Due date"]:::note
invoice["Invoice PDF"]:::red
pdf["PDF to text"]:::green
llm["LLM"]
subgraph tools["Tools"]
toolDb["update<br/>database"]:::green
end
update["update<br/>database"]:::green
record["Record<br/>created!"]
test_data[(Test Data)]
fields -.-> llm
subgraph metrics["Eval"]
date_eval["Evaluate date format"]
update_metric["Update Metric"]
metric_note["GROUND TRUTH:\nCorrectness score<br/>Tracks how many of the 20 examples<br/>have the right due date.<br/>Compare it after every change to the<br/>prompt or workflow: keep changes that<br/>raise it, revert those that lower it."]:::note
end
test_data -->|20 examples| invoice
invoice --> pdf --> llm --> date_eval --> update_metric --> update --> record
llm -.-> tools
update_metric -.- metric_noteResearch agent evals
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
query["User query"]:::red
subgraph tools["Tools"]
t_search["web search"]:::green
t_fetch["web fetch"]:::green
t_pdf["PDF to text"]:::green
end
agent["<b>Research agent (LLM)</b><br/>1. Search web<br/>2. Fetch 5 best sources<br/>3. Write essay draft"]
query --> agent --> llm_as_judge
agent -.-> tools
test_data[(Test Data)]
test_data -->|20 topic examples| query
subgraph metrics["Eval"]
llm_as_judge["<b>LLM-as-a-judge</b><br/>Counts gold-standard<br/>points in the essay"]
rubric_score["Rubric Score<br/>0-5"]:::green
judge_prompt["<b>LLM-as-a-judge prompt</b><br/>Determine how many of the 5 gold-standard<br/>talking points are present in the essay.<br/><br/><b>Inputs:</b> {original_prompt}, {essay_text},<br/>{gold_standard_points}<br/><b>Output:</b> JSON with<br/>score (0-5) and explanation<br/>(talking points present)"]:::note
rubric_note["<b>Rubric score</b><br/>Gold-standard talking points<br/>(3-5 per topic) are the ground truth.<br/>Score = how many appear in the essay.<br/>Average across the eval set to compare<br/>prompt and workflow changes."]:::note
end
gold["GROUND TRUTH:\nGold-standard talking points<br/>Black holes: Event horizon, radio telescope<br/>Robotic harvesting: RoboPick, pinchers"]:::note
llm_as_judge --> rubric_score
gold -..-> llm_as_judge
judge_prompt -.- llm_as_judge
rubric_score -.- rubric_note
rubric_score --> essay["Essay draft"]Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Error Analysis Process
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Component Level Evals
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
No title
Re-running the entire pipeline every time you tweak one component is slow and expensive.
Instead, build evals that test each component on its own.
For example, to test the web search component:
Create a list of gold-standard web resources for each query.
Compute a rubric score for how many of the returned results match those gold-standard resources.
Benefits of component level evaluations
Can provide cleaner signals for specific errors.
Avoids the noise from end to end system runs.
Lets each team own and track the metric for its own component.
How to address problems you identify?
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
Non LLM components
Non-LLM components often have hyperparameters you can tune, for example the web search settings, or how websites are chunked before being passed to the LLM.
Replace the component, try a different web search provider, etc.
LLM components
Improve your prompts, (improve instructions, add examples, with few-shot prompting)
Try a new model
Decompose a task into multiple steps
Fine-tune a model. (Should be the last option you consider because its expensive)
How to develop intution for model intelligence?
Play with models often
Have a personal set of evals
Read other peoples prompt for ideas of how to best use models
Use different models in your agentic workflows
See traces and component level evals
ai-suite makes it easy to quickly swap out models
Latency and Cost optimisation
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Latency
Benchmark each component runs
Consider parallelism
Try smaller models or a faster LLM provider if a step takes longer.
Costing your workflow
Consider token usage for each component.
No title
Get the output quality right before optimizing anything else.
Cost and latency are good problems to have: they only matter once the workflow works.
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Planning system design
Rendering diagram...
Because the plan is generated at run time,
developers can't anticipate its failure modes ahead of time,
which makes the system hard to control.
Rendering diagram...
How to get LLM's to create a good plan
Ask LLM to format the plan as a JSON / XML formats.
1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}1{
2 "plan": [
3 {
4 "step": 1,
5 "description": "Find round sunglasses",
6 "tool": "get_item descriptions",
7 "args": { "query": "round Sunglasses" }
8 },
9 {
10 "step": 2,
11 "description": "Check available stock",
12 "tool": "check_inventory",
13 "args": {
14 "items": "results from step 1"
15 }
16 }
17 ]
18}Planning with code executions
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
flowchart LR
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>Write code to solve the user's query.<br/>Return your answer as python code delimited<br/>with <execute_python> and </execute_python> tags."]:::note
query["How many unique<br/>transactions last week?"]:::red
answer["LLM answers<br/>the user"]:::blue
subgraph reflection [Reflection with external feedback]
llm["LLM:<br/>write code, or reflect on<br/>output/errors and fix it"]:::blue
code["<b>Generated code</b>\n<br/><execute_python><br/>import pandas as pd<br/>df = pd.read_csv('transactions.csv', parse_dates=['date'])<br/>today = pd.Timestamp.today()<br/>week_ago = today - pd.Timedelta(days=7)<br/>last_week = df[df['date'].between(week_ago, today)]<br/>print(last_week.drop_duplicates().shape[0])<br/></execute_python>"]
subgraph sandbox["Sandbox (e.g. Docker, E2B)"]
exec["Execute code"]:::green
end
result["Result or error"]
end
sysPrompt -.-> llm
query --> llm --> code --> exec --> result
result -->|feedback| llm
result -->|result is correct| answer
style reflection stroke:#cc0000,stroke-width:2px,stroke-dasharray: 5 5Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#ffcccc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
sysPrompt["<b>System prompt</b><br/>You are a marketing manager and have the following<br/>team of agents to work with: {description of agents}.<br/>Return a step-by-step plan to carry out the user's request."]:::note
query["Create a summer marketing<br/>campaign for sunglasses"]:::red
answer["Final report"]:::blue
subgraph planner [Planner]
llm1["LLM"]
llm2["LLM"]
llm3["LLM"]
llm4["LLM"]
end
subgraph agents [Agents]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
end
sysPrompt -.-> llm1
query --> llm1
llm1 -->|"1. Research current sunglasses trends"| researcher
researcher --> llm2
llm2 -->|"2. Create ad images"| designer
designer --> llm3
llm3 -->|"3. Create report"| writer
writer --> llm4
llm4 -->|"4. Review report"| answerMulti Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writerDeeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
flowchart TD
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
webResearcher["web researcher"]:::blue
factChecker["fact checker"]:::blue
styleWriter["style writer"]:::blue
citationChecker["citation checker"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> webResearcher
researcher <--> factChecker
writer <--> styleWriter
writer <--> citationCheckerAll to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
flowchart LR
classDef blue fill:#cce5ff,stroke:#000,stroke-width:2px,color:#000000;
manager["marketing manager"]:::blue
researcher["researcher"]:::blue
designer["graphic designer"]:::blue
writer["writer"]:::blue
manager <--> researcher
manager <--> designer
manager <--> writer
researcher <--> designer
researcher <--> writer
designer <--> writerThe challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
Rendering diagram...
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
Rendering diagram...
Planning
Rendering diagram...
Rendering diagram...
Multi-agent collaboration
Rendering diagram...
Rendering diagram...
Two-shot (two examples), or few-shot (multiple examples)
1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}1Convert to MM/DD/YYYY format
2
3Input: Jan 1st, 2025
4Output: 01/01/2025
5
6Input: 21st June, 2025
7Output: 06/21/2025
8
9Input: {input_date}Examples of reflection prompt
Brainstorming domain names
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.Improving email
1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Tips: clearly ask the model to review its output, give it specific criteria to check against, and end with what it should produce next.
1Review the domain names you suggested.
2
3Check if each name is easy to pronounce and thus easy to spread via word of mouth.
4
5Consider whether each name might mean something negative in other languages.
6
7Then output a shortlist of only the names that meet these criteria.1Review the email first draft.
2
3Check that the tone is professional and look for phrases that could be considered rude or insensitive.
4
5Verify all facts, dates, and promises are accurate.
6
7Then write the next draft of the email.Create a dataset of prompts and answers (Objective Reflection)
Rendering diagram...
Run the dataset each time you change the reflection prompt, and compare against the ground truth:
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
Rendering diagram...
Number of items sold in May 2025?
1201
980
1201
Most expensive item?
Airflow sneaker
Airflow sneaker
Airflow sneaker
How many styles carried?
14
14
14
Accuracy
87% correct
95% correct
LLM as judge (Subjective Reflection)
No title
When there's no ground truth to compare against (e.g. "is this chart good?"), use an LLM to grade the output.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Grading with a rubric
Rendering diagram...
Assess the attached image against this quality rubric. Each item should
receive a score of 1 (true) or 0 (false). Return the scores for each item
as a json object.
1. Has clear title
2. Axis labels present
3. Appropriate chart type
4. Axes use appropriate numerical range
5. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Comparing two outputs
Rendering diagram...
Known issues with asking an LLM to compare outputs:
Answers are often not very good.
Position bias: the LLM tends to pick whichever option is shown first (A over B).
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Using external feedback
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Mentioning competitors
Our company's shoes are better than RivalCo
Pattern matching for competitor names
Fact checking an essay
The Taj Mahal was built in 1648
Web search results
LLM won't follow output length guidelines
Essay is over word limit
Word count tool
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...1from datetime import datetime
2
3def get_current_time(): -> str:
4 return datetime.now().strftime('%I:%M%p')AI suite
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)sequenceDiagram
autonumber
actor User as User Script
participant Suite as aisuite.Client
participant Tool as Local Function (get_current_time)
participant LLM as OpenAI (gpt-4o)
User->>Suite: chat.completions.create(model, messages, tools=[...], max_turns=5)
activate Suite
Suite->>Suite: Inspect function signature & docstring<br/>to build JSON schema
loop Up to max_turns
Suite->>LLM: Send messages + tool schemas
activate LLM
LLM-->>Suite: Return response (Tool Call requested)
deactivate LLM
alt Tool Call requested
Suite->>Tool: Execute get_current_time()
activate Tool
Tool-->>Suite: Return "10:32PM"
deactivate Tool
Suite->>Suite: Append Tool Call & Tool Result to messages
else Normal text response (stop)
Suite-->>User: Return final completion response
end
end
deactivate SuiteModern LLMs are fine-tuned to recognize when to invoke external tools and generate structured arguments.
Supported models return explicit function-calling metadata (such as tool call IDs, function names, and arguments) within the message payload.
Client-side SDKs and orchestration frameworks parse these structured payloads into executable function calls automatically.
1from datetime import datetime
2
3def get_current_time(): -> str:
4 """Returns the current time as a string"""
5 return datetime.now().strftime('%I:%M%p')1import aisuite as ai
2client = ai.Client()
3
4response = client.chat.completions.create(
5 model="openai:gpt-4o",
6 messages=messages,
7 tools=[get_current_time]
8 max_turns=5
9)Code execution
Rendering diagram...
Allowing LLM's to run code outside of a sandbox can be highly risky. (they could run some distructive commands lik rm *.py by misate).
Use sandbox tools like docker, E2B to safegaurd your environemnt.
Rendering diagram...
Rendering diagram...
Invoice processor evals
Rendering diagram...
Rendering diagram...
Research agent evals
Rendering diagram...
Rendering diagram...
Two "axes" of evaluation
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Per example ground truth
Checking invoice date extractionif (extracted_date == actual_date): num_correct += 1
Counting gold-standard talking points
"Count the number of gold standard points in the following text..."
No per example ground truth
Checking marketing copy lengthif len(text) <= 10: num_correct += 1
Grading charts with a rubric
"Grade this chart according to (i) whether it has clear axes labels, (ii) ...."
Tips and Trics
Quick and dirty evals are okay to start.
Improve your evals, when they fail to capture human judgement. Improve it as a metric rather than trying to start with perfection.
Look for places where performance is worst than humans
Setting up Traces
No title
Rendering diagram...
Try to identify the points of failure in the pipeline.
See if a human expert would do it differently.
See intermidate outputs for issues.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
The challenge of planning with tools
Every new kind of question needs a new tool, so the tool list keeps growing.
Even a simple question needs a long plan that chains many tool calls.
Rendering diagram...
Planning with code execution
Instead of one tool per query, let the LLM write code that solves the query.
Run the code in a sandbox, so a mistake in the generated code can't harm your environment.
One "execute code" capability replaces the whole growing list of tools.
Rendering diagram...
Rendering diagram...
Multi agentic workflows
Some tasks are too broad for one person and need several specialists, such as a researcher, a graphic designer, and a writer.
Rendering diagram...
Rendering diagram...
Multi Agent Communication Patterns
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Hierarical Communications
A manager agent coordinates a flat team of agents and talks to each one directly.
Rendering diagram...
Rendering diagram...
Deeper Hierarichal Communications
Agents can manage their own sub-agents, so a complex sub-task is delegated further down the tree.
Rendering diagram...
Rendering diagram...
All to All
Every agent can talk to every other agent, with no fixed manager.
Hard to control
Caohs and unpredictable
Rendering diagram...
Rendering diagram...
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Grading with a rubric
Rendering diagram...
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...User query 1
4
6
User query 2
5
8
...
...
...
User query 10
5
7
Why a rubric is better than asking "which is better?"
Consistent: each output is scored on its own against fixed criteria, so the same output gets the same score across runs.
No position bias: outputs aren't compared side by side, so the order they're shown in doesn't matter.
Binary checks are easier to judge: "Has a clear title? 1 / 0" is a much easier question for an LLM than a vague overall quality call.
Actionable: the per-item scores show what is wrong (e.g. missing axis labels), which feeds straight into the next reflection.
Comparable over time: the totals give you a number to track as you change the reflection prompt, like the accuracy column for objective evals.
1Assess the attached image against this quality rubric. Each item should
2receive a score of 1 (true) or 0 (false). Return the scores for each item
3as a json object.
4
51. Has clear title
62. Axis labels present
73. Appropriate chart type
84. Axes use appropriate numerical range
95. ...Rendering diagram...
Counting up the errors
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction
Carry out error analysis to figure out which component performed poorly.
Focus on those areas to improve over-all output.
Recent developments in black hole science
Too many blog posts, not enough papers
Renting vs buying a home in Seattle
Missed well-known blog
Robotics for harvesting fruit
Terms too generic
Website for elementary school students
Batteries for electric vehicles
Only selected US-based companies
Missed magazine
5% dissatisfaction
45% dissatisfaction
10% dissatisfaction