An Innovathon 2026 proposal: a suite of research agents that works out which software licence each employee actually needs, from what they do in the app and from evidence on the vendor's own pages.
It builds on the existing licence usage dashboard PoC. The one-pager is on Confluence.
Contents
Why
To know which licence a person needs, you need two facts: which parts of the app they use, and which licence each part requires.
The PoC gets both, but in a fixed order. It first scrapes the vendor's whole documentation site and builds a catalogue of every licence and module, then clusters page-view traffic into the modules people use, and finally joins the two. The traffic half works well. The documentation half is where it struggles:
It has to guess up front which of thousands of pages matter, and most of its tuning goes into that guess.
The modules it finds in the docs are named differently from the ones it finds in traffic ("Salesforce CPQ" against "Quote Management"), so a separate step has to link them.
It can only see what vendors publish. On Datadog it predicted 33 admins where there were 2, because the customer's own roles never appear in vendor documentation.
Ground Truth flips the order. Traffic decides what gets researched. Each module people actually use triggers a small investigation into what that module is and which licence it needs, and every answer has to carry a quote that code can find on the page. Answers without evidence are marked unresolved, not guessed.
How it works
Six steps run in order. Steps 1, 2 and 5 are workflows: they follow fixed steps, and step 5, the Bundle Finder, is plain code with no LLM. Steps 3, 4 and 6 are research agents: they choose their own steps.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
pageVisits[(Page Visit\nEmbeddings)]:::orange
suggest[1. Suggest\nModule Names]:::blue
merge[2. Merge\nModule Names]:::blue
moduleResearcher[3. Module\nResearcher]:::blue
licenceResearcher[4. Licence\nResearcher]:::blue
bundleFinder[5. Bundle Finder\ncode only]
tierResearcher[6. Tier\nResearcher]:::blue
tagVisits[Tag Visits with\na Module Name]
predict[Predict Licence Usage\nexisting PoC, SQL]
dashboard[/Licence Usage Dashboard\nwho needs which licence/]:::green
pageVisits --> |untagged clusters| suggest --> |suggested names| merge
merge --> |joins a saved module| tagVisits
merge --> |new module| moduleResearcher
moduleResearcher --> |confirmed module| tagVisits
moduleResearcher --> |confirmed module| licenceResearcher
licenceResearcher --> |licence facts| bundleFinder --> |bundle groups| tierResearcher
tagVisits --> predict
licenceResearcher --> predict
tierResearcher --> |tier order| predict
predict --> dashboard1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
The last step, predicting who needs which licence, is the PoC's existing SQL join. It doesn't change.
How to read the diagrams
All diagrams use the same shapes and colours. Every orange step goes through the LLM call path. A rubric check is answered by the model about its own output, so nothing verifies it: a doubtful answer sends the item to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph steps[Steps]
direction TB
llmStep[Calls Claude]:::orange
codeStep[Runs code]
model((Model used))
end
subgraph checks[Checks]
direction TB
codeCheck{"Code check:\ntrusted"}
rubricCheck{"Rubric check:\nself-graded"}:::rubric
reflection{"Reflection:\na second\nmodel call"}:::orange
end
subgraph stores[Stores]
direction TB
cache[(Firecrawl cache)]:::yellow
store[(Table or\nvector store)]:::orange
end
subgraph results[Results]
direction TB
kept[/Kept/]:::green
dropped[/Dropped, failed\nor unresolved/]:::red
end
subgraph other[Links and notes]
direction TB
link[/Drawn in full in\nanother diagram/]:::next
note["A note"]:::note
end
codeCheck --> |yes| kept
codeCheck --> |no| droppedDesign principles
These hold for every agent below.
Evidence first. A claim is kept only with a quote, and code checks that the quote is on a page the agent actually read. This is the proof-of-work step in every agent.
Three layers of checks, cheapest first. Code checks run first and can be trusted. Rubric questions come next: the model answers them about its own output, so they only decide whether an item needs review. Reflection comes last, where a second model call reviews what is still doubtful.
An independent reviewer. Reflection uses Opus, and where Opus also did the research, it reviews with a fresh context: the evidence, not the reasoning that produced it.
Safe defaults. When an agent is unsure, it takes the outcome that costs least if wrong: a name is dropped and its pages are retried on the next run, a new module is suggested instead of a risky merge, or a research question is left unresolved.
Bounded autonomy. The research agents choose their own next step, but within a budget of tool calls and pages, on vendor domains only, and never scraping the same URL twice.
Pay once. Every Firecrawl answer is cached, so a page is never paid for twice, and cleaning rules can change without fetching the page again.
Shared building blocks
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
sendPrompt[Send Prompt]:::orange
networkOk{"Network ok?"}
backOff[Back Off]
shapeOk{"Answer usable?"}
callFailed[/Call Failed/]:::red
toProofOfWork[/Proof of Work/]:::next
sendPrompt --> networkOk
networkOk --> |no, attempt 1-2| backOff --> sendPrompt
networkOk --> |no, attempt 3| callFailed
networkOk --> |yes| shapeOk
shapeOk --> |no| callFailed
shapeOk --> |yes| toProofOfWork
networkNote["Network error, throttle or 5xx<br/>Retried after 1s, then 2s"]:::note
networkOk -.- networkNote
shapeNote["JSON of the expected shape,<br/>not cut off at max_tokens<br/>Not retried"]:::note
shapeOk -.- shapeNote
failedNote["The step's safe outcome:<br/>pages stay untagged, a new module<br/>is suggested, or Unresolved"]:::note
callFailed -.- failedNoteProof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromLlm[/From LLM call:\nanswer + quotes\n+ rubric answers/]:::next
subgraph proofOfWork[Proof of Work]
codeChecks{"Code checks:\nis the evidence real?"}
rubricChecks{"Rubric checks:\ndoes the answer\nhold up?"}:::rubric
end
kept[/Kept/]:::green
failed[/Safe default, or\nback to research\nwith the reason/]:::red
toReflection[/Reflection/]:::next
fromLlm --> codeChecks
codeChecks --> |no| failed
codeChecks --> |yes| rubricChecks
rubricChecks --> |yes| kept
rubricChecks --> |doubtful| toReflection
codeNote["e.g. quote found word for word<br/>on a chunk the model read"]:::note
codeChecks -.- codeNote
rubricNote["e.g. #quot;do the examples<br/>support the name?#quot;"]:::note
rubricChecks -.- rubricNoteReflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged item +\nits evidence/]:::next
review{"Does the evidence\nsupport it?"}:::orange
recheck{"Rename or new answer\npasses proof of work?"}
kept[/Kept/]:::green
safeDefault[/Safe default/]:::red
backToResearch[/Back to research,\nwith what is missing/]:::next
flagged --> review
claudeOpus --> review
review --> |yes| kept
review --> |no| safeDefault
review --> |a better answer| recheck
recheck --> |yes| kept
recheck --> |no| safeDefault
review --> |needs more evidence| backToResearch
recheckNote["Checked once only,<br/>so reflection can't loop"]:::note
recheck -.- recheckNote
backNote["Research agents only<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNoteHow each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Workflows
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNote2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
moduleEmbeddings[(Module\nEmbeddings)]:::orange
end
subgraph Tables
modules[(Modules)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
hybridModuleSearch[Hybrid Module Search\nBM25 + vector]
end
suggestions[/Suggested Module Names/]:::green
gatherSuggestion[1. Gather Suggestion\nname + its examples]
toSimilarityCheck[/2. Similarity Check/]:::next
subgraph llmSimilarityCheck[LLM Similarity Check]
comparePrompt[3. Prompt LLM to Compare\nsuggestion vs candidate]:::orange
toProofOfWork[/4. Proof of Work/]:::next
toReflection[/5. Reflection/]:::next
end
joinAtSimilarity[/Join Saved Module/]:::green
newAtSimilarity[/Suggest as New Module/]:::green
joinAtProofOfWork[/Join Saved Module/]:::green
newAtProofOfWork[/Suggest as New Module/]:::green
joinAtReflection[/Join Saved Module/]:::green
newAtReflection[/Suggest as New Module/]:::green
moduleEmbeddings --> hybridModuleSearch
modules --> hybridModuleSearch
suggestions --> gatherSuggestion
hybridModuleSearch --> |top K saved modules:\nname, description,\nexamples| gatherSuggestion
gatherSuggestion --> toSimilarityCheck
toSimilarityCheck --> |match| joinAtSimilarity
toSimilarityCheck --> |no candidate| newAtSimilarity
toSimilarityCheck --> |candidate| comparePrompt
claudeSonnet --> comparePrompt
comparePrompt --> |same module? +\nrubric answers +\ncited URLs| toProofOfWork
toProofOfWork --> |passes| joinAtProofOfWork
toProofOfWork --> |fails| newAtProofOfWork
toProofOfWork --> |flagged| toReflection
toReflection --> |yes| joinAtReflection
toReflection --> |no, or no answer| newAtReflection
searchNote["Vector: name embedding similarity<br/>BM25: name, merged names, description<br/>Scores pooled, top K kept"]:::note
hybridModuleSearch -.- searchNote
compareNote["Sees the saved module's<br/>name and description<br/>+ 8 pages of each<br/>At most 3 candidates per name"]:::note
comparePrompt -.- compareNote
joinNote["Suggested name saved in<br/>merged_names: next time<br/>it matches at step 2"]:::note
joinAtSimilarity -.- joinNoteSimilarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNote5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
collectEdges[1. Collect\nincluded-in edges]
includesLicence{"Something sold\non its own\ncomes inside it?"}
bundle[/Bundle/]:::green
licence[/Licence/]:::green
groupByVendor[2. Group Bundles\nby vendor]
toTierResearcher[/Tier Researcher/]:::next
moduleLicences --> collectEdges --> includesLicence
includesLicence --> |yes| bundle --> groupByVendor --> toTierResearcher
includesLicence --> |no| licence
edgesNote["Module M comes inside X<br/>and M is sold on its own:<br/>X includes a licence"]:::note
includesLicence -.- edgesNote
groupNote["One group per vendor<br/>e.g. Starter, Professional,<br/>Enterprise"]:::note
groupByVendor -.- groupNoteResearch Agents
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNote4. Licence Researcher
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
confirmedModules[/Confirmed Modules/]:::green
researchLoop[Research Loop]:::blue
licenceFacts[/Licence Facts/]:::green
unresolved[/Unresolved,\nretried next run/]:::red
confirmedModules --> |name, description,\ncitations| researchLoop
researchLoop --> |confirmed| licenceFacts
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: is it sold on its own?<br/>Plus one question per licence<br/>it may come inside<br/>Can be both"]:::note
researchLoop -.- checklistNote
factsNote["standalone: yes or no<br/>included in: licence names<br/>each with its quote"]:::note
licenceFacts -.- factsNoteProof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\nclaim, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesModule{"Quote names\nthe module?"}
claimType{"Which claim?"}
namesLicence{"Quote names every\nlicence it comes inside?"}
ownPrice{"Rubric:\nGives it its own\nprice, plan or SKU?"}:::rubric
onlyInside{"Rubric:\nSays it only comes\ninside other licences?"}:::rubric
inPassing{"Rubric:\nOnly listed\nin passing?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Licence Facts/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesModule
namesModule --> |no| backToNextAction
namesModule --> |yes| claimType
claimType --> |sold on its own| ownPrice
claimType --> |comes inside| namesLicence
namesLicence --> |no| backToNextAction
namesLicence --> |yes| onlyInside
ownPrice --> |yes| inPassing
ownPrice --> |no| toReflection
onlyInside --> |yes| inPassing
onlyInside --> |no| toReflection
inPassing --> |no| passes
inPassing --> |yes| toReflection
namesModuleNote["The module's name<br/>or one of its aliases"]:::note
namesModule -.- namesModuleNote
ownPriceNote["e.g. its own row<br/>in a pricing table"]:::note
ownPrice -.- ownPriceNote
onlyInsideNote["e.g. #quot;included in every<br/>Enterprise plan#quot;"]:::note
onlyInside -.- onlyInsideNote
inPassingNote["A feature grid or a<br/>#quot;works with#quot; list"]:::note
inPassing -.- inPassingNoteReflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNote6. Tier Researcher
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
bundleGroup[/Bundle Group\nfrom Bundle Finder/]:::next
researchLoop[Research Loop]:::blue
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
bundleGroup --> |bundle names, the\nmodules each includes| researchLoop
researchLoop --> |confirmed| tierOrder
researchLoop --> |rejected| notNested
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: one question<br/>per pair of bundles:<br/>does A include everything in B?"]:::note
researchLoop -.- checklistNote
tierOrderNote["Each pair with its quote<br/>Lets a user drop to the<br/>lowest tier they need"]:::note
tierOrder -.- tierOrderNoteProof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\npairs, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesBoth{"Quote names\nboth bundles?"}
noCycles{"Order has\nno cycles?"}
statesSuperset{"Rubric:\nSays one has everything\nin the other?"}:::rubric
aboutThesePlans{"Rubric:\nAbout these bundles,\nnot plans in general?"}:::rubric
comparisonTable{"Rubric:\nOnly a comparison\ntable cell?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Tier Order/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesBoth
namesBoth --> |no| backToNextAction
namesBoth --> |yes| noCycles
noCycles --> |no| backToNextAction
noCycles --> |yes| statesSuperset
statesSuperset --> |no| toReflection
statesSuperset --> |yes| aboutThesePlans
aboutThesePlans --> |no| toReflection
aboutThesePlans --> |yes| comparisonTable
comparisonTable --> |yes| toReflection
comparisonTable --> |no| passes
noCyclesNote["A includes B and B includes A<br/>means one answer is wrong"]:::note
noCycles -.- noCyclesNote
supersetNote["e.g. #quot;everything in<br/>Professional, plus#quot;"]:::note
statesSuperset -.- supersetNote
tableNote["Its meaning is in the<br/>table's header, not the cell"]:::note
comparisonTable -.- tableNoteReflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteHow we will measure it
One app with known ground truth, end to end. Salesforce is the likely first choice.
Side by side with the current pipeline: accuracy against the ground truth, pages scraped, and cost.
Component evals: clusters that are already tagged are ground truth for step 1, and saved merges are ground truth for step 2.
How often reflection overturns a draft. If it almost never does, the rubric and reflection steps can be cut to save cost.
Open questions
Planning. The researchers choose one step at a time. Adding a short up-front plan of the questions to answer would make "enough evidence" something code can check.
Embedding new pages. The URL Scraper doesn't embed, so pages scraped during a run aren't searchable in Vendor Documentation Embeddings until something embeds them.
What documentation can't show. Customer-defined roles and usage-metered products don't appear in vendor docs. They need the vendor's own admin or usage APIs.
Shelfware. A licence that is paid for but never used produces no traffic, so nothing triggers research on it. Finding it needs the customer's entitlement data.
Numbers to tune. The research budget (15 tool calls, 10 pages) is a placeholder. The merge thresholds (0.95 and 0.6), the 8 pages per module and the 3 candidates per name come from the PoC.
Cost. Opus drives every research loop, so the budget matters more than it would with Sonnet.
Tool Reference
No title
The agents reach the outside world through two Firecrawl tools. Both check the cache before spending credits, and both fill it afterwards. A third tool, the Checklist Updater, works on the agent's own checklist.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlSearchCache[(Firecrawl Domain\nSearch Cache)]:::yellow
end
request[/query + domains + limit/]
buildKey[1. Build Cache Key]
inCache{"2. In cache?"}
firecrawlSearch[3. Firecrawl Search\nweb results on\nthese domains only]:::blue
gotAnswer{"Answer after\nretries?"}
writeCache[4. Write to Cache]
cleanResults[5. Clean Results]
results[/Pages + metadata/]:::green
failed[/Error, nothing cached/]:::red
request --> buildKey --> inCache
firecrawlSearchCache --> inCache
inCache --> |yes| cleanResults
inCache --> |no| firecrawlSearch --> gotAnswer
gotAnswer --> |yes| writeCache --> cleanResults
gotAnswer --> |no| failed
writeCache --> firecrawlSearchCache
cleanResults --> results
keyNote["search:domains:query:limit<br/>Domains sorted, so order<br/>doesn't change the key"]:::note
buildKey -.- keyNote
refreshNote["refresh: skips the read,<br/>still writes the new answer"]:::note
inCache -.- refreshNote
retryNote["Rate-limited and retried<br/>with backoff"]:::note
firecrawlSearch -.- retryNote
cleanNote["Conservative: only removes<br/>duplicate and off-domain URLs"]:::note
cleanResults -.- cleanNote
resultsNote["url, title, description,<br/>position (search rank),<br/>category"]:::note
results -.- resultsNoteURL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlURLScrapeCache[(Firecrawl URL\nScrape Cache)]:::yellow
end
request[/url/]
inCache{"1. In cache?"}
firecrawlScrape[2. Firecrawl Scrape\nmain content\nas Markdown]:::blue
gotMarkdown{"Markdown after\nretries?"}
writeCache[3. Write Raw\nMarkdown to Cache]
skipped[/Skipped, nothing cached/]:::red
subgraph cleanUp[4. Clean Up]
usablePage{"Dead, empty or\na sign-up form?"}
cleanMarkdown[Strip page chrome]
droppedPage[/Page dropped/]:::red
end
chunkPage[5. Chunk Page]
page[/Page chunks/]:::green
request --> inCache
firecrawlURLScrapeCache --> inCache
inCache --> |yes| usablePage
inCache --> |no| firecrawlScrape --> gotMarkdown
gotMarkdown --> |yes| writeCache --> usablePage
gotMarkdown --> |no| skipped
writeCache --> firecrawlURLScrapeCache
usablePage --> |yes| droppedPage
usablePage --> |no| cleanMarkdown --> chunkPage --> page
scrapeNote["Main content only:<br/>no nav, header, footer"]:::note
firecrawlScrape -.- scrapeNote
rawNote["Raw, not cleaned:<br/>re-clean for free"]:::note
writeCache -.- rawNote
usableNote["Sign-up form: 6+ form-field<br/>lines, no prices"]:::note
usablePage -.- usableNote
cleanNote["Removes: images, cookie<br/>and feedback widgets<br/>Keeps all links, so an agent<br/>can follow them"]:::note
cleanMarkdown -.- cleanNote
chunkNote["800 chars, 150 overlap<br/>Each starts with its heading trail<br/>Under 40 chars dropped"]:::note
chunkPage -.- chunkNote
pageNote["Per chunk: final URL,<br/>chunk index, text, links"]:::note
page -.- pageNoteThree smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
Technology
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
LangGraph nodes
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
ground_truth/
src/
core/ plain TypeScript, no LangChain, unit-tested
checklist/types.ts the checklist object
checklist/ops.ts add_item, set_note, propose_answer, ...
checklist/views.ts the Next Action, Draft and Reflection views
checklist/seeds/module.ts the four seeded items
evidence/ text normalisation and the verbatim quote check
urlRules.ts
budget.ts
tools/ LangChain tool wrappers: schemas and descriptions only
domainSearch.ts
scrapeUrl.ts
docSearch.ts
updateChecklist.ts
cache/firecrawlCache.ts raw Firecrawl answers on disk
llm/models.ts the Bedrock chat model
agents/moduleResearcher/
state.ts
graph.ts
prompts.ts
nodes/ seed, nextAction, callTools, draft, proofOfWork
reflection/graph.ts the reflection subgraph
cli/researchModule.ts
tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
An Innovathon 2026 proposal: a suite of research agents that works out which software licence each employee actually needs, from what they do in the app and from evidence on the vendor's own pages.
It builds on the existing licence usage dashboard PoC. The one-pager is on Confluence.
Contents
Why
To know which licence a person needs, you need two facts: which parts of the app they use, and which licence each part requires.
The PoC gets both, but in a fixed order. It first scrapes the vendor's whole documentation site and builds a catalogue of every licence and module, then clusters page-view traffic into the modules people use, and finally joins the two. The traffic half works well. The documentation half is where it struggles:
It has to guess up front which of thousands of pages matter, and most of its tuning goes into that guess.
The modules it finds in the docs are named differently from the ones it finds in traffic ("Salesforce CPQ" against "Quote Management"), so a separate step has to link them.
It can only see what vendors publish. On Datadog it predicted 33 admins where there were 2, because the customer's own roles never appear in vendor documentation.
Ground Truth flips the order. Traffic decides what gets researched. Each module people actually use triggers a small investigation into what that module is and which licence it needs, and every answer has to carry a quote that code can find on the page. Answers without evidence are marked unresolved, not guessed.
How it works
Six steps run in order. Steps 1, 2 and 5 are workflows: they follow fixed steps, and step 5, the Bundle Finder, is plain code with no LLM. Steps 3, 4 and 6 are research agents: they choose their own steps.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
pageVisits[(Page Visit\nEmbeddings)]:::orange
suggest[1. Suggest\nModule Names]:::blue
merge[2. Merge\nModule Names]:::blue
moduleResearcher[3. Module\nResearcher]:::blue
licenceResearcher[4. Licence\nResearcher]:::blue
bundleFinder[5. Bundle Finder\ncode only]
tierResearcher[6. Tier\nResearcher]:::blue
tagVisits[Tag Visits with\na Module Name]
predict[Predict Licence Usage\nexisting PoC, SQL]
dashboard[/Licence Usage Dashboard\nwho needs which licence/]:::green
pageVisits --> |untagged clusters| suggest --> |suggested names| merge
merge --> |joins a saved module| tagVisits
merge --> |new module| moduleResearcher
moduleResearcher --> |confirmed module| tagVisits
moduleResearcher --> |confirmed module| licenceResearcher
licenceResearcher --> |licence facts| bundleFinder --> |bundle groups| tierResearcher
tagVisits --> predict
licenceResearcher --> predict
tierResearcher --> |tier order| predict
predict --> dashboard1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
The last step, predicting who needs which licence, is the PoC's existing SQL join. It doesn't change.
How to read the diagrams
All diagrams use the same shapes and colours. Every orange step goes through the LLM call path. A rubric check is answered by the model about its own output, so nothing verifies it: a doubtful answer sends the item to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph steps[Steps]
direction TB
llmStep[Calls Claude]:::orange
codeStep[Runs code]
model((Model used))
end
subgraph checks[Checks]
direction TB
codeCheck{"Code check:\ntrusted"}
rubricCheck{"Rubric check:\nself-graded"}:::rubric
reflection{"Reflection:\na second\nmodel call"}:::orange
end
subgraph stores[Stores]
direction TB
cache[(Firecrawl cache)]:::yellow
store[(Table or\nvector store)]:::orange
end
subgraph results[Results]
direction TB
kept[/Kept/]:::green
dropped[/Dropped, failed\nor unresolved/]:::red
end
subgraph other[Links and notes]
direction TB
link[/Drawn in full in\nanother diagram/]:::next
note["A note"]:::note
end
codeCheck --> |yes| kept
codeCheck --> |no| droppedDesign principles
These hold for every agent below.
Evidence first. A claim is kept only with a quote, and code checks that the quote is on a page the agent actually read. This is the proof-of-work step in every agent.
Three layers of checks, cheapest first. Code checks run first and can be trusted. Rubric questions come next: the model answers them about its own output, so they only decide whether an item needs review. Reflection comes last, where a second model call reviews what is still doubtful.
An independent reviewer. Reflection uses Opus, and where Opus also did the research, it reviews with a fresh context: the evidence, not the reasoning that produced it.
Safe defaults. When an agent is unsure, it takes the outcome that costs least if wrong: a name is dropped and its pages are retried on the next run, a new module is suggested instead of a risky merge, or a research question is left unresolved.
Bounded autonomy. The research agents choose their own next step, but within a budget of tool calls and pages, on vendor domains only, and never scraping the same URL twice.
Pay once. Every Firecrawl answer is cached, so a page is never paid for twice, and cleaning rules can change without fetching the page again.
Shared building blocks
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
sendPrompt[Send Prompt]:::orange
networkOk{"Network ok?"}
backOff[Back Off]
shapeOk{"Answer usable?"}
callFailed[/Call Failed/]:::red
toProofOfWork[/Proof of Work/]:::next
sendPrompt --> networkOk
networkOk --> |no, attempt 1-2| backOff --> sendPrompt
networkOk --> |no, attempt 3| callFailed
networkOk --> |yes| shapeOk
shapeOk --> |no| callFailed
shapeOk --> |yes| toProofOfWork
networkNote["Network error, throttle or 5xx<br/>Retried after 1s, then 2s"]:::note
networkOk -.- networkNote
shapeNote["JSON of the expected shape,<br/>not cut off at max_tokens<br/>Not retried"]:::note
shapeOk -.- shapeNote
failedNote["The step's safe outcome:<br/>pages stay untagged, a new module<br/>is suggested, or Unresolved"]:::note
callFailed -.- failedNoteProof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromLlm[/From LLM call:\nanswer + quotes\n+ rubric answers/]:::next
subgraph proofOfWork[Proof of Work]
codeChecks{"Code checks:\nis the evidence real?"}
rubricChecks{"Rubric checks:\ndoes the answer\nhold up?"}:::rubric
end
kept[/Kept/]:::green
failed[/Safe default, or\nback to research\nwith the reason/]:::red
toReflection[/Reflection/]:::next
fromLlm --> codeChecks
codeChecks --> |no| failed
codeChecks --> |yes| rubricChecks
rubricChecks --> |yes| kept
rubricChecks --> |doubtful| toReflection
codeNote["e.g. quote found word for word<br/>on a chunk the model read"]:::note
codeChecks -.- codeNote
rubricNote["e.g. #quot;do the examples<br/>support the name?#quot;"]:::note
rubricChecks -.- rubricNoteReflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged item +\nits evidence/]:::next
review{"Does the evidence\nsupport it?"}:::orange
recheck{"Rename or new answer\npasses proof of work?"}
kept[/Kept/]:::green
safeDefault[/Safe default/]:::red
backToResearch[/Back to research,\nwith what is missing/]:::next
flagged --> review
claudeOpus --> review
review --> |yes| kept
review --> |no| safeDefault
review --> |a better answer| recheck
recheck --> |yes| kept
recheck --> |no| safeDefault
review --> |needs more evidence| backToResearch
recheckNote["Checked once only,<br/>so reflection can't loop"]:::note
recheck -.- recheckNote
backNote["Research agents only<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNoteHow each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Workflows
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNote2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
moduleEmbeddings[(Module\nEmbeddings)]:::orange
end
subgraph Tables
modules[(Modules)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
hybridModuleSearch[Hybrid Module Search\nBM25 + vector]
end
suggestions[/Suggested Module Names/]:::green
gatherSuggestion[1. Gather Suggestion\nname + its examples]
toSimilarityCheck[/2. Similarity Check/]:::next
subgraph llmSimilarityCheck[LLM Similarity Check]
comparePrompt[3. Prompt LLM to Compare\nsuggestion vs candidate]:::orange
toProofOfWork[/4. Proof of Work/]:::next
toReflection[/5. Reflection/]:::next
end
joinAtSimilarity[/Join Saved Module/]:::green
newAtSimilarity[/Suggest as New Module/]:::green
joinAtProofOfWork[/Join Saved Module/]:::green
newAtProofOfWork[/Suggest as New Module/]:::green
joinAtReflection[/Join Saved Module/]:::green
newAtReflection[/Suggest as New Module/]:::green
moduleEmbeddings --> hybridModuleSearch
modules --> hybridModuleSearch
suggestions --> gatherSuggestion
hybridModuleSearch --> |top K saved modules:\nname, description,\nexamples| gatherSuggestion
gatherSuggestion --> toSimilarityCheck
toSimilarityCheck --> |match| joinAtSimilarity
toSimilarityCheck --> |no candidate| newAtSimilarity
toSimilarityCheck --> |candidate| comparePrompt
claudeSonnet --> comparePrompt
comparePrompt --> |same module? +\nrubric answers +\ncited URLs| toProofOfWork
toProofOfWork --> |passes| joinAtProofOfWork
toProofOfWork --> |fails| newAtProofOfWork
toProofOfWork --> |flagged| toReflection
toReflection --> |yes| joinAtReflection
toReflection --> |no, or no answer| newAtReflection
searchNote["Vector: name embedding similarity<br/>BM25: name, merged names, description<br/>Scores pooled, top K kept"]:::note
hybridModuleSearch -.- searchNote
compareNote["Sees the saved module's<br/>name and description<br/>+ 8 pages of each<br/>At most 3 candidates per name"]:::note
comparePrompt -.- compareNote
joinNote["Suggested name saved in<br/>merged_names: next time<br/>it matches at step 2"]:::note
joinAtSimilarity -.- joinNoteSimilarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNote5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
collectEdges[1. Collect\nincluded-in edges]
includesLicence{"Something sold\non its own\ncomes inside it?"}
bundle[/Bundle/]:::green
licence[/Licence/]:::green
groupByVendor[2. Group Bundles\nby vendor]
toTierResearcher[/Tier Researcher/]:::next
moduleLicences --> collectEdges --> includesLicence
includesLicence --> |yes| bundle --> groupByVendor --> toTierResearcher
includesLicence --> |no| licence
edgesNote["Module M comes inside X<br/>and M is sold on its own:<br/>X includes a licence"]:::note
includesLicence -.- edgesNote
groupNote["One group per vendor<br/>e.g. Starter, Professional,<br/>Enterprise"]:::note
groupByVendor -.- groupNoteResearch Agents
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNote4. Licence Researcher
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
confirmedModules[/Confirmed Modules/]:::green
researchLoop[Research Loop]:::blue
licenceFacts[/Licence Facts/]:::green
unresolved[/Unresolved,\nretried next run/]:::red
confirmedModules --> |name, description,\ncitations| researchLoop
researchLoop --> |confirmed| licenceFacts
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: is it sold on its own?<br/>Plus one question per licence<br/>it may come inside<br/>Can be both"]:::note
researchLoop -.- checklistNote
factsNote["standalone: yes or no<br/>included in: licence names<br/>each with its quote"]:::note
licenceFacts -.- factsNoteProof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\nclaim, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesModule{"Quote names\nthe module?"}
claimType{"Which claim?"}
namesLicence{"Quote names every\nlicence it comes inside?"}
ownPrice{"Rubric:\nGives it its own\nprice, plan or SKU?"}:::rubric
onlyInside{"Rubric:\nSays it only comes\ninside other licences?"}:::rubric
inPassing{"Rubric:\nOnly listed\nin passing?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Licence Facts/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesModule
namesModule --> |no| backToNextAction
namesModule --> |yes| claimType
claimType --> |sold on its own| ownPrice
claimType --> |comes inside| namesLicence
namesLicence --> |no| backToNextAction
namesLicence --> |yes| onlyInside
ownPrice --> |yes| inPassing
ownPrice --> |no| toReflection
onlyInside --> |yes| inPassing
onlyInside --> |no| toReflection
inPassing --> |no| passes
inPassing --> |yes| toReflection
namesModuleNote["The module's name<br/>or one of its aliases"]:::note
namesModule -.- namesModuleNote
ownPriceNote["e.g. its own row<br/>in a pricing table"]:::note
ownPrice -.- ownPriceNote
onlyInsideNote["e.g. #quot;included in every<br/>Enterprise plan#quot;"]:::note
onlyInside -.- onlyInsideNote
inPassingNote["A feature grid or a<br/>#quot;works with#quot; list"]:::note
inPassing -.- inPassingNoteReflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNote6. Tier Researcher
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
bundleGroup[/Bundle Group\nfrom Bundle Finder/]:::next
researchLoop[Research Loop]:::blue
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
bundleGroup --> |bundle names, the\nmodules each includes| researchLoop
researchLoop --> |confirmed| tierOrder
researchLoop --> |rejected| notNested
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: one question<br/>per pair of bundles:<br/>does A include everything in B?"]:::note
researchLoop -.- checklistNote
tierOrderNote["Each pair with its quote<br/>Lets a user drop to the<br/>lowest tier they need"]:::note
tierOrder -.- tierOrderNoteProof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\npairs, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesBoth{"Quote names\nboth bundles?"}
noCycles{"Order has\nno cycles?"}
statesSuperset{"Rubric:\nSays one has everything\nin the other?"}:::rubric
aboutThesePlans{"Rubric:\nAbout these bundles,\nnot plans in general?"}:::rubric
comparisonTable{"Rubric:\nOnly a comparison\ntable cell?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Tier Order/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesBoth
namesBoth --> |no| backToNextAction
namesBoth --> |yes| noCycles
noCycles --> |no| backToNextAction
noCycles --> |yes| statesSuperset
statesSuperset --> |no| toReflection
statesSuperset --> |yes| aboutThesePlans
aboutThesePlans --> |no| toReflection
aboutThesePlans --> |yes| comparisonTable
comparisonTable --> |yes| toReflection
comparisonTable --> |no| passes
noCyclesNote["A includes B and B includes A<br/>means one answer is wrong"]:::note
noCycles -.- noCyclesNote
supersetNote["e.g. #quot;everything in<br/>Professional, plus#quot;"]:::note
statesSuperset -.- supersetNote
tableNote["Its meaning is in the<br/>table's header, not the cell"]:::note
comparisonTable -.- tableNoteReflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteHow we will measure it
One app with known ground truth, end to end. Salesforce is the likely first choice.
Side by side with the current pipeline: accuracy against the ground truth, pages scraped, and cost.
Component evals: clusters that are already tagged are ground truth for step 1, and saved merges are ground truth for step 2.
How often reflection overturns a draft. If it almost never does, the rubric and reflection steps can be cut to save cost.
Open questions
Planning. The researchers choose one step at a time. Adding a short up-front plan of the questions to answer would make "enough evidence" something code can check.
Embedding new pages. The URL Scraper doesn't embed, so pages scraped during a run aren't searchable in Vendor Documentation Embeddings until something embeds them.
What documentation can't show. Customer-defined roles and usage-metered products don't appear in vendor docs. They need the vendor's own admin or usage APIs.
Shelfware. A licence that is paid for but never used produces no traffic, so nothing triggers research on it. Finding it needs the customer's entitlement data.
Numbers to tune. The research budget (15 tool calls, 10 pages) is a placeholder. The merge thresholds (0.95 and 0.6), the 8 pages per module and the 3 candidates per name come from the PoC.
Cost. Opus drives every research loop, so the budget matters more than it would with Sonnet.
Tool Reference
No title
The agents reach the outside world through two Firecrawl tools. Both check the cache before spending credits, and both fill it afterwards. A third tool, the Checklist Updater, works on the agent's own checklist.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlSearchCache[(Firecrawl Domain\nSearch Cache)]:::yellow
end
request[/query + domains + limit/]
buildKey[1. Build Cache Key]
inCache{"2. In cache?"}
firecrawlSearch[3. Firecrawl Search\nweb results on\nthese domains only]:::blue
gotAnswer{"Answer after\nretries?"}
writeCache[4. Write to Cache]
cleanResults[5. Clean Results]
results[/Pages + metadata/]:::green
failed[/Error, nothing cached/]:::red
request --> buildKey --> inCache
firecrawlSearchCache --> inCache
inCache --> |yes| cleanResults
inCache --> |no| firecrawlSearch --> gotAnswer
gotAnswer --> |yes| writeCache --> cleanResults
gotAnswer --> |no| failed
writeCache --> firecrawlSearchCache
cleanResults --> results
keyNote["search:domains:query:limit<br/>Domains sorted, so order<br/>doesn't change the key"]:::note
buildKey -.- keyNote
refreshNote["refresh: skips the read,<br/>still writes the new answer"]:::note
inCache -.- refreshNote
retryNote["Rate-limited and retried<br/>with backoff"]:::note
firecrawlSearch -.- retryNote
cleanNote["Conservative: only removes<br/>duplicate and off-domain URLs"]:::note
cleanResults -.- cleanNote
resultsNote["url, title, description,<br/>position (search rank),<br/>category"]:::note
results -.- resultsNoteURL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlURLScrapeCache[(Firecrawl URL\nScrape Cache)]:::yellow
end
request[/url/]
inCache{"1. In cache?"}
firecrawlScrape[2. Firecrawl Scrape\nmain content\nas Markdown]:::blue
gotMarkdown{"Markdown after\nretries?"}
writeCache[3. Write Raw\nMarkdown to Cache]
skipped[/Skipped, nothing cached/]:::red
subgraph cleanUp[4. Clean Up]
usablePage{"Dead, empty or\na sign-up form?"}
cleanMarkdown[Strip page chrome]
droppedPage[/Page dropped/]:::red
end
chunkPage[5. Chunk Page]
page[/Page chunks/]:::green
request --> inCache
firecrawlURLScrapeCache --> inCache
inCache --> |yes| usablePage
inCache --> |no| firecrawlScrape --> gotMarkdown
gotMarkdown --> |yes| writeCache --> usablePage
gotMarkdown --> |no| skipped
writeCache --> firecrawlURLScrapeCache
usablePage --> |yes| droppedPage
usablePage --> |no| cleanMarkdown --> chunkPage --> page
scrapeNote["Main content only:<br/>no nav, header, footer"]:::note
firecrawlScrape -.- scrapeNote
rawNote["Raw, not cleaned:<br/>re-clean for free"]:::note
writeCache -.- rawNote
usableNote["Sign-up form: 6+ form-field<br/>lines, no prices"]:::note
usablePage -.- usableNote
cleanNote["Removes: images, cookie<br/>and feedback widgets<br/>Keeps all links, so an agent<br/>can follow them"]:::note
cleanMarkdown -.- cleanNote
chunkNote["800 chars, 150 overlap<br/>Each starts with its heading trail<br/>Under 40 chars dropped"]:::note
chunkPage -.- chunkNote
pageNote["Per chunk: final URL,<br/>chunk index, text, links"]:::note
page -.- pageNoteThree smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
Technology
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
LangGraph nodes
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
ground_truth/
src/
core/ plain TypeScript, no LangChain, unit-tested
checklist/types.ts the checklist object
checklist/ops.ts add_item, set_note, propose_answer, ...
checklist/views.ts the Next Action, Draft and Reflection views
checklist/seeds/module.ts the four seeded items
evidence/ text normalisation and the verbatim quote check
urlRules.ts
budget.ts
tools/ LangChain tool wrappers: schemas and descriptions only
domainSearch.ts
scrapeUrl.ts
docSearch.ts
updateChecklist.ts
cache/firecrawlCache.ts raw Firecrawl answers on disk
llm/models.ts the Bedrock chat model
agents/moduleResearcher/
state.ts
graph.ts
prompts.ts
nodes/ seed, nextAction, callTools, draft, proofOfWork
reflection/graph.ts the reflection subgraph
cli/researchModule.ts
tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
No title
An Innovathon 2026 proposal: a suite of research agents that works out which software licence each employee actually needs, from what they do in the app and from evidence on the vendor's own pages.
It builds on the existing licence usage dashboard PoC. The one-pager is on Confluence.
Why
To know which licence a person needs, you need two facts: which parts of the app they use, and which licence each part requires.
The PoC gets both, but in a fixed order. It first scrapes the vendor's whole documentation site and builds a catalogue of every licence and module, then clusters page-view traffic into the modules people use, and finally joins the two. The traffic half works well. The documentation half is where it struggles:
It has to guess up front which of thousands of pages matter, and most of its tuning goes into that guess.
The modules it finds in the docs are named differently from the ones it finds in traffic ("Salesforce CPQ" against "Quote Management"), so a separate step has to link them.
It can only see what vendors publish. On Datadog it predicted 33 admins where there were 2, because the customer's own roles never appear in vendor documentation.
Ground Truth flips the order. Traffic decides what gets researched. Each module people actually use triggers a small investigation into what that module is and which licence it needs, and every answer has to carry a quote that code can find on the page. Answers without evidence are marked unresolved, not guessed.
How it works
Six steps run in order. Steps 1, 2 and 5 are workflows: they follow fixed steps, and step 5, the Bundle Finder, is plain code with no LLM. Steps 3, 4 and 6 are research agents: they choose their own steps.
Rendering diagram...
1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
The last step, predicting who needs which licence, is the PoC's existing SQL join. It doesn't change.
Rendering diagram...
1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
How to read the diagrams
All diagrams use the same shapes and colours. Every orange step goes through the LLM call path. A rubric check is answered by the model about its own output, so nothing verifies it: a doubtful answer sends the item to reflection.
Rendering diagram...
Rendering diagram...
Design principles
These hold for every agent below.
Evidence first. A claim is kept only with a quote, and code checks that the quote is on a page the agent actually read. This is the proof-of-work step in every agent.
Three layers of checks, cheapest first. Code checks run first and can be trusted. Rubric questions come next: the model answers them about its own output, so they only decide whether an item needs review. Reflection comes last, where a second model call reviews what is still doubtful.
An independent reviewer. Reflection uses Opus, and where Opus also did the research, it reviews with a fresh context: the evidence, not the reasoning that produced it.
Safe defaults. When an agent is unsure, it takes the outcome that costs least if wrong: a name is dropped and its pages are retried on the next run, a new module is suggested instead of a risky merge, or a research question is left unresolved.
Bounded autonomy. The research agents choose their own next step, but within a budget of tool calls and pages, on vendor domains only, and never scraping the same URL twice.
Pay once. Every Firecrawl answer is cached, so a page is never paid for twice, and cleaning rules can change without fetching the page again.
Shared building blocks
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
sendPrompt[Send Prompt]:::orange
networkOk{"Network ok?"}
backOff[Back Off]
shapeOk{"Answer usable?"}
callFailed[/Call Failed/]:::red
toProofOfWork[/Proof of Work/]:::next
sendPrompt --> networkOk
networkOk --> |no, attempt 1-2| backOff --> sendPrompt
networkOk --> |no, attempt 3| callFailed
networkOk --> |yes| shapeOk
shapeOk --> |no| callFailed
shapeOk --> |yes| toProofOfWork
networkNote["Network error, throttle or 5xx<br/>Retried after 1s, then 2s"]:::note
networkOk -.- networkNote
shapeNote["JSON of the expected shape,<br/>not cut off at max_tokens<br/>Not retried"]:::note
shapeOk -.- shapeNote
failedNote["The step's safe outcome:<br/>pages stay untagged, a new module<br/>is suggested, or Unresolved"]:::note
callFailed -.- failedNoteProof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromLlm[/From LLM call:\nanswer + quotes\n+ rubric answers/]:::next
subgraph proofOfWork[Proof of Work]
codeChecks{"Code checks:\nis the evidence real?"}
rubricChecks{"Rubric checks:\ndoes the answer\nhold up?"}:::rubric
end
kept[/Kept/]:::green
failed[/Safe default, or\nback to research\nwith the reason/]:::red
toReflection[/Reflection/]:::next
fromLlm --> codeChecks
codeChecks --> |no| failed
codeChecks --> |yes| rubricChecks
rubricChecks --> |yes| kept
rubricChecks --> |doubtful| toReflection
codeNote["e.g. quote found word for word<br/>on a chunk the model read"]:::note
codeChecks -.- codeNote
rubricNote["e.g. #quot;do the examples<br/>support the name?#quot;"]:::note
rubricChecks -.- rubricNoteReflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged item +\nits evidence/]:::next
review{"Does the evidence\nsupport it?"}:::orange
recheck{"Rename or new answer\npasses proof of work?"}
kept[/Kept/]:::green
safeDefault[/Safe default/]:::red
backToResearch[/Back to research,\nwith what is missing/]:::next
flagged --> review
claudeOpus --> review
review --> |yes| kept
review --> |no| safeDefault
review --> |a better answer| recheck
recheck --> |yes| kept
recheck --> |no| safeDefault
review --> |needs more evidence| backToResearch
recheckNote["Checked once only,<br/>so reflection can't loop"]:::note
recheck -.- recheckNote
backNote["Research agents only<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNoteHow each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
Rendering diagram...
Rendering diagram...
Proof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
Rendering diagram...
Rendering diagram...
Reflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
Rendering diagram...
Rendering diagram...
How each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Workflows
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNote2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
moduleEmbeddings[(Module\nEmbeddings)]:::orange
end
subgraph Tables
modules[(Modules)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
hybridModuleSearch[Hybrid Module Search\nBM25 + vector]
end
suggestions[/Suggested Module Names/]:::green
gatherSuggestion[1. Gather Suggestion\nname + its examples]
toSimilarityCheck[/2. Similarity Check/]:::next
subgraph llmSimilarityCheck[LLM Similarity Check]
comparePrompt[3. Prompt LLM to Compare\nsuggestion vs candidate]:::orange
toProofOfWork[/4. Proof of Work/]:::next
toReflection[/5. Reflection/]:::next
end
joinAtSimilarity[/Join Saved Module/]:::green
newAtSimilarity[/Suggest as New Module/]:::green
joinAtProofOfWork[/Join Saved Module/]:::green
newAtProofOfWork[/Suggest as New Module/]:::green
joinAtReflection[/Join Saved Module/]:::green
newAtReflection[/Suggest as New Module/]:::green
moduleEmbeddings --> hybridModuleSearch
modules --> hybridModuleSearch
suggestions --> gatherSuggestion
hybridModuleSearch --> |top K saved modules:\nname, description,\nexamples| gatherSuggestion
gatherSuggestion --> toSimilarityCheck
toSimilarityCheck --> |match| joinAtSimilarity
toSimilarityCheck --> |no candidate| newAtSimilarity
toSimilarityCheck --> |candidate| comparePrompt
claudeSonnet --> comparePrompt
comparePrompt --> |same module? +\nrubric answers +\ncited URLs| toProofOfWork
toProofOfWork --> |passes| joinAtProofOfWork
toProofOfWork --> |fails| newAtProofOfWork
toProofOfWork --> |flagged| toReflection
toReflection --> |yes| joinAtReflection
toReflection --> |no, or no answer| newAtReflection
searchNote["Vector: name embedding similarity<br/>BM25: name, merged names, description<br/>Scores pooled, top K kept"]:::note
hybridModuleSearch -.- searchNote
compareNote["Sees the saved module's<br/>name and description<br/>+ 8 pages of each<br/>At most 3 candidates per name"]:::note
comparePrompt -.- compareNote
joinNote["Suggested name saved in<br/>merged_names: next time<br/>it matches at step 2"]:::note
joinAtSimilarity -.- joinNoteSimilarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNote5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
collectEdges[1. Collect\nincluded-in edges]
includesLicence{"Something sold\non its own\ncomes inside it?"}
bundle[/Bundle/]:::green
licence[/Licence/]:::green
groupByVendor[2. Group Bundles\nby vendor]
toTierResearcher[/Tier Researcher/]:::next
moduleLicences --> collectEdges --> includesLicence
includesLicence --> |yes| bundle --> groupByVendor --> toTierResearcher
includesLicence --> |no| licence
edgesNote["Module M comes inside X<br/>and M is sold on its own:<br/>X includes a licence"]:::note
includesLicence -.- edgesNote
groupNote["One group per vendor<br/>e.g. Starter, Professional,<br/>Enterprise"]:::note
groupByVendor -.- groupNoteNo title
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
No title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNoteNo title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNoteNo title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
Rendering diagram...
Rendering diagram...
Research Agents
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNote4. Licence Researcher
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
confirmedModules[/Confirmed Modules/]:::green
researchLoop[Research Loop]:::blue
licenceFacts[/Licence Facts/]:::green
unresolved[/Unresolved,\nretried next run/]:::red
confirmedModules --> |name, description,\ncitations| researchLoop
researchLoop --> |confirmed| licenceFacts
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: is it sold on its own?<br/>Plus one question per licence<br/>it may come inside<br/>Can be both"]:::note
researchLoop -.- checklistNote
factsNote["standalone: yes or no<br/>included in: licence names<br/>each with its quote"]:::note
licenceFacts -.- factsNoteProof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\nclaim, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesModule{"Quote names\nthe module?"}
claimType{"Which claim?"}
namesLicence{"Quote names every\nlicence it comes inside?"}
ownPrice{"Rubric:\nGives it its own\nprice, plan or SKU?"}:::rubric
onlyInside{"Rubric:\nSays it only comes\ninside other licences?"}:::rubric
inPassing{"Rubric:\nOnly listed\nin passing?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Licence Facts/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesModule
namesModule --> |no| backToNextAction
namesModule --> |yes| claimType
claimType --> |sold on its own| ownPrice
claimType --> |comes inside| namesLicence
namesLicence --> |no| backToNextAction
namesLicence --> |yes| onlyInside
ownPrice --> |yes| inPassing
ownPrice --> |no| toReflection
onlyInside --> |yes| inPassing
onlyInside --> |no| toReflection
inPassing --> |no| passes
inPassing --> |yes| toReflection
namesModuleNote["The module's name<br/>or one of its aliases"]:::note
namesModule -.- namesModuleNote
ownPriceNote["e.g. its own row<br/>in a pricing table"]:::note
ownPrice -.- ownPriceNote
onlyInsideNote["e.g. #quot;included in every<br/>Enterprise plan#quot;"]:::note
onlyInside -.- onlyInsideNote
inPassingNote["A feature grid or a<br/>#quot;works with#quot; list"]:::note
inPassing -.- inPassingNoteReflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNote6. Tier Researcher
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
bundleGroup[/Bundle Group\nfrom Bundle Finder/]:::next
researchLoop[Research Loop]:::blue
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
bundleGroup --> |bundle names, the\nmodules each includes| researchLoop
researchLoop --> |confirmed| tierOrder
researchLoop --> |rejected| notNested
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: one question<br/>per pair of bundles:<br/>does A include everything in B?"]:::note
researchLoop -.- checklistNote
tierOrderNote["Each pair with its quote<br/>Lets a user drop to the<br/>lowest tier they need"]:::note
tierOrder -.- tierOrderNoteProof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\npairs, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesBoth{"Quote names\nboth bundles?"}
noCycles{"Order has\nno cycles?"}
statesSuperset{"Rubric:\nSays one has everything\nin the other?"}:::rubric
aboutThesePlans{"Rubric:\nAbout these bundles,\nnot plans in general?"}:::rubric
comparisonTable{"Rubric:\nOnly a comparison\ntable cell?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Tier Order/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesBoth
namesBoth --> |no| backToNextAction
namesBoth --> |yes| noCycles
noCycles --> |no| backToNextAction
noCycles --> |yes| statesSuperset
statesSuperset --> |no| toReflection
statesSuperset --> |yes| aboutThesePlans
aboutThesePlans --> |no| toReflection
aboutThesePlans --> |yes| comparisonTable
comparisonTable --> |yes| toReflection
comparisonTable --> |no| passes
noCyclesNote["A includes B and B includes A<br/>means one answer is wrong"]:::note
noCycles -.- noCyclesNote
supersetNote["e.g. #quot;everything in<br/>Professional, plus#quot;"]:::note
statesSuperset -.- supersetNote
tableNote["Its meaning is in the<br/>table's header, not the cell"]:::note
comparisonTable -.- tableNoteReflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNoteNo title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
4. Licence Researcher
No title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNoteNo title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
6. Tier Researcher
No title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
How we will measure it
One app with known ground truth, end to end. Salesforce is the likely first choice.
Side by side with the current pipeline: accuracy against the ground truth, pages scraped, and cost.
Component evals: clusters that are already tagged are ground truth for step 1, and saved merges are ground truth for step 2.
How often reflection overturns a draft. If it almost never does, the rubric and reflection steps can be cut to save cost.
Open questions
Planning. The researchers choose one step at a time. Adding a short up-front plan of the questions to answer would make "enough evidence" something code can check.
Embedding new pages. The URL Scraper doesn't embed, so pages scraped during a run aren't searchable in Vendor Documentation Embeddings until something embeds them.
What documentation can't show. Customer-defined roles and usage-metered products don't appear in vendor docs. They need the vendor's own admin or usage APIs.
Shelfware. A licence that is paid for but never used produces no traffic, so nothing triggers research on it. Finding it needs the customer's entitlement data.
Numbers to tune. The research budget (15 tool calls, 10 pages) is a placeholder. The merge thresholds (0.95 and 0.6), the 8 pages per module and the 3 candidates per name come from the PoC.
Cost. Opus drives every research loop, so the budget matters more than it would with Sonnet.
Tool Reference
No title
The agents reach the outside world through two Firecrawl tools. Both check the cache before spending credits, and both fill it afterwards. A third tool, the Checklist Updater, works on the agent's own checklist.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlURLScrapeCache[(Firecrawl URL\nScrape Cache)]:::yellow
end
request[/url/]
inCache{"1. In cache?"}
firecrawlScrape[2. Firecrawl Scrape\nmain content\nas Markdown]:::blue
gotMarkdown{"Markdown after\nretries?"}
writeCache[3. Write Raw\nMarkdown to Cache]
skipped[/Skipped, nothing cached/]:::red
subgraph cleanUp[4. Clean Up]
usablePage{"Dead, empty or\na sign-up form?"}
cleanMarkdown[Strip page chrome]
droppedPage[/Page dropped/]:::red
end
chunkPage[5. Chunk Page]
page[/Page chunks/]:::green
request --> inCache
firecrawlURLScrapeCache --> inCache
inCache --> |yes| usablePage
inCache --> |no| firecrawlScrape --> gotMarkdown
gotMarkdown --> |yes| writeCache --> usablePage
gotMarkdown --> |no| skipped
writeCache --> firecrawlURLScrapeCache
usablePage --> |yes| droppedPage
usablePage --> |no| cleanMarkdown --> chunkPage --> page
scrapeNote["Main content only:<br/>no nav, header, footer"]:::note
firecrawlScrape -.- scrapeNote
rawNote["Raw, not cleaned:<br/>re-clean for free"]:::note
writeCache -.- rawNote
usableNote["Sign-up form: 6+ form-field<br/>lines, no prices"]:::note
usablePage -.- usableNote
cleanNote["Removes: images, cookie<br/>and feedback widgets<br/>Keeps all links, so an agent<br/>can follow them"]:::note
cleanMarkdown -.- cleanNote
chunkNote["800 chars, 150 overlap<br/>Each starts with its heading trail<br/>Under 40 chars dropped"]:::note
chunkPage -.- chunkNote
pageNote["Per chunk: final URL,<br/>chunk index, text, links"]:::note
page -.- pageNoteThree smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
Rendering diagram...
Three smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Rendering diagram...
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
Technology
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
LangGraph nodes
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
ground_truth/
src/
core/ plain TypeScript, no LangChain, unit-tested
checklist/types.ts the checklist object
checklist/ops.ts add_item, set_note, propose_answer, ...
checklist/views.ts the Next Action, Draft and Reflection views
checklist/seeds/module.ts the four seeded items
evidence/ text normalisation and the verbatim quote check
urlRules.ts
budget.ts
tools/ LangChain tool wrappers: schemas and descriptions only
domainSearch.ts
scrapeUrl.ts
docSearch.ts
updateChecklist.ts
cache/firecrawlCache.ts raw Firecrawl answers on disk
llm/models.ts the Bedrock chat model
agents/moduleResearcher/
state.ts
graph.ts
prompts.ts
nodes/ seed, nextAction, callTools, draft, proofOfWork
reflection/graph.ts the reflection subgraph
cli/researchModule.ts
tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
No title
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
LangGraph nodes
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
No title
An Innovathon 2026 proposal: a suite of research agents that works out which software licence each employee actually needs, from what they do in the app and from evidence on the vendor's own pages.
It builds on the existing licence usage dashboard PoC. The one-pager is on Confluence.
Why
To know which licence a person needs, you need two facts: which parts of the app they use, and which licence each part requires.
The PoC gets both, but in a fixed order. It first scrapes the vendor's whole documentation site and builds a catalogue of every licence and module, then clusters page-view traffic into the modules people use, and finally joins the two. The traffic half works well. The documentation half is where it struggles:
It has to guess up front which of thousands of pages matter, and most of its tuning goes into that guess.
The modules it finds in the docs are named differently from the ones it finds in traffic ("Salesforce CPQ" against "Quote Management"), so a separate step has to link them.
It can only see what vendors publish. On Datadog it predicted 33 admins where there were 2, because the customer's own roles never appear in vendor documentation.
Ground Truth flips the order. Traffic decides what gets researched. Each module people actually use triggers a small investigation into what that module is and which licence it needs, and every answer has to carry a quote that code can find on the page. Answers without evidence are marked unresolved, not guessed.
How it works
Six steps run in order. Steps 1, 2 and 5 are workflows: they follow fixed steps, and step 5, the Bundle Finder, is plain code with no LLM. Steps 3, 4 and 6 are research agents: they choose their own steps.
Rendering diagram...
1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
The last step, predicting who needs which licence, is the PoC's existing SQL join. It doesn't change.
Rendering diagram...
1. Suggest Module Names
An untagged traffic cluster
What would you call this area of the app?
Suggested module names
Sonnet drafts, Opus reviews
2. Merge Module Names
A suggested name and the closest saved modules
Is this a module we already have?
Join a saved module, or suggest a new one
Sonnet compares, Opus reviews
3. Module Researcher
A new module name
Is it really a module in the vendor's product, and what is it?
A confirmed module: title, description, citations
Opus
4. Licence Researcher
A confirmed module
Is it sold on its own, and which licences does it come inside?
Licence facts, each with a quote
Opus
5. Bundle Finder
Licence facts
Which entries are bundles?
Bundle groups, one per vendor
Code
6. Tier Researcher
A bundle group
Which bundle includes everything in which?
Tier order
Opus
How to read the diagrams
All diagrams use the same shapes and colours. Every orange step goes through the LLM call path. A rubric check is answered by the model about its own output, so nothing verifies it: a doubtful answer sends the item to reflection.
Rendering diagram...
Rendering diagram...
Design principles
These hold for every agent below.
Evidence first. A claim is kept only with a quote, and code checks that the quote is on a page the agent actually read. This is the proof-of-work step in every agent.
Three layers of checks, cheapest first. Code checks run first and can be trusted. Rubric questions come next: the model answers them about its own output, so they only decide whether an item needs review. Reflection comes last, where a second model call reviews what is still doubtful.
An independent reviewer. Reflection uses Opus, and where Opus also did the research, it reviews with a fresh context: the evidence, not the reasoning that produced it.
Safe defaults. When an agent is unsure, it takes the outcome that costs least if wrong: a name is dropped and its pages are retried on the next run, a new module is suggested instead of a risky merge, or a research question is left unresolved.
Bounded autonomy. The research agents choose their own next step, but within a budget of tool calls and pages, on vendor domains only, and never scraping the same URL twice.
Pay once. Every Firecrawl answer is cached, so a page is never paid for twice, and cleaning rules can change without fetching the page again.
Shared building blocks
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
sendPrompt[Send Prompt]:::orange
networkOk{"Network ok?"}
backOff[Back Off]
shapeOk{"Answer usable?"}
callFailed[/Call Failed/]:::red
toProofOfWork[/Proof of Work/]:::next
sendPrompt --> networkOk
networkOk --> |no, attempt 1-2| backOff --> sendPrompt
networkOk --> |no, attempt 3| callFailed
networkOk --> |yes| shapeOk
shapeOk --> |no| callFailed
shapeOk --> |yes| toProofOfWork
networkNote["Network error, throttle or 5xx<br/>Retried after 1s, then 2s"]:::note
networkOk -.- networkNote
shapeNote["JSON of the expected shape,<br/>not cut off at max_tokens<br/>Not retried"]:::note
shapeOk -.- shapeNote
failedNote["The step's safe outcome:<br/>pages stay untagged, a new module<br/>is suggested, or Unresolved"]:::note
callFailed -.- failedNoteProof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromLlm[/From LLM call:\nanswer + quotes\n+ rubric answers/]:::next
subgraph proofOfWork[Proof of Work]
codeChecks{"Code checks:\nis the evidence real?"}
rubricChecks{"Rubric checks:\ndoes the answer\nhold up?"}:::rubric
end
kept[/Kept/]:::green
failed[/Safe default, or\nback to research\nwith the reason/]:::red
toReflection[/Reflection/]:::next
fromLlm --> codeChecks
codeChecks --> |no| failed
codeChecks --> |yes| rubricChecks
rubricChecks --> |yes| kept
rubricChecks --> |doubtful| toReflection
codeNote["e.g. quote found word for word<br/>on a chunk the model read"]:::note
codeChecks -.- codeNote
rubricNote["e.g. #quot;do the examples<br/>support the name?#quot;"]:::note
rubricChecks -.- rubricNoteReflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged item +\nits evidence/]:::next
review{"Does the evidence\nsupport it?"}:::orange
recheck{"Rename or new answer\npasses proof of work?"}
kept[/Kept/]:::green
safeDefault[/Safe default/]:::red
backToResearch[/Back to research,\nwith what is missing/]:::next
flagged --> review
claudeOpus --> review
review --> |yes| kept
review --> |no| safeDefault
review --> |a better answer| recheck
recheck --> |yes| kept
recheck --> |no| safeDefault
review --> |needs more evidence| backToResearch
recheckNote["Checked once only,<br/>so reflection can't loop"]:::note
recheck -.- recheckNote
backNote["Research agents only<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNoteHow each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
Rendering diagram...
Rendering diagram...
Proof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
Rendering diagram...
Rendering diagram...
Reflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
Rendering diagram...
Rendering diagram...
How each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
Workflows
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNote2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
moduleEmbeddings[(Module\nEmbeddings)]:::orange
end
subgraph Tables
modules[(Modules)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
hybridModuleSearch[Hybrid Module Search\nBM25 + vector]
end
suggestions[/Suggested Module Names/]:::green
gatherSuggestion[1. Gather Suggestion\nname + its examples]
toSimilarityCheck[/2. Similarity Check/]:::next
subgraph llmSimilarityCheck[LLM Similarity Check]
comparePrompt[3. Prompt LLM to Compare\nsuggestion vs candidate]:::orange
toProofOfWork[/4. Proof of Work/]:::next
toReflection[/5. Reflection/]:::next
end
joinAtSimilarity[/Join Saved Module/]:::green
newAtSimilarity[/Suggest as New Module/]:::green
joinAtProofOfWork[/Join Saved Module/]:::green
newAtProofOfWork[/Suggest as New Module/]:::green
joinAtReflection[/Join Saved Module/]:::green
newAtReflection[/Suggest as New Module/]:::green
moduleEmbeddings --> hybridModuleSearch
modules --> hybridModuleSearch
suggestions --> gatherSuggestion
hybridModuleSearch --> |top K saved modules:\nname, description,\nexamples| gatherSuggestion
gatherSuggestion --> toSimilarityCheck
toSimilarityCheck --> |match| joinAtSimilarity
toSimilarityCheck --> |no candidate| newAtSimilarity
toSimilarityCheck --> |candidate| comparePrompt
claudeSonnet --> comparePrompt
comparePrompt --> |same module? +\nrubric answers +\ncited URLs| toProofOfWork
toProofOfWork --> |passes| joinAtProofOfWork
toProofOfWork --> |fails| newAtProofOfWork
toProofOfWork --> |flagged| toReflection
toReflection --> |yes| joinAtReflection
toReflection --> |no, or no answer| newAtReflection
searchNote["Vector: name embedding similarity<br/>BM25: name, merged names, description<br/>Scores pooled, top K kept"]:::note
hybridModuleSearch -.- searchNote
compareNote["Sees the saved module's<br/>name and description<br/>+ 8 pages of each<br/>At most 3 candidates per name"]:::note
comparePrompt -.- compareNote
joinNote["Suggested name saved in<br/>merged_names: next time<br/>it matches at step 2"]:::note
joinAtSimilarity -.- joinNoteSimilarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNote5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
flowchart LR
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
collectEdges[1. Collect\nincluded-in edges]
includesLicence{"Something sold\non its own\ncomes inside it?"}
bundle[/Bundle/]:::green
licence[/Licence/]:::green
groupByVendor[2. Group Bundles\nby vendor]
toTierResearcher[/Tier Researcher/]:::next
moduleLicences --> collectEdges --> includesLicence
includesLicence --> |yes| bundle --> groupByVendor --> toTierResearcher
includesLicence --> |no| licence
edgesNote["Module M comes inside X<br/>and M is sold on its own:<br/>X includes a licence"]:::note
includesLicence -.- edgesNote
groupNote["One group per vendor<br/>e.g. Starter, Professional,<br/>Enterprise"]:::note
groupByVendor -.- groupNoteNo title
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
No title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNoteNo title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNoteNo title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
Rendering diagram...
Rendering diagram...
Research Agents
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNote4. Licence Researcher
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
confirmedModules[/Confirmed Modules/]:::green
researchLoop[Research Loop]:::blue
licenceFacts[/Licence Facts/]:::green
unresolved[/Unresolved,\nretried next run/]:::red
confirmedModules --> |name, description,\ncitations| researchLoop
researchLoop --> |confirmed| licenceFacts
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: is it sold on its own?<br/>Plus one question per licence<br/>it may come inside<br/>Can be both"]:::note
researchLoop -.- checklistNote
factsNote["standalone: yes or no<br/>included in: licence names<br/>each with its quote"]:::note
licenceFacts -.- factsNoteProof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\nclaim, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesModule{"Quote names\nthe module?"}
claimType{"Which claim?"}
namesLicence{"Quote names every\nlicence it comes inside?"}
ownPrice{"Rubric:\nGives it its own\nprice, plan or SKU?"}:::rubric
onlyInside{"Rubric:\nSays it only comes\ninside other licences?"}:::rubric
inPassing{"Rubric:\nOnly listed\nin passing?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Licence Facts/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesModule
namesModule --> |no| backToNextAction
namesModule --> |yes| claimType
claimType --> |sold on its own| ownPrice
claimType --> |comes inside| namesLicence
namesLicence --> |no| backToNextAction
namesLicence --> |yes| onlyInside
ownPrice --> |yes| inPassing
ownPrice --> |no| toReflection
onlyInside --> |yes| inPassing
onlyInside --> |no| toReflection
inPassing --> |no| passes
inPassing --> |yes| toReflection
namesModuleNote["The module's name<br/>or one of its aliases"]:::note
namesModule -.- namesModuleNote
ownPriceNote["e.g. its own row<br/>in a pricing table"]:::note
ownPrice -.- ownPriceNote
onlyInsideNote["e.g. #quot;included in every<br/>Enterprise plan#quot;"]:::note
onlyInside -.- onlyInsideNote
inPassingNote["A feature grid or a<br/>#quot;works with#quot; list"]:::note
inPassing -.- inPassingNoteReflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNote6. Tier Researcher
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
bundleGroup[/Bundle Group\nfrom Bundle Finder/]:::next
researchLoop[Research Loop]:::blue
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
bundleGroup --> |bundle names, the\nmodules each includes| researchLoop
researchLoop --> |confirmed| tierOrder
researchLoop --> |rejected| notNested
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: one question<br/>per pair of bundles:<br/>does A include everything in B?"]:::note
researchLoop -.- checklistNote
tierOrderNote["Each pair with its quote<br/>Lets a user drop to the<br/>lowest tier they need"]:::note
tierOrder -.- tierOrderNoteProof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\npairs, quotes,\ncited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
namesBoth{"Quote names\nboth bundles?"}
noCycles{"Order has\nno cycles?"}
statesSuperset{"Rubric:\nSays one has everything\nin the other?"}:::rubric
aboutThesePlans{"Rubric:\nAbout these bundles,\nnot plans in general?"}:::rubric
comparisonTable{"Rubric:\nOnly a comparison\ntable cell?"}:::rubric
end
backToNextAction[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
passes[/Tier Order/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToNextAction
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToNextAction
urlsScraped --> |yes| namesBoth
namesBoth --> |no| backToNextAction
namesBoth --> |yes| noCycles
noCycles --> |no| backToNextAction
noCycles --> |yes| statesSuperset
statesSuperset --> |no| toReflection
statesSuperset --> |yes| aboutThesePlans
aboutThesePlans --> |no| toReflection
aboutThesePlans --> |yes| comparisonTable
comparisonTable --> |yes| toReflection
comparisonTable --> |no| passes
noCyclesNote["A includes B and B includes A<br/>means one answer is wrong"]:::note
noCycles -.- noCyclesNote
supersetNote["e.g. #quot;everything in<br/>Professional, plus#quot;"]:::note
statesSuperset -.- supersetNote
tableNote["Its meaning is in the<br/>table's header, not the cell"]:::note
comparisonTable -.- tableNoteReflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNoteNo title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
4. Licence Researcher
No title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNoteNo title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
6. Tier Researcher
No title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
How we will measure it
One app with known ground truth, end to end. Salesforce is the likely first choice.
Side by side with the current pipeline: accuracy against the ground truth, pages scraped, and cost.
Component evals: clusters that are already tagged are ground truth for step 1, and saved merges are ground truth for step 2.
How often reflection overturns a draft. If it almost never does, the rubric and reflection steps can be cut to save cost.
Open questions
Planning. The researchers choose one step at a time. Adding a short up-front plan of the questions to answer would make "enough evidence" something code can check.
Embedding new pages. The URL Scraper doesn't embed, so pages scraped during a run aren't searchable in Vendor Documentation Embeddings until something embeds them.
What documentation can't show. Customer-defined roles and usage-metered products don't appear in vendor docs. They need the vendor's own admin or usage APIs.
Shelfware. A licence that is paid for but never used produces no traffic, so nothing triggers research on it. Finding it needs the customer's entitlement data.
Numbers to tune. The research budget (15 tool calls, 10 pages) is a placeholder. The merge thresholds (0.95 and 0.6), the 8 pages per module and the 3 candidates per name come from the PoC.
Cost. Opus drives every research loop, so the budget matters more than it would with Sonnet.
Tool Reference
No title
The agents reach the outside world through two Firecrawl tools. Both check the cache before spending credits, and both fill it afterwards. A third tool, the Checklist Updater, works on the agent's own checklist.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
subgraph Cache
firecrawlURLScrapeCache[(Firecrawl URL\nScrape Cache)]:::yellow
end
request[/url/]
inCache{"1. In cache?"}
firecrawlScrape[2. Firecrawl Scrape\nmain content\nas Markdown]:::blue
gotMarkdown{"Markdown after\nretries?"}
writeCache[3. Write Raw\nMarkdown to Cache]
skipped[/Skipped, nothing cached/]:::red
subgraph cleanUp[4. Clean Up]
usablePage{"Dead, empty or\na sign-up form?"}
cleanMarkdown[Strip page chrome]
droppedPage[/Page dropped/]:::red
end
chunkPage[5. Chunk Page]
page[/Page chunks/]:::green
request --> inCache
firecrawlURLScrapeCache --> inCache
inCache --> |yes| usablePage
inCache --> |no| firecrawlScrape --> gotMarkdown
gotMarkdown --> |yes| writeCache --> usablePage
gotMarkdown --> |no| skipped
writeCache --> firecrawlURLScrapeCache
usablePage --> |yes| droppedPage
usablePage --> |no| cleanMarkdown --> chunkPage --> page
scrapeNote["Main content only:<br/>no nav, header, footer"]:::note
firecrawlScrape -.- scrapeNote
rawNote["Raw, not cleaned:<br/>re-clean for free"]:::note
writeCache -.- rawNote
usableNote["Sign-up form: 6+ form-field<br/>lines, no prices"]:::note
usablePage -.- usableNote
cleanNote["Removes: images, cookie<br/>and feedback widgets<br/>Keeps all links, so an agent<br/>can follow them"]:::note
cleanMarkdown -.- cleanNote
chunkNote["800 chars, 150 overlap<br/>Each starts with its heading trail<br/>Under 40 chars dropped"]:::note
chunkPage -.- chunkNote
pageNote["Per chunk: final URL,<br/>chunk index, text, links"]:::note
page -.- pageNoteThree smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
Rendering diagram...
Three smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Rendering diagram...
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
Technology
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
LangGraph nodes
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
ground_truth/
src/
core/ plain TypeScript, no LangChain, unit-tested
checklist/types.ts the checklist object
checklist/ops.ts add_item, set_note, propose_answer, ...
checklist/views.ts the Next Action, Draft and Reflection views
checklist/seeds/module.ts the four seeded items
evidence/ text normalisation and the verbatim quote check
urlRules.ts
budget.ts
tools/ LangChain tool wrappers: schemas and descriptions only
domainSearch.ts
scrapeUrl.ts
docSearch.ts
updateChecklist.ts
cache/firecrawlCache.ts raw Firecrawl answers on disk
llm/models.ts the Bedrock chat model
agents/moduleResearcher/
state.ts
graph.ts
prompts.ts
nodes/ seed, nextAction, callTools, draft, proofOfWork
reflection/graph.ts the reflection subgraph
cli/researchModule.ts
tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
No title
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
LangGraph nodes
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
Rendering diagram...
Rendering diagram...
Proof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
Rendering diagram...
Rendering diagram...
Reflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
Rendering diagram...
Rendering diagram...
How each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
No title
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
No title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNoteNo title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNoteNo title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
Rendering diagram...
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
No title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
No title
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNoteNo title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
4. Licence Researcher
No title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNoteNo title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
6. Tier Researcher
No title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}No title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
Rendering diagram...
Three smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Rendering diagram...
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
No title
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
LangGraph nodes
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
Stores
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
Page Visit Embeddings
One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.
Vendor Documentation Embeddings
Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.
Modules and Module Embeddings
Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).
Module Licences
For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.
Bundle Tiers
Pairs of bundles where one includes everything in the other, each with its quote.
Firecrawl Domain Search Cache and URL Scrape Cache
Firecrawl's raw answers on disk, keyed by the request.
LLM Calls
Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.
Rendering diagram...
Rendering diagram...
Proof of Work
Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.
Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.
Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.
Rendering diagram...
Rendering diagram...
Reflection
A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.
Rendering diagram...
Rendering diagram...
How each agent uses them
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
1. Suggest Module Names
No ids or digits in the name; not a generic word
Do the examples support the name?
Keep, drop, or rename
2. Merge Module Names
Cited URLs are from the pages sent; one name contains the other
Do users of both do the same task? Do they work on different records?
Join, or suggest as new
3. Module Researcher
Quotes are on the cited chunks; cited URLs were scraped this run
Do the quotes describe something users work in?
Confirm, reject, or ask for more evidence
4. Licence Researcher
The same, plus: the quote names the module, and every licence it comes inside
Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?
Confirm, or ask for more evidence
6. Tier Researcher
The same, plus: the quote names both bundles; the order has no cycles
Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?
Confirm, say not nested, or ask for more evidence
No title
These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.
1. Suggest Module Names
No title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
end
claudeSonnet((Claude Sonnet))
subgraph Tools
retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
end
gatherClusterInfo[1. Gather Cluster Info]
moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
toProofOfWork[/3. Proof of Work/]:::next
toReflection[/4. Reflection/]:::next
suggestions[/Suggested Module Names/]:::green
dropSuggestion[/Drop Suggestion/]:::red
pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
gatherClusterInfo --> moduleNamePrompt
claudeSonnet --> moduleNamePrompt
moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
toProofOfWork --> |passes| suggestions
toProofOfWork --> |id or digits| dropSuggestion
toProofOfWork --> |flagged| toReflection
toReflection --> |keep, or rename passes| suggestions
toReflection --> |drop| dropSuggestion
gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
gatherClusterInfo -.- gatherNote
promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
moduleNamePrompt -.- promptNote
dropNote["Pages stay untagged,<br/>retried next run"]:::note
dropSuggestion -.- dropNoteProof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next
subgraph proofOfWorkCheck[3. Proof of Work]
hasIdOrDigits{"Name has\n<id> or digits?"}
isGenericName{"Name on the\ngeneric word list?"}
isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
dropAtProofOfWork[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
toReflection[/Reflection/]:::next
fromPrompt --> hasIdOrDigits
hasIdOrDigits --> |yes| dropAtProofOfWork
hasIdOrDigits --> |no| isGenericName
isGenericName --> |yes| toReflection
isGenericName --> |no| isSupported
isSupported --> |yes| keepSuggestion
isSupported --> |no| toReflection
idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with <id>"]:::note
hasIdOrDigits -.- idOrDigitsNote
genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
isGenericName -.- genericNameNote
rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
isSupported -.- rubricNoteReflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nflagged suggestion/]:::next
subgraph reflection[4. Reflection]
conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
subgraph reflectionProofOfWork[4.1 Proof of Work]
renameHasIdOrDigits{"Rename has\n<id> or digits?"}
renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
end
dropAtReflection[Drop Suggestion]:::red
end
keepSuggestion[/Keep Suggestion/]:::green
flagged --> conferKeepOrDrop
claudeOpus --> conferKeepOrDrop
conferKeepOrDrop --> |keep| keepSuggestion
conferKeepOrDrop --> |drop| dropAtReflection
conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
renameHasIdOrDigits --> |yes| dropAtReflection
renameHasIdOrDigits --> |no| renameIsSupported
renameIsSupported --> |yes| keepSuggestion
renameIsSupported --> |no| dropAtReflection
reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
conferKeepOrDrop -.- reflectionNoteNo title
Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.
In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
2. Merge Module Names
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next
subgraph similarityCheck[2. Similarity Check]
sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toCompare[/3. Prompt LLM to Compare/]:::next
fromGather --> sameName
sameName --> |yes| joinModule
sameName --> |no| isCandidate
isCandidate --> |no| newModule
isCandidate --> |yes| toCompare
sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
sameName -.- sameNameNote
candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
isCandidate -.- candidateNoteProof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next
subgraph proofOfWorkCheck[4. Proof of Work]
citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
differentObjects{"Rubric:\nDifferent objects?"}:::rubric
nameContains{"One name\ncontains the other?"}
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
toReflection[/Reflection/]:::next
fromCompare --> citedUrlsSent
citedUrlsSent --> |no| newModule
citedUrlsSent --> |yes| sameJob
sameJob --> |no| newModule
sameJob --> |yes| differentObjects
differentObjects --> |yes| newModule
differentObjects --> |no| nameContains
nameContains --> |yes| joinModule
nameContains --> |no| toReflection
citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
citedUrlsSent -.- citedUrlsNote
sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
sameJob -.- sameJobNote
differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
differentObjects -.- differentObjectsNote
nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
nameContains -.- nameContainsNoteReflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
flagged[/From Proof of Work:\nnames don't bear it out/]:::next
subgraph reflection[5. Reflection]
confirmSameModule{"Same module?"}:::orange
end
joinModule[/Join Saved Module/]:::green
newModule[/Suggest as New Module/]:::green
flagged --> confirmSameModule
claudeOpus --> confirmSameModule
confirmSameModule --> |yes| joinModule
confirmSameModule --> |no, or no answer| newModule
reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
confirmSameModule -.- reflectionNoteNo title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
5. Bundle Finder
A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.
Rendering diagram...
Rendering diagram...
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
No title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
No title
These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.
The research loop
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
claudeOpus((Claude Opus))
input[/Input/]
seed[1. Seed Checklist]
subgraph loop[Loop]
checklistDone{"2. Checklist?"}
budgetLeft{"3. Budget left?"}
nextAction["4. Next Action\npick the next action"]:::orange
callTool[5. Call Tool]
draft["6. Describe or Decide\nfrom the closed items"]:::orange
toProofOfWork[/7. Proof of Work/]:::next
toReflection[/8. Reflection/]:::next
end
confirmed[/Confirmed/]:::green
rejected[/Rejected/]:::red
unresolved[/Unresolved,\nopen items saved/]:::red
input --> seed --> checklistDone
checklistDone --> |items still open| budgetLeft
checklistDone --> |every item closed| draft
checklistDone --> |closed negative| rejected
budgetLeft --> |yes| nextAction
budgetLeft --> |no| unresolved
claudeOpus --> nextAction
nextAction --> |search, scrape or\nupdate the checklist| callTool
callTool --> |updated checklist\n+ evidence| checklistDone
draft --> toProofOfWork
toProofOfWork --> |passes| confirmed
toProofOfWork --> |fails: reopen items| checklistDone
toProofOfWork --> |flagged| toReflection
toReflection --> |confirm| confirmed
toReflection --> |reject| rejected
toReflection --> |needs more:\nreopen items| checklistDone
seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
seed -.- seedNote
checklistNote["Code, not the model:<br/>counts the open items"]:::note
checklistDone -.- checklistNote
budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
budgetLeft -.- budgetNote
nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
nextAction -.- nextNote
callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
callTool -.- callNote
rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
rejected -.- rejectedNoteCall Tool
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Vectors
vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
end
subgraph Tools
domainSearcher[Domain Searcher]
urlScraper[URL Scraper]
vendorDocSearcher[Vendor Documents\nSearcher]
end
fromNextAction[/From Next Action:\naction + arguments/]:::next
countCall[1. Count the Call\nagainst the budget]
whichTool{"2. Which tool?"}
urlAllowed{"URL allowed?"}
toolAnswered{"3. Tool answered?"}
applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
logCall[4. Update Agent State\ntrail, chunks, checklist]
toReadResults[/Research Loop:\n2. Checklist?/]:::next
backToNextAction[/Next Action,\nwith the error/]:::next
fromNextAction --> |search or scrape| countCall --> whichTool
fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
whichTool --> |search the site| domainSearcher
whichTool --> |search stored docs| vendorDocSearcher
whichTool --> |scrape or\nfollow a link| urlAllowed
urlAllowed --> |yes| urlScraper
urlAllowed --> |no| backToNextAction
vendorDocEmbeddings --> vendorDocSearcher
domainSearcher --> toolAnswered
vendorDocSearcher --> toolAnswered
urlScraper --> toolAnswered
toolAnswered --> |yes| logCall --> toReadResults
toolAnswered --> |no| backToNextAction
countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
countCall -.- countNote
applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
applyOps -.- applyNote
toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
urlScraper -.- toolsNote
urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
urlAllowed -.- urlNote
logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
logCall -.- logNoteState
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
3. Module Researcher
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
suggestions[/Suggested Module Names/]:::green
researchLoop[Research Loop]:::blue
confirmedModule[/Confirmed Module/]:::green
notModule[/Not a Module/]:::red
unresolved[/Unresolved,\nretried next run/]:::red
suggestions --> |name + examples| researchLoop
researchLoop --> |confirmed| confirmedModule
researchLoop --> |rejected| notModule
researchLoop --> |unresolved| unresolved
checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
researchLoop -.- checklistNote
moduleNote["title, description,<br/>citations, merged_names"]:::note
confirmedModule -.- moduleNoteChecklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next
subgraph proofOfWorkCheck[5. Proof of Work]
quotesOnChunks{"Quotes are on\nthe cited chunks?"}
urlsScraped{"Cited URLs were\nscraped this run?"}
usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
end
backToResearch[/Next Action,\nwith the reason/]:::next
toReflection[/Reflection/]:::next
confirmedModule[/Confirmed Module/]:::green
draft --> quotesOnChunks
quotesOnChunks --> |no| backToResearch
quotesOnChunks --> |yes| urlsScraped
urlsScraped --> |no| backToResearch
urlsScraped --> |yes| usersWorkIn
usersWorkIn --> |yes| confirmedModule
usersWorkIn --> |no| toReflection
quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
quotesOnChunks -.- quotesNote
urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
urlsScraped -.- urlsNote
rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
usersWorkIn -.- rubricNote
backNote["Retries count against<br/>the loop's budget"]:::note
backToResearch -.- backNoteReflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
modules[(Modules)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
passed[/From Proof of Work:\nConfirmed Module/]:::next
subgraph reflection[6. Reflection]
reviewModule{"Is this a module?"}:::orange
notModule[/Not a Module/]:::red
end
confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
backToResearch[/Next Action,\nwith what is missing/]:::next
flagged --> reviewModule
claudeOpus --> reviewModule
reviewModule --> |yes| confirmedModule
reviewModule --> |no| notModule
reviewModule --> |needs more\nevidence| backToResearch
passed --> confirmedModule
confirmedModule --> modules
reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
reviewModule -.- reflectionNote
backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
backToResearch -.- backNote
confirmedNote["Only confirmed modules<br/>are saved"]:::note
confirmedModule -.- confirmedNoteNo title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
4. Licence Researcher
No title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
moduleLicences[(Module Licences)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
passed[/From Proof of Work:\nLicence Facts/]:::next
subgraph reflection[6. Reflection]
reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
licenceFacts[/Licence Facts/]:::green
flagged --> reviewClaim
claudeOpus --> reviewClaim
reviewClaim --> |yes| licenceFacts
reviewClaim --> |no, or needs\nmore evidence| backToNextAction
passed --> licenceFacts
licenceFacts --> moduleLicences
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewClaim -.- reflectionNote
backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
backToNextAction -.- backNoteNo title
For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.
In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
6. Tier Researcher
No title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
flowchart LR
classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;
subgraph Tables
bundleTiers[(Bundle Tiers)]:::orange
end
subgraph Models
claudeOpus((Claude Opus))
end
flagged[/From Proof of Work:\npairs + evidence trail/]:::next
passed[/From Proof of Work:\nTier Order/]:::next
subgraph reflection[6. Reflection]
reviewPair{"Does one bundle include\neverything in the other?"}:::orange
end
backToNextAction[/Next Action,\nwith what is missing/]:::next
tierOrder[/Tier Order/]:::green
notNested[/Not Nested/]:::red
flagged --> reviewPair
claudeOpus --> reviewPair
reviewPair --> |yes| tierOrder
reviewPair --> |no, separate products| notNested
reviewPair --> |needs more evidence| backToNextAction
passed --> tierOrder
tierOrder --> bundleTiers
reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
reviewPair -.- reflectionNote
notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
notNested -.- notNestedNoteNo title
For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.
In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}No title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
Domain Searcher
Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.
Rendering diagram...
URL Scraper
Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.
Rendering diagram...
Three smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).
Rendering diagram...
Checklist Updater
The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.
Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.
update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.
The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.
add_item
Open a new item it found while reading
Rejects duplicates, sets origin: 'added', caps the number of items
set_note
Record a working guess for an item
Stores it, and never treats it as evidence
add_lead
Queue a URL or a search to try later
Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk
drop_lead
Drop a lead it no longer needs
Removes it
propose_answer
Claim an item is answered, with quotes
Runs proof of work at once, then closes the item or returns the reason it failed
propose_negative
Claim an item is answered no, with quotes
The same, and closes it as closed-negative
No title
Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.
Stack
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.
The design maps onto LangGraph like this:
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.
Check three things before building on them:
The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
If not, write a small LangChain chat model over the Anthropic Bedrock SDK.
LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.
API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.
@langchain/langgraph
1.7.2
State, graph, subgraphs, Send fan-out
@langchain/core
1.3.5
Messages and tools
@langchain/aws
1.5.2
ChatBedrockConverse, Claude on Bedrock
@langchain/langgraph-checkpoint-postgres
1.1.2
Saving state for resume
@mendable/firecrawl-js
4.52.0
Domain Searcher and URL Scraper
Checklist in memory
A field in the graph state, defined with Annotation.Root
Research loop
Nodes and conditional edges
"Every item closed or out of budget"
A conditional edge after each tool call
Page budget
nextAction binds the URL Scraper only while pages are left
Independent reviewer
Reflection is a subgraph with its own smaller state, so it can't see notes
Save and resume
Postgres checkpointer, one thread_id per module
Whole pipeline
A parent graph, with one Module Researcher per new name through Send
LangGraph nodes
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
No title
Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Seed Checklist
seed node
Checklist? and Budget left?
Conditional edges after callTools
Next Action
nextAction node, with the URL Scraper bound only while pages are left
Call Tool
callTools node, custom
Describe or Decide
draft node
Proof of Work
proofOfWork node
Reflection
A subgraph with its own smaller state
Why callTools is custom
LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:
Counts it against the budget, except update_checklist.
Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.
Runs the tool, which checks its cache before Firecrawl.
Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.
Applies checklist operations, running proof of work for propose_answer and propose_negative.
Returns one tool message per call, with the result and the current Next Action view.
Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.
Project layout
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.
1ground_truth/
2 src/
3 core/ plain TypeScript, no LangChain, unit-tested
4 checklist/types.ts the checklist object
5 checklist/ops.ts add_item, set_note, propose_answer, ...
6 checklist/views.ts the Next Action, Draft and Reflection views
7 checklist/seeds/module.ts the four seeded items
8 evidence/ text normalisation and the verbatim quote check
9 urlRules.ts
10 budget.ts
11 tools/ LangChain tool wrappers: schemas and descriptions only
12 domainSearch.ts
13 scrapeUrl.ts
14 docSearch.ts
15 updateChecklist.ts
16 cache/firecrawlCache.ts raw Firecrawl answers on disk
17 llm/models.ts the Bedrock chat model
18 agents/moduleResearcher/
19 state.ts
20 graph.ts
21 prompts.ts
22 nodes/ seed, nextAction, callTools, draft, proofOfWork
23 reflection/graph.ts the reflection subgraph
24 cli/researchModule.ts
25 tests/Build order
Each step works and can be shown before the next one starts.
Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.
Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.
Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.
Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.
Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.
Reflection subgraph, Postgres checkpointer and CLI.
First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.
Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
No title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}No title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Rendering diagram...
Proof of Work
Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.
Rendering diagram...
Rendering diagram...
Reflection
Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.
Rendering diagram...
Rendering diagram...
No title
Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.
In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.
Overview
Rendering diagram...
Similarity Check
Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.
Rendering diagram...
Rendering diagram...
Proof of Work
Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.
Rendering diagram...
Rendering diagram...
Reflection
For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.
Rendering diagram...
Rendering diagram...
No title
All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:
Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.
Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.
Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.
Research Loop
Rendering diagram...
Rendering diagram...
Call Tool
Rendering diagram...
Rendering diagram...
State
Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';
interface ResearchState<Input, Draft> {
// The agent's input. Set once at the start of the run.
input: Input;
// The questions this run must answer. Replaced as a whole, only by code:
// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
checklist: Checklist;
// Every chunk read this run, by chunk id. Quotes can only come from these.
// New chunks are added, old ones kept.
chunks: Record<string, Chunk>;
// The evidence trail: every tool call, in order. Appended.
// Code also uses it to check that a URL was found during the run before scraping it.
trail: TrailEntry[];
// The conversation with the model. Appended, never rewritten.
messages: BaseMessage[];
// The proposed answer, once every checklist item is closed
draft: Draft | null;
// How the run ended, once it has
outcome: Outcome | null;
}
interface Checklist {
// Which research agent this checklist belongs to
agent: 'module' | 'licence' | 'tier';
// What every question is about, e.g. the suggested module name "Quote Management".
// Set when the checklist is created, and never changed.
subject: string;
items: ChecklistItem[];
// URLs and searches the model queued to try later
leads: string[];
// What is left: searches and scrapes only
budget: { toolCalls: number; pages: number };
}
interface ChecklistItem {
// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
id: string;
question: string;
// seeded by code at the start, or added by Next Action while reading
origin: 'seeded' | 'added';
// Only proof of work closes an item
status: 'open' | 'closed' | 'closed-negative' | 'reopened';
// Set only when a quote passes proof of work
answer?: string;
// The quotes that closed the item, each on a chunk read this run
evidence: Evidence[];
// The model's working guess. Never evidence, and never shown to reflection.
note?: string;
// Why the item was reopened
failure?: string;
// Tool calls spent on this item
tried: string[];
}
interface Evidence {
chunkId: string;
url: string;
quote: string;
}
interface Chunk {
// The page's final URL, after redirects
url: string;
// The headings above this piece of the page, kept apart from the text
headings: string;
// About 800 characters of the cleaned page
text: string;
// Links inside this piece, which the agent may follow
links: string[];
}
// A search result is a snippet from the search index, never evidence
interface SearchResult {
url: string;
title: string;
description: string;
// Rank in the search results
position: number;
}
type TrailEntry =
// Site searches keep their results in full
| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
// the text is already in chunks
| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
| {
tool: 'docSearch';
args: { query: string };
result: { chunkIds: string[] };
};
// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';
// Module Researcher
interface Suggestion {
// The name step 1 suggested for a cluster of page visits
name: string;
// Page visits from that cluster, as title:::path
examples: string[];
}
interface ModuleDraft {
// The vendor's own name for the module
title: string;
// Written from the closed items' answers, and nothing else
description: string;
// The pages behind the closing quotes, built by code
citations: string[];
// Other names it is known by, such as the name traffic gave it
merged_names: string[];
}
type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
// Licence Researcher
interface LicenceFacts {
// Sold as a licence of its own
standalone: boolean;
// The licences it comes inside
includedIn: string[];
}
type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
// Tier Researcher
interface BundleGroup {
vendor: string;
// Each bundle, with the modules it includes
bundles: { name: string; modules: string[] }[];
}
interface TierOrder {
// Each pair where one bundle includes everything in the other
pairs: { superset: string; subset: string }[];
}
type TierResearchState = ResearchState<BundleGroup, TierOrder>;The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;Updating the state
Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.
// Sketch: ground_truth/src/core/research/updateState.ts
// What each tool returns. No tool touches the state itself.
type ToolResult =
// Domain Searcher
| { kind: 'search'; results: SearchResult[] }
// URL Scraper and Vendor Documents Searcher
| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
// Checklist Updater, after the code checks
| { kind: 'checklist'; checklist: Checklist };
// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
call: { tool: string; args: unknown },
result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
switch (result.kind) {
case 'search':
return {
trail: [{ tool: call.tool, args: call.args, result: result.results }],
};
case 'chunks':
return {
chunks: Object.fromEntries(
result.chunks.map(({ id, ...chunk }) => [id, chunk])
),
trail: [
{
tool: call.tool,
args: call.args,
result: { chunkIds: result.chunks.map(chunk => chunk.id) },
},
],
};
case 'checklist':
return { checklist: result.checklist };
}
}Keeping this out of the tools has two benefits:
The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.
One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.
1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}Who sees what
Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.
Next Action
Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left
Picks the item to work on and the tool call most likely to close it
Draft
Closed items only, with their question, answer and evidence
The description can't use unchecked guesses
Reflection
Each item's question, answer and quotes. No notes, leads, failures or calls tried
Keeps the reviewer independent of the researcher's reasoning
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}No title
Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.
In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.
Research Loop
Rendering diagram...
Rendering diagram...
Checklist questions
Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.
Next Action can open three kinds of item as it reads:
vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.
part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.
ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.
The research ends in one of three ways:
Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.
Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.
Unresolved: the budget runs out with items still open. The open items are saved for the next run.
exists
Does the vendor name this thing?
A quote on a chunk scraped this run contains the suggested name or a close variant
None
is-workspace
Is it somewhere users work, not a setting, API, report type or marketing term?
The quote passes the evidence check and names the module
Do the quotes describe something users work in?
what-it-does
What do users do in it?
At least one quote states its purpose, so the description stands on quotes
Is the description supported by the quotes alone?
matches-traffic
Is this the same thing the traffic shows?
The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/
Do the examples fit this description?
Proof of Work
Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.
Rendering diagram...
Rendering diagram...
Reflection
Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.
Rendering diagram...
Rendering diagram...
Proof of Work
The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.
Rendering diagram...
Reflection
A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.
Rendering diagram...
Rendering diagram...
Proof of Work
As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.
Rendering diagram...
Reflection
"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.
Rendering diagram...
Rendering diagram...
1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}1// Sketch: ground_truth/src/core/research/state.ts
2import type { BaseMessage } from '@langchain/core/messages';
3
4interface ResearchState<Input, Draft> {
5 // The agent's input. Set once at the start of the run.
6 input: Input;
7 // The questions this run must answer. Replaced as a whole, only by code:
8 // Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
9 checklist: Checklist;
10 // Every chunk read this run, by chunk id. Quotes can only come from these.
11 // New chunks are added, old ones kept.
12 chunks: Record<string, Chunk>;
13 // The evidence trail: every tool call, in order. Appended.
14 // Code also uses it to check that a URL was found during the run before scraping it.
15 trail: TrailEntry[];
16 // The conversation with the model. Appended, never rewritten.
17 messages: BaseMessage[];
18 // The proposed answer, once every checklist item is closed
19 draft: Draft | null;
20 // How the run ended, once it has
21 outcome: Outcome | null;
22}
23
24interface Checklist {
25 // Which research agent this checklist belongs to
26 agent: 'module' | 'licence' | 'tier';
27 // What every question is about, e.g. the suggested module name "Quote Management".
28 // Set when the checklist is created, and never changed.
29 subject: string;
30 items: ChecklistItem[];
31 // URLs and searches the model queued to try later
32 leads: string[];
33 // What is left: searches and scrapes only
34 budget: { toolCalls: number; pages: number };
35}
36
37interface ChecklistItem {
38 // e.g. "is-workspace", "vendor-name:Salesforce CPQ"
39 id: string;
40 question: string;
41 // seeded by code at the start, or added by Next Action while reading
42 origin: 'seeded' | 'added';
43 // Only proof of work closes an item
44 status: 'open' | 'closed' | 'closed-negative' | 'reopened';
45 // Set only when a quote passes proof of work
46 answer?: string;
47 // The quotes that closed the item, each on a chunk read this run
48 evidence: Evidence[];
49 // The model's working guess. Never evidence, and never shown to reflection.
50 note?: string;
51 // Why the item was reopened
52 failure?: string;
53 // Tool calls spent on this item
54 tried: string[];
55}
56
57interface Evidence {
58 chunkId: string;
59 url: string;
60 quote: string;
61}
62
63interface Chunk {
64 // The page's final URL, after redirects
65 url: string;
66 // The headings above this piece of the page, kept apart from the text
67 headings: string;
68 // About 800 characters of the cleaned page
69 text: string;
70 // Links inside this piece, which the agent may follow
71 links: string[];
72}
73
74// A search result is a snippet from the search index, never evidence
75interface SearchResult {
76 url: string;
77 title: string;
78 description: string;
79 // Rank in the search results
80 position: number;
81}
82
83type TrailEntry =
84 // Site searches keep their results in full
85 | { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
86 // Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
87 // the text is already in chunks
88 | { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
89 | {
90 tool: 'docSearch';
91 args: { query: string };
92 result: { chunkIds: string[] };
93 };
94
95// confirmed: the draft is saved.
96// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
97// unresolved: the budget ran out, and the open items are saved for the next run.
98type Outcome = 'confirmed' | 'rejected' | 'unresolved';
99
100// Module Researcher
101interface Suggestion {
102 // The name step 1 suggested for a cluster of page visits
103 name: string;
104 // Page visits from that cluster, as title:::path
105 examples: string[];
106}
107
108interface ModuleDraft {
109 // The vendor's own name for the module
110 title: string;
111 // Written from the closed items' answers, and nothing else
112 description: string;
113 // The pages behind the closing quotes, built by code
114 citations: string[];
115 // Other names it is known by, such as the name traffic gave it
116 merged_names: string[];
117}
118
119type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;
120
121// Licence Researcher
122interface LicenceFacts {
123 // Sold as a licence of its own
124 standalone: boolean;
125 // The licences it comes inside
126 includedIn: string[];
127}
128
129type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;
130
131// Tier Researcher
132interface BundleGroup {
133 vendor: string;
134 // Each bundle, with the modules it includes
135 bundles: { name: string; modules: string[] }[];
136}
137
138interface TierOrder {
139 // Each pair where one bundle includes everything in the other
140 pairs: { superset: string; subset: string }[];
141}
142
143type TierResearchState = ResearchState<BundleGroup, TierOrder>;1// Sketch: ground_truth/src/core/research/updateState.ts
2
3// What each tool returns. No tool touches the state itself.
4type ToolResult =
5 // Domain Searcher
6 | { kind: 'search'; results: SearchResult[] }
7 // URL Scraper and Vendor Documents Searcher
8 | { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
9 // Checklist Updater, after the code checks
10 | { kind: 'checklist'; checklist: Checklist };
11
12// Returns only the fields that change. LangGraph merges them in:
13// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
14function updateState(
15 call: { tool: string; args: unknown },
16 result: ToolResult
17): Partial<ResearchState<unknown, unknown>> {
18 switch (result.kind) {
19 case 'search':
20 return {
21 trail: [{ tool: call.tool, args: call.args, result: result.results }],
22 };
23 case 'chunks':
24 return {
25 chunks: Object.fromEntries(
26 result.chunks.map(({ id, ...chunk }) => [id, chunk])
27 ),
28 trail: [
29 {
30 tool: call.tool,
31 args: call.args,
32 result: { chunkIds: result.chunks.map(chunk => chunk.id) },
33 },
34 ],
35 };
36 case 'checklist':
37 return { checklist: result.checklist };
38 }
39}