Wide layout

Table of contents

Ground Truth

An Innovathon 2026 proposal: a suite of research agents that works out which software licence each employee actually needs, from what they do in the app and from evidence on the vendor's own pages.

It builds on the existing licence usage dashboard PoC. The one-pager is on Confluence.

Why

To know which licence a person needs, you need two facts: which parts of the app they use, and which licence each part requires.

The PoC gets both, but in a fixed order. It first scrapes the vendor's whole documentation site and builds a catalogue of every licence and module, then clusters page-view traffic into the modules people use, and finally joins the two. The traffic half works well. The documentation half is where it struggles:

  • It has to guess up front which of thousands of pages matter, and most of its tuning goes into that guess.

  • The modules it finds in the docs are named differently from the ones it finds in traffic ("Salesforce CPQ" against "Quote Management"), so a separate step has to link them.

  • It can only see what vendors publish. On Datadog it predicted 33 admins where there were 2, because the customer's own roles never appear in vendor documentation.

Ground Truth flips the order. Traffic decides what gets researched. Each module people actually use triggers a small investigation into what that module is and which licence it needs, and every answer has to carry a quote that code can find on the page. Answers without evidence are marked unresolved, not guessed.

How it works

Six steps run in order. Steps 1, 2 and 5 are workflows: they follow fixed steps, and step 5, the Bundle Finder, is plain code with no LLM. Steps 3, 4 and 6 are research agents: they choose their own steps.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;

    pageVisits[(Page Visit\nEmbeddings)]:::orange
    suggest[1. Suggest\nModule Names]:::blue
    merge[2. Merge\nModule Names]:::blue
    moduleResearcher[3. Module\nResearcher]:::blue
    licenceResearcher[4. Licence\nResearcher]:::blue
    bundleFinder[5. Bundle Finder\ncode only]
    tierResearcher[6. Tier\nResearcher]:::blue
    tagVisits[Tag Visits with\na Module Name]
    predict[Predict Licence Usage\nexisting PoC, SQL]
    dashboard[/Licence Usage Dashboard\nwho needs which licence/]:::green

    pageVisits --> |untagged clusters| suggest --> |suggested names| merge
    merge --> |joins a saved module| tagVisits
    merge --> |new module| moduleResearcher
    moduleResearcher --> |confirmed module| tagVisits
    moduleResearcher --> |confirmed module| licenceResearcher
    licenceResearcher --> |licence facts| bundleFinder --> |bundle groups| tierResearcher
    tagVisits --> predict
    licenceResearcher --> predict
    tierResearcher --> |tier order| predict
    predict --> dashboard
Step
Input
Question it answers
Output
Models

1. Suggest Module Names

An untagged traffic cluster

What would you call this area of the app?

Suggested module names

Sonnet drafts, Opus reviews

2. Merge Module Names

A suggested name and the closest saved modules

Is this a module we already have?

Join a saved module, or suggest a new one

Sonnet compares, Opus reviews

3. Module Researcher

A new module name

Is it really a module in the vendor's product, and what is it?

A confirmed module: title, description, citations

Opus

4. Licence Researcher

A confirmed module

Is it sold on its own, and which licences does it come inside?

Licence facts, each with a quote

Opus

5. Bundle Finder

Licence facts

Which entries are bundles?

Bundle groups, one per vendor

Code

6. Tier Researcher

A bundle group

Which bundle includes everything in which?

Tier order

Opus

The last step, predicting who needs which licence, is the PoC's existing SQL join. It doesn't change.

How to read the diagrams

All diagrams use the same shapes and colours. Every orange step goes through the LLM call path. A rubric check is answered by the model about its own output, so nothing verifies it: a doubtful answer sends the item to reflection.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph steps[Steps]
        direction TB
        llmStep[Calls Claude]:::orange
        codeStep[Runs code]
        model((Model used))
    end

    subgraph checks[Checks]
        direction TB
        codeCheck{"Code check:\ntrusted"}
        rubricCheck{"Rubric check:\nself-graded"}:::rubric
        reflection{"Reflection:\na second\nmodel call"}:::orange
    end

    subgraph stores[Stores]
        direction TB
        cache[(Firecrawl cache)]:::yellow
        store[(Table or\nvector store)]:::orange
    end

    subgraph results[Results]
        direction TB
        kept[/Kept/]:::green
        dropped[/Dropped, failed\nor unresolved/]:::red
    end

    subgraph other[Links and notes]
        direction TB
        link[/Drawn in full in\nanother diagram/]:::next
        note["A note"]:::note
    end

    codeCheck --> |yes| kept
    codeCheck --> |no| dropped

Design principles

These hold for every agent below.

  • Evidence first. A claim is kept only with a quote, and code checks that the quote is on a page the agent actually read. This is the proof-of-work step in every agent.

  • Three layers of checks, cheapest first. Code checks run first and can be trusted. Rubric questions come next: the model answers them about its own output, so they only decide whether an item needs review. Reflection comes last, where a second model call reviews what is still doubtful.

  • An independent reviewer. Reflection uses Opus, and where Opus also did the research, it reviews with a fresh context: the evidence, not the reasoning that produced it.

  • Safe defaults. When an agent is unsure, it takes the outcome that costs least if wrong: a name is dropped and its pages are retried on the next run, a new module is suggested instead of a risky merge, or a research question is left unresolved.

  • Bounded autonomy. The research agents choose their own next step, but within a budget of tool calls and pages, on vendor domains only, and never scraping the same URL twice.

  • Pay once. Every Firecrawl answer is cached, so a page is never paid for twice, and cleaning rules can change without fetching the page again.

Shared building blocks

Stores

Store
What it holds

Page Visit Embeddings

One row per page seen in traffic, embedded from title:::path, for example Quote Management:::/lightning/o/Quote/list. The host is left out because it is the same for every page, and ids and personal data in titles are masked as <id>. MiniLM, 384 dimensions.

Vendor Documentation Embeddings

Vendor pages cut into 800-character chunks with 150 characters of overlap. Each chunk starts with the headings above it ("Pricing › Log Management"), so a table row cut off from its heading still says what it prices. Searched by embedding similarity and BM25 keywords together.

Modules and Module Embeddings

Confirmed modules only: name, description, citation URLs, and the other names it has been merged under (merged_names).

Module Licences

For each module: whether it is sold on its own, and the licences it comes inside, each with its quote.

Bundle Tiers

Pairs of bundles where one includes everything in the other, each with its quote.

Firecrawl Domain Search Cache and URL Scrape Cache

Firecrawl's raw answers on disk, keyed by the request.

LLM Calls

Every LLM call takes this path, Opus reflection calls included, so every orange step in the diagrams below is one of these. Network failures are retried with backoff. An unusable answer is not retried, because the same prompt gives the same answer. A call that fails falls back to its step's safe default.

mermaid
flowchart LR
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    sendPrompt[Send Prompt]:::orange
    networkOk{"Network ok?"}
    backOff[Back Off]
    shapeOk{"Answer usable?"}
    callFailed[/Call Failed/]:::red
    toProofOfWork[/Proof of Work/]:::next

    sendPrompt --> networkOk
    networkOk --> |no, attempt 1-2| backOff --> sendPrompt
    networkOk --> |no, attempt 3| callFailed
    networkOk --> |yes| shapeOk
    shapeOk --> |no| callFailed
    shapeOk --> |yes| toProofOfWork

    networkNote["Network error, throttle or 5xx<br/>Retried after 1s, then 2s"]:::note
    networkOk -.- networkNote

    shapeNote["JSON of the expected shape,<br/>not cut off at max_tokens<br/>Not retried"]:::note
    shapeOk -.- shapeNote

    failedNote["The step's safe outcome:<br/>pages stay untagged, a new module<br/>is suggested, or Unresolved"]:::note
    callFailed -.- failedNote

Proof of Work

Every LLM answer is checked before anything is kept. The checks run cheapest and most trustworthy first.

  1. Code checks ask whether the evidence is real: is each quote on a page the model was shown, is every cited URL one it read, does the quote name what the answer claims? Code can check these, so a failure is final for this answer. The item falls back to its safe default, or a research agent goes back to research with the reason.

  2. Rubric checks are yes/no questions the model answers in the same response, about its own answer. Nothing can verify them, so they never keep an item on their own authority: they only decide whether it needs a second look.

mermaid
flowchart LR
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    fromLlm[/From LLM call:\nanswer + quotes\n+ rubric answers/]:::next

    subgraph proofOfWork[Proof of Work]
        codeChecks{"Code checks:\nis the evidence real?"}
        rubricChecks{"Rubric checks:\ndoes the answer\nhold up?"}:::rubric
    end

    kept[/Kept/]:::green
    failed[/Safe default, or\nback to research\nwith the reason/]:::red
    toReflection[/Reflection/]:::next

    fromLlm --> codeChecks
    codeChecks --> |no| failed
    codeChecks --> |yes| rubricChecks
    rubricChecks --> |yes| kept
    rubricChecks --> |doubtful| toReflection

    codeNote["e.g. quote found word for word<br/>on a chunk the model read"]:::note
    codeChecks -.- codeNote

    rubricNote["e.g. #quot;do the examples<br/>support the name?#quot;"]:::note
    rubricChecks -.- rubricNote

Reflection

A flagged item gets a second opinion from Opus. The reviewer is independent of the answer it reviews: either a different model from the one that drafted it, or Opus with a fresh context that sees the evidence but not the reasoning behind it.

mermaid
flowchart LR
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    claudeOpus((Claude Opus))

    flagged[/From Proof of Work:\nflagged item +\nits evidence/]:::next
    review{"Does the evidence\nsupport it?"}:::orange
    recheck{"Rename or new answer\npasses proof of work?"}

    kept[/Kept/]:::green
    safeDefault[/Safe default/]:::red
    backToResearch[/Back to research,\nwith what is missing/]:::next

    flagged --> review
    claudeOpus --> review
    review --> |yes| kept
    review --> |no| safeDefault
    review --> |a better answer| recheck
    recheck --> |yes| kept
    recheck --> |no| safeDefault
    review --> |needs more evidence| backToResearch

    recheckNote["Checked once only,<br/>so reflection can't loop"]:::note
    recheck -.- recheckNote

    backNote["Research agents only<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
    backToResearch -.- backNote

How each agent uses them

Agent
Code checks
Rubric questions
Reflection can

1. Suggest Module Names

No ids or digits in the name; not a generic word

Do the examples support the name?

Keep, drop, or rename

2. Merge Module Names

Cited URLs are from the pages sent; one name contains the other

Do users of both do the same task? Do they work on different records?

Join, or suggest as new

3. Module Researcher

Quotes are on the cited chunks; cited URLs were scraped this run

Do the quotes describe something users work in?

Confirm, reject, or ask for more evidence

4. Licence Researcher

The same, plus: the quote names the module, and every licence it comes inside

Does it get its own price, plan or SKU? Does it only come inside other licences? Is it only listed in passing?

Confirm, or ask for more evidence

6. Tier Researcher

The same, plus: the quote names both bundles; the order has no cycles

Does one have everything in the other? Is it about these bundles? Is it only a comparison table cell?

Confirm, say not nested, or ask for more evidence

Workflows

These run the same steps every time. Where there is an LLM, it writes and judges, but it never chooses what to do next, which keeps them cheap and easy to test. Steps 1 and 2 turn raw traffic into module names. Step 5 runs later, between the Licence and Tier Researchers, and uses no LLM at all.

1. Suggest Module Names

Takes an untagged traffic cluster and proposes names for the area of the app its pages belong to. The names are suggestions only: step 2 decides whether each one is new.

In: the cluster's K most distinct URL and title pairs. Out: suggested module names. Models: Sonnet drafts, Opus reviews.

Overview

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Vectors
        pageVisitEmbeddings[(Page Visit\nEmbeddings)]:::orange
    end

    claudeSonnet((Claude Sonnet))

    subgraph Tools
        retriveUntaggedClusterForTagging[Retrieve Untagged\nClusters for Tagging]
    end

    gatherClusterInfo[1. Gather Cluster Info]
    moduleNamePrompt[2. Prompt LLM to Generate Module Names]:::orange
    toProofOfWork[/3. Proof of Work/]:::next
    toReflection[/4. Reflection/]:::next
    suggestions[/Suggested Module Names/]:::green
    dropSuggestion[/Drop Suggestion/]:::red

    pageVisitEmbeddings --> |top N\nUntagged clusters| retriveUntaggedClusterForTagging --> |top K distinct\nurl + title| gatherClusterInfo
    gatherClusterInfo --> moduleNamePrompt
    claudeSonnet --> moduleNamePrompt
    moduleNamePrompt --> |suggestions +\nrubric answer| toProofOfWork
    toProofOfWork --> |passes| suggestions
    toProofOfWork --> |id or digits| dropSuggestion
    toProofOfWork --> |flagged| toReflection
    toReflection --> |keep, or rename passes| suggestions
    toReflection --> |drop| dropSuggestion

    gatherNote["Distinct url + title pairs<br/>ids and personal data masked"]:::note
    gatherClusterInfo -.- gatherNote

    promptNote["Sonnet drafts names<br/>and answers the rubric"]:::note
    moduleNamePrompt -.- promptNote

    dropNote["Pages stay untagged,<br/>retried next run"]:::note
    dropSuggestion -.- dropNote

Proof of Work

Two code checks and one rubric question. A name built from a record (an id, digits) is dropped outright. A generic name, or one the examples don't support, goes to reflection.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    fromPrompt[/From Prompt:\nsuggestions +\nrubric answer/]:::next

    subgraph proofOfWorkCheck[3. Proof of Work]
        hasIdOrDigits{"Name has\n&lt;id&gt; or digits?"}
        isGenericName{"Name on the\ngeneric word list?"}
        isSupported{"Rubric:\nDo the examples\nsupport the name?"}:::rubric
        dropAtProofOfWork[Drop Suggestion]:::red
    end

    keepSuggestion[/Keep Suggestion/]:::green
    toReflection[/Reflection/]:::next

    fromPrompt --> hasIdOrDigits
    hasIdOrDigits --> |yes| dropAtProofOfWork
    hasIdOrDigits --> |no| isGenericName
    isGenericName --> |yes| toReflection
    isGenericName --> |no| isSupported
    isSupported --> |yes| keepSuggestion
    isSupported --> |no| toReflection

    idOrDigitsNote["Named after a record, not a screen<br/>e.g. Case 00123, Meeting with &lt;id&gt;"]:::note
    hasIdOrDigits -.- idOrDigitsNote

    genericNameNote["Too broad to be a module<br/>e.g. Home, Settings, List, View"]:::note
    isGenericName -.- genericNameNote

    rubricNote["Sonnet answers per name, same response<br/>{ name: #quot;Quote Management#quot;,<br/>supported_by_examples: true }<br/>Self-graded, not checked"]:::note
    isSupported -.- rubricNote

Reflection

Opus can keep the name, drop it, or suggest a better one. A rename is checked once more, and dropped if it fails, so the loop can't repeat.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    claudeOpus((Claude Opus))

    flagged[/From Proof of Work:\nflagged suggestion/]:::next

    subgraph reflection[4. Reflection]
        conferKeepOrDrop{"Keep, drop\nor rename?"}:::orange
        subgraph reflectionProofOfWork[4.1 Proof of Work]
            renameHasIdOrDigits{"Rename has\n&lt;id&gt; or digits?"}
            renameIsSupported{"Rubric:\nDo the examples\nsupport the rename?"}:::rubric
        end
        dropAtReflection[Drop Suggestion]:::red
    end

    keepSuggestion[/Keep Suggestion/]:::green

    flagged --> conferKeepOrDrop
    claudeOpus --> conferKeepOrDrop
    conferKeepOrDrop --> |keep| keepSuggestion
    conferKeepOrDrop --> |drop| dropAtReflection
    conferKeepOrDrop --> |rename +\nrubric answer| renameHasIdOrDigits
    renameHasIdOrDigits --> |yes| dropAtReflection
    renameHasIdOrDigits --> |no| renameIsSupported
    renameIsSupported --> |yes| keepSuggestion
    renameIsSupported --> |no| dropAtReflection

    reflectionNote["Opus, a different model<br/>Sees the examples and<br/>why the check failed<br/>A rename is checked once,<br/>failing any check drops it"]:::note
    conferKeepOrDrop -.- reflectionNote

2. Merge Module Names

Decides whether each suggested name is a module the app already has ("Opportunity Mgmt" and "Opportunity Management") or a new one. A wrong merge files one module's pages under another, so when in doubt it suggests a new module instead.

In: a suggested name, and the closest saved modules from Hybrid Module Search. Out: join a saved module, or suggest a new one. Models: Sonnet compares, Opus reviews.

Overview

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Vectors
        moduleEmbeddings[(Module\nEmbeddings)]:::orange
    end

    subgraph Tables
        modules[(Modules)]:::orange
    end

    claudeSonnet((Claude Sonnet))

    subgraph Tools
        hybridModuleSearch[Hybrid Module Search\nBM25 + vector]
    end

    suggestions[/Suggested Module Names/]:::green
    gatherSuggestion[1. Gather Suggestion\nname + its examples]
    toSimilarityCheck[/2. Similarity Check/]:::next

    subgraph llmSimilarityCheck[LLM Similarity Check]
        comparePrompt[3. Prompt LLM to Compare\nsuggestion vs candidate]:::orange
        toProofOfWork[/4. Proof of Work/]:::next
        toReflection[/5. Reflection/]:::next
    end

    joinAtSimilarity[/Join Saved Module/]:::green
    newAtSimilarity[/Suggest as New Module/]:::green
    joinAtProofOfWork[/Join Saved Module/]:::green
    newAtProofOfWork[/Suggest as New Module/]:::green
    joinAtReflection[/Join Saved Module/]:::green
    newAtReflection[/Suggest as New Module/]:::green

    moduleEmbeddings --> hybridModuleSearch
    modules --> hybridModuleSearch
    suggestions --> gatherSuggestion
    hybridModuleSearch --> |top K saved modules:\nname, description,\nexamples| gatherSuggestion
    gatherSuggestion --> toSimilarityCheck
    toSimilarityCheck --> |match| joinAtSimilarity
    toSimilarityCheck --> |no candidate| newAtSimilarity
    toSimilarityCheck --> |candidate| comparePrompt
    claudeSonnet --> comparePrompt
    comparePrompt --> |same module? +\nrubric answers +\ncited URLs| toProofOfWork
    toProofOfWork --> |passes| joinAtProofOfWork
    toProofOfWork --> |fails| newAtProofOfWork
    toProofOfWork --> |flagged| toReflection
    toReflection --> |yes| joinAtReflection
    toReflection --> |no, or no answer| newAtReflection

    searchNote["Vector: name embedding similarity<br/>BM25: name, merged names, description<br/>Scores pooled, top K kept"]:::note
    hybridModuleSearch -.- searchNote

    compareNote["Sees the saved module's<br/>name and description<br/>+ 8 pages of each<br/>At most 3 candidates per name"]:::note
    comparePrompt -.- compareNote

    joinNote["Suggested name saved in<br/>merged_names: next time<br/>it matches at step 2"]:::note
    joinAtSimilarity -.- joinNote

Similarity Check

Settles the easy cases without an LLM: the same name or a near-identical one joins, and a name with no close candidate is new. Only the middle band goes to Sonnet.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    fromGather[/From Gather:\nsuggestion + top K\nsaved modules/]:::next

    subgraph similarityCheck[2. Similarity Check]
        sameName{"Same name, merged name,\nor similarity 0.95 or more?"}
        isCandidate{"Similarity 0.6 or more,\nor one name's words\nall in the other?"}
    end

    joinModule[/Join Saved Module/]:::green
    newModule[/Suggest as New Module/]:::green
    toCompare[/3. Prompt LLM to Compare/]:::next

    fromGather --> sameName
    sameName --> |yes| joinModule
    sameName --> |no| isCandidate
    isCandidate --> |no| newModule
    isCandidate --> |yes| toCompare

    sameNameNote["Case-insensitive<br/>billing → Billing<br/>Opportunity Mgmt → Opportunity Management"]:::note
    sameName -.- sameNameNote

    candidateNote["Communications and Protocols<br/>→ Communications (0.74)"]:::note
    isCandidate -.- candidateNote

Proof of Work

Sonnet's answer must cite pages that were actually sent, and its rubric answers must agree that both modules serve the same task on the same records.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    fromCompare[/From Prompt:\nsame module? +\nrubric answers +\ncited URLs/]:::next

    subgraph proofOfWorkCheck[4. Proof of Work]
        citedUrlsSent{"Cited URLs are\nfrom the pages sent?"}
        sameJob{"Rubric:\nDo users of both modules\ndo the same task?"}:::rubric
        differentObjects{"Rubric:\nDifferent objects?"}:::rubric
        nameContains{"One name\ncontains the other?"}
    end

    joinModule[/Join Saved Module/]:::green
    newModule[/Suggest as New Module/]:::green
    toReflection[/Reflection/]:::next

    fromCompare --> citedUrlsSent
    citedUrlsSent --> |no| newModule
    citedUrlsSent --> |yes| sameJob
    sameJob --> |no| newModule
    sameJob --> |yes| differentObjects
    differentObjects --> |yes| newModule
    differentObjects --> |no| nameContains
    nameContains --> |yes| joinModule
    nameContains --> |no| toReflection

    citedUrlsNote["Sonnet cites 1-3 URLs per module<br/>as evidence for its answer<br/>A URL not among the 8 sent<br/>means it was made up"]:::note
    citedUrlsSent -.- citedUrlsNote

    sameJobNote["Judged on what users do,<br/>not how the names sound<br/>Invoicing vs Billing: yes,<br/>both create and send invoices<br/>Cases vs Case Reports: no,<br/>working cases vs analysing them"]:::note
    sameJob -.- sameJobNote

    differentObjectsNote["Work on different records?<br/>Leads vs Contacts: yes"]:::note
    differentObjects -.- differentObjectsNote

    nameContainsNote["Checked in code<br/>Quote → Quote Management"]:::note
    nameContains -.- nameContainsNote

Reflection

For pairs the names don't settle ("Invoicing" and "Billing"), Opus looks at both modules' descriptions and pages.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    claudeOpus((Claude Opus))

    flagged[/From Proof of Work:\nnames don't bear it out/]:::next

    subgraph reflection[5. Reflection]
        confirmSameModule{"Same module?"}:::orange
    end

    joinModule[/Join Saved Module/]:::green
    newModule[/Suggest as New Module/]:::green

    flagged --> confirmSameModule
    claudeOpus --> confirmSameModule
    confirmSameModule --> |yes| joinModule
    confirmSameModule --> |no, or no answer| newModule

    reflectionNote["Opus sees both modules'<br/>description and pages<br/>Invoicing vs Billing"]:::note
    confirmSameModule -.- reflectionNote

5. Bundle Finder

A code step between the Licence and Tier Researchers, not an agent. We define a bundle as an entry that includes other licences, so code reads bundles straight off the licence facts and groups them by vendor for the Tier Researcher.

mermaid
flowchart LR
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Tables
        moduleLicences[(Module Licences)]:::orange
    end

    collectEdges[1. Collect\nincluded-in edges]
    includesLicence{"Something sold\non its own\ncomes inside it?"}
    bundle[/Bundle/]:::green
    licence[/Licence/]:::green
    groupByVendor[2. Group Bundles\nby vendor]
    toTierResearcher[/Tier Researcher/]:::next

    moduleLicences --> collectEdges --> includesLicence
    includesLicence --> |yes| bundle --> groupByVendor --> toTierResearcher
    includesLicence --> |no| licence

    edgesNote["Module M comes inside X<br/>and M is sold on its own:<br/>X includes a licence"]:::note
    includesLicence -.- edgesNote

    groupNote["One group per vendor<br/>e.g. Starter, Professional,<br/>Enterprise"]:::note
    groupByVendor -.- groupNote

Research Agents

These choose their own steps. Given a question, they decide what to search, which pages to read and which links to follow, until every question on their checklist is answered or they run out of budget. They turn module names into checked facts about the vendor's product and its licences.

The research loop

All three research agents share one loop. Each one reads its input, and code seeds its checklist of questions. It then repeats two steps until every item on the checklist is closed, or the budget runs out:

  • Next Action: Opus looks at the open checklist items and the evidence so far, and picks the item to work on. It then searches, scrapes a page, follows a link from a page it has read, or updates the checklist: proposing an answer, adding a question or noting a lead.

  • Call Tool: runs the chosen tool, counts the call against the budget, and logs the result to the evidence trail that reflection reads later. A proposed answer goes through the code checks at once, and closes its item only if it passes.

Code, not the model, decides when the research is done. Once every item is closed, the researcher writes a draft answer from the closed items, and the draft goes through that researcher's own proof of work and reflection. A draft that fails reopens the items behind it and goes back to Next Action with the reason, still within the same budget.

Research Loop

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    claudeOpus((Claude Opus))

    input[/Input/]
    seed[1. Seed Checklist]
    subgraph loop[Loop]
        checklistDone{"2. Checklist?"}
        budgetLeft{"3. Budget left?"}
        nextAction["4. Next Action\npick the next action"]:::orange
        callTool[5. Call Tool]
        draft["6. Describe or Decide\nfrom the closed items"]:::orange
        toProofOfWork[/7. Proof of Work/]:::next
        toReflection[/8. Reflection/]:::next
    end
    confirmed[/Confirmed/]:::green
    rejected[/Rejected/]:::red
    unresolved[/Unresolved,\nopen items saved/]:::red

    input --> seed --> checklistDone
    checklistDone --> |items still open| budgetLeft
    checklistDone --> |every item closed| draft
    checklistDone --> |closed negative| rejected
    budgetLeft --> |yes| nextAction
    budgetLeft --> |no| unresolved
    claudeOpus --> nextAction
    nextAction --> |search, scrape or\nupdate the checklist| callTool
    callTool --> |updated checklist\n+ evidence| checklistDone
    draft --> toProofOfWork
    toProofOfWork --> |passes| confirmed
    toProofOfWork --> |fails: reopen items| checklistDone
    toProofOfWork --> |flagged| toReflection
    toReflection --> |confirm| confirmed
    toReflection --> |reject| rejected
    toReflection --> |needs more:\nreopen items| checklistDone

    seedNote["Code: the fixed questions<br/>from the agent's output<br/>and rubric"]:::note
    seed -.- seedNote

    checklistNote["Code, not the model:<br/>counts the open items"]:::note
    checklistDone -.- checklistNote

    budgetNote["Code: e.g. 15 tool calls, 10 pages<br/>Searches and scrapes only<br/>Vendor domains only"]:::note
    budgetLeft -.- budgetNote

    nextNote["Asks: which open item,<br/>and which call closes it?<br/>Never decides it has<br/>enough evidence"]:::note
    nextAction -.- nextNote

    callNote["Code: runs the tool, applies<br/>checklist updates with the<br/>code checks"]:::note
    callTool -.- callNote

    rejectedNote["e.g. not a module, not nested<br/>The Licence Researcher<br/>can't reject"]:::note
    rejected -.- rejectedNote

Call Tool

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Vectors
        vendorDocEmbeddings[(Vendor\nDocumentation\nEmbeddings)]:::orange
    end

    subgraph Tools
        domainSearcher[Domain Searcher]
        urlScraper[URL Scraper]
        vendorDocSearcher[Vendor Documents\nSearcher]
    end

    fromNextAction[/From Next Action:\naction + arguments/]:::next
    countCall[1. Count the Call\nagainst the budget]
    whichTool{"2. Which tool?"}
    urlAllowed{"URL allowed?"}
    toolAnswered{"3. Tool answered?"}
    applyOps[Apply to Checklist\ncode checks on a\nproposed answer]
    logCall[4. Update Agent State\ntrail, chunks, checklist]
    toReadResults[/Research Loop:\n2. Checklist?/]:::next
    backToNextAction[/Next Action,\nwith the error/]:::next

    fromNextAction --> |search or scrape| countCall --> whichTool
    fromNextAction --> |propose an answer\nor update the checklist| applyOps --> logCall
    whichTool --> |search the site| domainSearcher
    whichTool --> |search stored docs| vendorDocSearcher
    whichTool --> |scrape or\nfollow a link| urlAllowed
    urlAllowed --> |yes| urlScraper
    urlAllowed --> |no| backToNextAction
    vendorDocEmbeddings --> vendorDocSearcher
    domainSearcher --> toolAnswered
    vendorDocSearcher --> toolAnswered
    urlScraper --> toolAnswered
    toolAnswered --> |yes| logCall --> toReadResults
    toolAnswered --> |no| backToNextAction

    countNote["Every search or scrape spends<br/>budget, even a failed one<br/>Checklist updates are free"]:::note
    countCall -.- countNote

    applyNote["A quote that passes closes its item<br/>One that fails leaves it open,<br/>with the reason"]:::note
    applyOps -.- applyNote

    toolsNote["Each tool checks its<br/>cache before Firecrawl"]:::note
    urlScraper -.- toolsNote

    urlNote["Vendor domain only<br/>Not scraped this run<br/>Found in a search result<br/>or on a page read"]:::note
    urlAllowed -.- urlNote

    logNote["One shared function, not a tool:<br/>runs after every call, so no<br/>result goes unrecorded<br/>Writes to the agent state:<br/>trail: tool, arguments, result,<br/>every call<br/>chunks: new chunks from a<br/>scrape or stored-docs search<br/>checklist: items opened, closed<br/>or reopened<br/>The trail is what Reflection reads,<br/>and how code checks a URL<br/>was found before scraping it"]:::note
    logCall -.- logNote

State

Everything a research agent knows during one run. The three agents share this shape and differ only in their input and their draft.

typescript
// Sketch: ground_truth/src/core/research/state.ts
import type { BaseMessage } from '@langchain/core/messages';

interface ResearchState<Input, Draft> {
	// The agent's input. Set once at the start of the run.
	input: Input;
	// The questions this run must answer. Replaced as a whole, only by code:
	// Call Tool and Proof of Work. Saved after every change, so a run that dies can resume.
	checklist: Checklist;
	// Every chunk read this run, by chunk id. Quotes can only come from these.
	// New chunks are added, old ones kept.
	chunks: Record<string, Chunk>;
	// The evidence trail: every tool call, in order. Appended.
	// Code also uses it to check that a URL was found during the run before scraping it.
	trail: TrailEntry[];
	// The conversation with the model. Appended, never rewritten.
	messages: BaseMessage[];
	// The proposed answer, once every checklist item is closed
	draft: Draft | null;
	// How the run ended, once it has
	outcome: Outcome | null;
}

interface Checklist {
	// Which research agent this checklist belongs to
	agent: 'module' | 'licence' | 'tier';
	// What every question is about, e.g. the suggested module name "Quote Management".
	// Set when the checklist is created, and never changed.
	subject: string;
	items: ChecklistItem[];
	// URLs and searches the model queued to try later
	leads: string[];
	// What is left: searches and scrapes only
	budget: { toolCalls: number; pages: number };
}

interface ChecklistItem {
	// e.g. "is-workspace", "vendor-name:Salesforce CPQ"
	id: string;
	question: string;
	// seeded by code at the start, or added by Next Action while reading
	origin: 'seeded' | 'added';
	// Only proof of work closes an item
	status: 'open' | 'closed' | 'closed-negative' | 'reopened';
	// Set only when a quote passes proof of work
	answer?: string;
	// The quotes that closed the item, each on a chunk read this run
	evidence: Evidence[];
	// The model's working guess. Never evidence, and never shown to reflection.
	note?: string;
	// Why the item was reopened
	failure?: string;
	// Tool calls spent on this item
	tried: string[];
}

interface Evidence {
	chunkId: string;
	url: string;
	quote: string;
}

interface Chunk {
	// The page's final URL, after redirects
	url: string;
	// The headings above this piece of the page, kept apart from the text
	headings: string;
	// About 800 characters of the cleaned page
	text: string;
	// Links inside this piece, which the agent may follow
	links: string[];
}

// A search result is a snippet from the search index, never evidence
interface SearchResult {
	url: string;
	title: string;
	description: string;
	// Rank in the search results
	position: number;
}

type TrailEntry =
	// Site searches keep their results in full
	| { tool: 'domainSearch'; args: { query: string }; result: SearchResult[] }
	// Scrapes and stored-docs searches return chunks, so they keep only chunk ids:
	// the text is already in chunks
	| { tool: 'scrapeUrl'; args: { url: string }; result: { chunkIds: string[] } }
	| {
			tool: 'docSearch';
			args: { query: string };
			result: { chunkIds: string[] };
	  };

// confirmed: the draft is saved.
// rejected: not a module (Module Researcher) or not nested (Tier Researcher).
// unresolved: the budget ran out, and the open items are saved for the next run.
type Outcome = 'confirmed' | 'rejected' | 'unresolved';

// Module Researcher
interface Suggestion {
	// The name step 1 suggested for a cluster of page visits
	name: string;
	// Page visits from that cluster, as title:::path
	examples: string[];
}

interface ModuleDraft {
	// The vendor's own name for the module
	title: string;
	// Written from the closed items' answers, and nothing else
	description: string;
	// The pages behind the closing quotes, built by code
	citations: string[];
	// Other names it is known by, such as the name traffic gave it
	merged_names: string[];
}

type ModuleResearchState = ResearchState<Suggestion, ModuleDraft>;

// Licence Researcher
interface LicenceFacts {
	// Sold as a licence of its own
	standalone: boolean;
	// The licences it comes inside
	includedIn: string[];
}

type LicenceResearchState = ResearchState<ModuleDraft, LicenceFacts>;

// Tier Researcher
interface BundleGroup {
	vendor: string;
	// Each bundle, with the modules it includes
	bundles: { name: string; modules: string[] }[];
}

interface TierOrder {
	// Each pair where one bundle includes everything in the other
	pairs: { superset: string; subset: string }[];
}

type TierResearchState = ResearchState<BundleGroup, TierOrder>;

The quotes behind each answer stay on the checklist items that hold them, so the drafts carry no evidence of their own.

Updating the state

Step 4 of Call Tool is one shared function, not a tool. It runs after every tool call and turns the result into a state update, so no result goes unrecorded.

typescript
// Sketch: ground_truth/src/core/research/updateState.ts

// What each tool returns. No tool touches the state itself.
type ToolResult =
	// Domain Searcher
	| { kind: 'search'; results: SearchResult[] }
	// URL Scraper and Vendor Documents Searcher
	| { kind: 'chunks'; chunks: (Chunk & { id: string })[] }
	// Checklist Updater, after the code checks
	| { kind: 'checklist'; checklist: Checklist };

// Returns only the fields that change. LangGraph merges them in:
// chunks are added, the trail is appended to, and the checklist is replaced as a whole.
function updateState(
	call: { tool: string; args: unknown },
	result: ToolResult
): Partial<ResearchState<unknown, unknown>> {
	switch (result.kind) {
		case 'search':
			return {
				trail: [{ tool: call.tool, args: call.args, result: result.results }],
			};
		case 'chunks':
			return {
				chunks: Object.fromEntries(
					result.chunks.map(({ id, ...chunk }) => [id, chunk])
				),
				trail: [
					{
						tool: call.tool,
						args: call.args,
						result: { chunkIds: result.chunks.map(chunk => chunk.id) },
					},
				],
			};
		case 'checklist':
			return { checklist: result.checklist };
	}
}

Keeping this out of the tools has two benefits:

  • The tools stay pure. Each one only searches, scrapes or applies checklist operations, so each can be tested on its own, without a graph.

  • One place writes the state. The rule that every call is recorded lives in one function with one set of tests, so a new tool can't forget to record itself.

Who sees what

Each LLM step gets its own view of the checklist, built by code. No step sees the whole object.

Step
Sees
Why

Next Action

Every item: closed ones trimmed to id and quote, open ones with their note, failure and the calls tried; the leads; the budget left

Picks the item to work on and the tool call most likely to close it

Draft

Closed items only, with their question, answer and evidence

The description can't use unchecked guesses

Reflection

Each item's question, answer and quotes. No notes, leads, failures or calls tried

Keeps the reviewer independent of the researcher's reasoning

Prompts show quotes by chunk id, with the chunk text under its own heading outside the JSON. Quotes inside JSON get escaped (\", \n), and a model that copies the escaped form fails the verbatim check.

3. Module Researcher

Checks that a new name is really a module in the vendor's product, and writes its title and description from the vendor's own pages. It is the most autonomous agent: it chooses its own searches and decides which links to follow. Only modules it confirms are saved.

In: a new suggested name and its examples. Out: a confirmed module with title, description and citation URLs, or "not a module". Model: Opus.

Research Loop

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    suggestions[/Suggested Module Names/]:::green
    researchLoop[Research Loop]:::blue
    confirmedModule[/Confirmed Module/]:::green
    notModule[/Not a Module/]:::red
    unresolved[/Unresolved,\nretried next run/]:::red

    suggestions --> |name + examples| researchLoop
    researchLoop --> |confirmed| confirmedModule
    researchLoop --> |rejected| notModule
    researchLoop --> |unresolved| unresolved

    checklistNote["Checklist: exists, is-workspace,<br/>what-it-does, matches-traffic"]:::note
    researchLoop -.- checklistNote

    moduleNote["title, description,<br/>citations, merged_names"]:::note
    confirmedModule -.- moduleNote

Checklist questions

Code seeds four items from the suggested name and its traffic examples, for example Quote Management and Quote Management:::/lightning/o/Quote/list.

Item
Question
Closed when (code check)
Rubric (doubtful goes to reflection)

exists

Does the vendor name this thing?

A quote on a chunk scraped this run contains the suggested name or a close variant

None

is-workspace

Is it somewhere users work, not a setting, API, report type or marketing term?

The quote passes the evidence check and names the module

Do the quotes describe something users work in?

what-it-does

What do users do in it?

At least one quote states its purpose, so the description stands on quotes

Is the description supported by the quotes alone?

matches-traffic

Is this the same thing the traffic shows?

The cited page mentions a term from the example paths or titles, such as Quote from /o/Quote/

Do the examples fit this description?

The matches-traffic item stops the agent from confirming a real module that isn't the one people used. For example, "Quotes" can lead to the Salesforce CPQ docs while the traffic is the standard Quote object. Its code check is weak, because path terms like Quote are generic, so it only decides whether the item goes to reflection.

Next Action can open three kinds of item as it reads:

  • vendor-name: the vendor calls it something else, such as "Salesforce CPQ" for "Quote Management". The confirmed module takes the vendor's title, and the suggested name goes into merged_names. The Licence Researcher then searches by the vendor's name.

  • part-of: a page says it is a feature inside another module. If so, the result is "not a module", with a merge candidate for the other module, instead of a silent drop.

  • ambiguous: the name matches two different products. Each candidate gets its own item, closed by the matches-traffic check.

The research ends in one of three ways:

  • Confirmed Module: items exists to matches-traffic are closed, along with any items Next Action opened. Code builds the citations from the closing quotes.

  • Not a Module: a quote closes is-workspace or part-of negatively, for example "a setting in Setup". A rejection needs a quote too, or a wrong one drops a real module without trace.

  • Unresolved: the budget runs out with items still open. The open items are saved for the next run.

Proof of Work

Every quote must be on a chunk the agent read in this run, from a URL it actually scraped. A failure goes back to research with the reason.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    draft[/From Research Loop:\ntitle, description,\nquotes, cited URLs/]:::next

    subgraph proofOfWorkCheck[5. Proof of Work]
        quotesOnChunks{"Quotes are on\nthe cited chunks?"}
        urlsScraped{"Cited URLs were\nscraped this run?"}
        usersWorkIn{"Rubric:\nDo the quotes describe\nsomething users work in?"}:::rubric
    end

    backToResearch[/Next Action,\nwith the reason/]:::next
    toReflection[/Reflection/]:::next
    confirmedModule[/Confirmed Module/]:::green

    draft --> quotesOnChunks
    quotesOnChunks --> |no| backToResearch
    quotesOnChunks --> |yes| urlsScraped
    urlsScraped --> |no| backToResearch
    urlsScraped --> |yes| usersWorkIn
    usersWorkIn --> |yes| confirmedModule
    usersWorkIn --> |no| toReflection

    quotesNote["Each quote found word for word<br/>in a chunk the agent read"]:::note
    quotesOnChunks -.- quotesNote

    urlsNote["A URL not scraped this run<br/>means it was made up"]:::note
    urlsScraped -.- urlsNote

    rubricNote["Opus answers with the draft<br/>Self-graded, not checked"]:::note
    usersWorkIn -.- rubricNote

    backNote["Retries count against<br/>the loop's budget"]:::note
    backToResearch -.- backNote

Reflection

Opus reviews the draft with a fresh context. It can confirm it, reject it, or send it back for the evidence that is missing.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Tables
        modules[(Modules)]:::orange
    end

    subgraph Models
        claudeOpus((Claude Opus))
    end

    flagged[/From Proof of Work:\ndraft + evidence trail/]:::next
    passed[/From Proof of Work:\nConfirmed Module/]:::next

    subgraph reflection[6. Reflection]
        reviewModule{"Is this a module?"}:::orange
        notModule[/Not a Module/]:::red
    end

    confirmedModule[/Confirmed Module\ntitle, description,\ncitation URLs/]:::green
    backToResearch[/Next Action,\nwith what is missing/]:::next

    flagged --> reviewModule
    claudeOpus --> reviewModule
    reviewModule --> |yes| confirmedModule
    reviewModule --> |no| notModule
    reviewModule --> |needs more\nevidence| backToResearch
    passed --> confirmedModule
    confirmedModule --> modules

    reflectionNote["Opus, fresh context<br/>Sees the evidence trail,<br/>not the research reasoning"]:::note
    reviewModule -.- reflectionNote

    backNote["Opus says what is missing,<br/>not which tool to call<br/>Same budget: out of budget<br/>ends as Unresolved"]:::note
    backToResearch -.- backNote

    confirmedNote["Only confirmed modules<br/>are saved"]:::note
    confirmedModule -.- confirmedNote

4. Licence Researcher

For each confirmed module, finds out whether it is sold as a licence of its own, which licences it comes inside, or both. It uses the shared research loop.

In: a confirmed module. Out: licence facts, each with its quote. Model: Opus.

Research Loop

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    confirmedModules[/Confirmed Modules/]:::green
    researchLoop[Research Loop]:::blue
    licenceFacts[/Licence Facts/]:::green
    unresolved[/Unresolved,\nretried next run/]:::red

    confirmedModules --> |name, description,\ncitations| researchLoop
    researchLoop --> |confirmed| licenceFacts
    researchLoop --> |unresolved| unresolved

    checklistNote["Checklist: is it sold on its own?<br/>Plus one question per licence<br/>it may come inside<br/>Can be both"]:::note
    researchLoop -.- checklistNote

    factsNote["standalone: yes or no<br/>included in: licence names<br/>each with its quote"]:::note
    licenceFacts -.- factsNote

Proof of Work

The checks follow the claim. "Sold on its own" needs a quote that gives it its own price, plan or SKU. "Comes inside" needs a quote that names every licence it comes inside. Either way, a module only mentioned in passing goes to review.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    draft[/From Research Loop:\nclaim, quotes,\ncited URLs/]:::next

    subgraph proofOfWorkCheck[5. Proof of Work]
        quotesOnChunks{"Quotes are on\nthe cited chunks?"}
        urlsScraped{"Cited URLs were\nscraped this run?"}
        namesModule{"Quote names\nthe module?"}
        claimType{"Which claim?"}
        namesLicence{"Quote names every\nlicence it comes inside?"}
        ownPrice{"Rubric:\nGives it its own\nprice, plan or SKU?"}:::rubric
        onlyInside{"Rubric:\nSays it only comes\ninside other licences?"}:::rubric
        inPassing{"Rubric:\nOnly listed\nin passing?"}:::rubric
    end

    backToNextAction[/Next Action,\nwith the reason/]:::next
    toReflection[/Reflection/]:::next
    passes[/Licence Facts/]:::green

    draft --> quotesOnChunks
    quotesOnChunks --> |no| backToNextAction
    quotesOnChunks --> |yes| urlsScraped
    urlsScraped --> |no| backToNextAction
    urlsScraped --> |yes| namesModule
    namesModule --> |no| backToNextAction
    namesModule --> |yes| claimType
    claimType --> |sold on its own| ownPrice
    claimType --> |comes inside| namesLicence
    namesLicence --> |no| backToNextAction
    namesLicence --> |yes| onlyInside
    ownPrice --> |yes| inPassing
    ownPrice --> |no| toReflection
    onlyInside --> |yes| inPassing
    onlyInside --> |no| toReflection
    inPassing --> |no| passes
    inPassing --> |yes| toReflection

    namesModuleNote["The module's name<br/>or one of its aliases"]:::note
    namesModule -.- namesModuleNote

    ownPriceNote["e.g. its own row<br/>in a pricing table"]:::note
    ownPrice -.- ownPriceNote

    onlyInsideNote["e.g. #quot;included in every<br/>Enterprise plan#quot;"]:::note
    onlyInside -.- onlyInsideNote

    inPassingNote["A feature grid or a<br/>#quot;works with#quot; list"]:::note
    inPassing -.- inPassingNote

Reflection

A rejected claim is not final here: every module is licensed somehow, so a "no" sends the researcher back for better evidence.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Tables
        moduleLicences[(Module Licences)]:::orange
    end

    subgraph Models
        claudeOpus((Claude Opus))
    end

    flagged[/From Proof of Work:\nclaim + evidence trail/]:::next
    passed[/From Proof of Work:\nLicence Facts/]:::next

    subgraph reflection[6. Reflection]
        reviewClaim{"Does the evidence\nsupport the claim?"}:::orange
    end

    backToNextAction[/Next Action,\nwith what is missing/]:::next
    licenceFacts[/Licence Facts/]:::green

    flagged --> reviewClaim
    claudeOpus --> reviewClaim
    reviewClaim --> |yes| licenceFacts
    reviewClaim --> |no, or needs\nmore evidence| backToNextAction
    passed --> licenceFacts
    licenceFacts --> moduleLicences

    reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
    reviewClaim -.- reflectionNote

    backNote["A rejected claim is not final:<br/>every module is licensed somehow<br/>Out of budget ends as Unresolved"]:::note
    backToNextAction -.- backNote

6. Tier Researcher

For one vendor's group of bundles, works out which includes everything in which, so a user can be moved to the lowest tier that covers what they do. It uses the shared research loop.

In: a bundle group. Out: tier order, each pair with its quote, or "not nested". Model: Opus.

Research Loop

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    bundleGroup[/Bundle Group\nfrom Bundle Finder/]:::next
    researchLoop[Research Loop]:::blue
    tierOrder[/Tier Order/]:::green
    notNested[/Not Nested/]:::red
    unresolved[/Unresolved,\nretried next run/]:::red

    bundleGroup --> |bundle names, the\nmodules each includes| researchLoop
    researchLoop --> |confirmed| tierOrder
    researchLoop --> |rejected| notNested
    researchLoop --> |unresolved| unresolved

    checklistNote["Checklist: one question<br/>per pair of bundles:<br/>does A include everything in B?"]:::note
    researchLoop -.- checklistNote

    tierOrderNote["Each pair with its quote<br/>Lets a user drop to the<br/>lowest tier they need"]:::note
    tierOrder -.- tierOrderNote

Proof of Work

As well as the usual quote checks, code rejects an order with cycles: if A includes B and B includes A, one of the answers is wrong. Price is deliberately not evidence, because a dearer plan doesn't have to include a cheaper one.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    draft[/From Research Loop:\npairs, quotes,\ncited URLs/]:::next

    subgraph proofOfWorkCheck[5. Proof of Work]
        quotesOnChunks{"Quotes are on\nthe cited chunks?"}
        urlsScraped{"Cited URLs were\nscraped this run?"}
        namesBoth{"Quote names\nboth bundles?"}
        noCycles{"Order has\nno cycles?"}
        statesSuperset{"Rubric:\nSays one has everything\nin the other?"}:::rubric
        aboutThesePlans{"Rubric:\nAbout these bundles,\nnot plans in general?"}:::rubric
        comparisonTable{"Rubric:\nOnly a comparison\ntable cell?"}:::rubric
    end

    backToNextAction[/Next Action,\nwith the reason/]:::next
    toReflection[/Reflection/]:::next
    passes[/Tier Order/]:::green

    draft --> quotesOnChunks
    quotesOnChunks --> |no| backToNextAction
    quotesOnChunks --> |yes| urlsScraped
    urlsScraped --> |no| backToNextAction
    urlsScraped --> |yes| namesBoth
    namesBoth --> |no| backToNextAction
    namesBoth --> |yes| noCycles
    noCycles --> |no| backToNextAction
    noCycles --> |yes| statesSuperset
    statesSuperset --> |no| toReflection
    statesSuperset --> |yes| aboutThesePlans
    aboutThesePlans --> |no| toReflection
    aboutThesePlans --> |yes| comparisonTable
    comparisonTable --> |yes| toReflection
    comparisonTable --> |no| passes

    noCyclesNote["A includes B and B includes A<br/>means one answer is wrong"]:::note
    noCycles -.- noCyclesNote

    supersetNote["e.g. #quot;everything in<br/>Professional, plus#quot;"]:::note
    statesSuperset -.- supersetNote

    tableNote["Its meaning is in the<br/>table's header, not the cell"]:::note
    comparisonTable -.- tableNote

Reflection

"Not nested" is a valid final answer here: some bundles are separate products that sit side by side.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef orange fill:#ffcc99,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef rubric fill:#e0f2fe,stroke:#0369a1,stroke-width:2px,stroke-dasharray: 4 2,color:#000000;
    classDef next fill:#ffffff,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3,color:#000000;

    subgraph Tables
        bundleTiers[(Bundle Tiers)]:::orange
    end

    subgraph Models
        claudeOpus((Claude Opus))
    end

    flagged[/From Proof of Work:\npairs + evidence trail/]:::next
    passed[/From Proof of Work:\nTier Order/]:::next

    subgraph reflection[6. Reflection]
        reviewPair{"Does one bundle include\neverything in the other?"}:::orange
    end

    backToNextAction[/Next Action,\nwith what is missing/]:::next
    tierOrder[/Tier Order/]:::green
    notNested[/Not Nested/]:::red

    flagged --> reviewPair
    claudeOpus --> reviewPair
    reviewPair --> |yes| tierOrder
    reviewPair --> |no, separate products| notNested
    reviewPair --> |needs more evidence| backToNextAction
    passed --> tierOrder
    tierOrder --> bundleTiers

    reflectionNote["Opus, fresh context<br/>Sees the quotes in place<br/>on their pages"]:::note
    reviewPair -.- reflectionNote

    notNestedNote["A final answer: some bundles<br/>sit side by side, e.g. two<br/>separate product clouds"]:::note
    notNested -.- notNestedNote

How we will measure it

  • One app with known ground truth, end to end. Salesforce is the likely first choice.

  • Side by side with the current pipeline: accuracy against the ground truth, pages scraped, and cost.

  • Component evals: clusters that are already tagged are ground truth for step 1, and saved merges are ground truth for step 2.

  • How often reflection overturns a draft. If it almost never does, the rubric and reflection steps can be cut to save cost.

Open questions

  • Planning. The researchers choose one step at a time. Adding a short up-front plan of the questions to answer would make "enough evidence" something code can check.

  • Embedding new pages. The URL Scraper doesn't embed, so pages scraped during a run aren't searchable in Vendor Documentation Embeddings until something embeds them.

  • What documentation can't show. Customer-defined roles and usage-metered products don't appear in vendor docs. They need the vendor's own admin or usage APIs.

  • Shelfware. A licence that is paid for but never used produces no traffic, so nothing triggers research on it. Finding it needs the customer's entitlement data.

  • Numbers to tune. The research budget (15 tool calls, 10 pages) is a placeholder. The merge thresholds (0.95 and 0.6), the 8 pages per module and the 3 candidates per name come from the PoC.

  • Cost. Opus drives every research loop, so the budget matters more than it would with Sonnet.

Tool Reference

No title

The agents reach the outside world through two Firecrawl tools. Both check the cache before spending credits, and both fill it afterwards. A third tool, the Checklist Updater, works on the agent's own checklist.

Domain Searcher

Finds vendor pages for a keyword, limited to the vendor's own domains. Only real answers are cached: a call that still fails after its retries caches nothing.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;

    subgraph Cache
        firecrawlSearchCache[(Firecrawl Domain\nSearch Cache)]:::yellow
    end

    request[/query + domains + limit/]
    buildKey[1. Build Cache Key]
    inCache{"2. In cache?"}
    firecrawlSearch[3. Firecrawl Search\nweb results on\nthese domains only]:::blue
    gotAnswer{"Answer after\nretries?"}
    writeCache[4. Write to Cache]
    cleanResults[5. Clean Results]
    results[/Pages + metadata/]:::green
    failed[/Error, nothing cached/]:::red

    request --> buildKey --> inCache
    firecrawlSearchCache --> inCache
    inCache --> |yes| cleanResults
    inCache --> |no| firecrawlSearch --> gotAnswer
    gotAnswer --> |yes| writeCache --> cleanResults
    gotAnswer --> |no| failed
    writeCache --> firecrawlSearchCache
    cleanResults --> results

    keyNote["search:domains:query:limit<br/>Domains sorted, so order<br/>doesn't change the key"]:::note
    buildKey -.- keyNote

    refreshNote["refresh: skips the read,<br/>still writes the new answer"]:::note
    inCache -.- refreshNote

    retryNote["Rate-limited and retried<br/>with backoff"]:::note
    firecrawlSearch -.- retryNote

    cleanNote["Conservative: only removes<br/>duplicate and off-domain URLs"]:::note
    cleanResults -.- cleanNote

    resultsNote["url, title, description,<br/>position (search rank),<br/>category"]:::note
    results -.- resultsNote

URL Scraper

Fetches one page and returns it as chunks. It caches Firecrawl's raw answer before cleaning, so the cleaning and chunking rules can change without paying for the page again. Cleaning is conservative: it removes images and cookie and feedback widgets, and keeps every link so an agent can follow it.

mermaid
flowchart LR
    classDef blue fill:#ccccff,stroke:#000,stroke-width:2px,color:#000000;
    classDef green fill:#ccffcc,stroke:#000,stroke-width:2px,color:#000000;
    classDef red fill:#fecaca,stroke:#000,stroke-width:2px,color:#000000;
    classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,stroke-dasharray: 5 5,color:#333;
    classDef yellow fill:#ffff99,stroke:#000,stroke-width:2px,color:#000000;

    subgraph Cache
        firecrawlURLScrapeCache[(Firecrawl URL\nScrape Cache)]:::yellow
    end

    request[/url/]
    inCache{"1. In cache?"}
    firecrawlScrape[2. Firecrawl Scrape\nmain content\nas Markdown]:::blue
    gotMarkdown{"Markdown after\nretries?"}
    writeCache[3. Write Raw\nMarkdown to Cache]
    skipped[/Skipped, nothing cached/]:::red

    subgraph cleanUp[4. Clean Up]
        usablePage{"Dead, empty or\na sign-up form?"}
        cleanMarkdown[Strip page chrome]
        droppedPage[/Page dropped/]:::red
    end

    chunkPage[5. Chunk Page]
    page[/Page chunks/]:::green

    request --> inCache
    firecrawlURLScrapeCache --> inCache
    inCache --> |yes| usablePage
    inCache --> |no| firecrawlScrape --> gotMarkdown
    gotMarkdown --> |yes| writeCache --> usablePage
    gotMarkdown --> |no| skipped
    writeCache --> firecrawlURLScrapeCache
    usablePage --> |yes| droppedPage
    usablePage --> |no| cleanMarkdown --> chunkPage --> page

    scrapeNote["Main content only:<br/>no nav, header, footer"]:::note
    firecrawlScrape -.- scrapeNote

    rawNote["Raw, not cleaned:<br/>re-clean for free"]:::note
    writeCache -.- rawNote

    usableNote["Sign-up form: 6+ form-field<br/>lines, no prices"]:::note
    usablePage -.- usableNote

    cleanNote["Removes: images, cookie<br/>and feedback widgets<br/>Keeps all links, so an agent<br/>can follow them"]:::note
    cleanMarkdown -.- cleanNote

    chunkNote["800 chars, 150 overlap<br/>Each starts with its heading trail<br/>Under 40 chars dropped"]:::note
    chunkPage -.- chunkNote

    pageNote["Per chunk: final URL,<br/>chunk index, text, links"]:::note
    page -.- pageNote

Three smaller tools read the stores: the Untagged Clusters Retriever (Page Visit Embeddings), Hybrid Module Search (Modules) and the Vendor Documents Searcher (Vendor Documentation Embeddings).

Checklist Updater

The agent changes its checklist through one tool, update_checklist, with typed operations. Unlike the Firecrawl tools, it never leaves the run: it changes only the checklist in the agent's state. The agent can call it in the same turn as a search or a scrape, so bookkeeping costs no extra round trip.

Operation
The model can
Code then

add_item

Open a new item it found while reading

Rejects duplicates, sets origin: 'added', caps the number of items

set_note

Record a working guess for an item

Stores it, and never treats it as evidence

add_lead

Queue a URL or a search to try later

Applies the URL rules: vendor domain only, not scraped this run, and found in a search result or on a chunk

drop_lead

Drop a lead it no longer needs

Removes it

propose_answer

Claim an item is answered, with quotes

Runs proof of work at once, then closes the item or returns the reason it failed

propose_negative

Claim an item is answered no, with quotes

The same, and closes it as closed-negative

The model can't set a status, edit the evidence of a closed item, delete a seeded item or change the budget. Only proof of work and reflection can reopen an item.

Running proof of work inside propose_answer gives feedback straight away: a paraphrased quote fails on the turn it was written, while its chunk is still in context. By the time every item is closed, the answer already exists, so the draft only has to assemble it.

update_checklist doesn't count against the budget of 15 tool calls; only searches and scrapes do. Otherwise the model would learn to skip bookkeeping to save budget.

The conversation is append-only. Code never rewrites earlier turns to show a fresh checklist. On Claude Opus 5.5, editing earlier turns invalidates the model's thinking blocks and breaks prompt caching. Instead, every update_checklist result carries the current Next Action view, so the latest checklist is always the latest tool result.

Technology

Ground Truth is a new TypeScript project in the ground_truth directory, next to the PoC's license_optimiser. It is built on LangGraph and LangChain. The Module Researcher comes first, because every later agent starts from the modules it confirms.

Stack

Package
Version checked
Used for

@langchain/langgraph

1.7.2

State, graph, subgraphs, Send fan-out

@langchain/core

1.3.5

Messages and tools

@langchain/aws

1.5.2

ChatBedrockConverse, Claude on Bedrock

@langchain/langgraph-checkpoint-postgres

1.1.2

Saving state for resume

@mendable/firecrawl-js

4.52.0

Domain Searcher and URL Scraper

Versions were checked on npm on 2026-11-07 and are pinned, because these packages must match each other.

The design maps onto LangGraph like this:

Design
LangGraph

Checklist in memory

A field in the graph state, defined with Annotation.Root

Research loop

Nodes and conditional edges

"Every item closed or out of budget"

A conditional edge after each tool call

Page budget

nextAction binds the URL Scraper only while pages are left

Independent reviewer

Reflection is a subgraph with its own smaller state, so it can't see notes

Save and resume

Postgres checkpointer, one thread_id per module

Whole pipeline

A parent graph, with one Module Researcher per new name through Send

The checks that make the answers trustworthy stay plain TypeScript with their own unit tests: the checklist operations, the verbatim quote check and the URL rules. LangGraph only calls them.

Check three things before building on them:

  • The model wrapper. The ChatBedrockConverse docs don't mention thinking or effort.
    Confirm that Opus 5.5 on Bedrock gets tool calling, effort and adaptive thinking through it, and that thinking blocks survive a tool round trip.
    If not, write a small LangChain chat model over the Anthropic Bedrock SDK.

  • LangSmith and data policy. Traces would send page titles, URLs and vendor text to LangSmith's service. Check what Nexthink allows; LangSmith can be self-hosted, and tracing can be turned off.

  • API names. Check Annotation.Root, MessagesAnnotation, Send and PostgresSaver against the installed versions.

LangGraph nodes

Each step of the shared research loop is a LangGraph node or edge. The graph runs on the shared research state.

Research loop step
LangGraph

Seed Checklist

seed node

Checklist? and Budget left?

Conditional edges after callTools

Next Action

nextAction node, with the URL Scraper bound only while pages are left

Call Tool

callTools node, custom

Describe or Decide

draft node

Proof of Work

proofOfWork node

Reflection

A subgraph with its own smaller state

Why callTools is custom

LangGraph has a prebuilt ToolNode, where tools can return a Command to update state. It doesn't fit here, because propose_answer must run proof of work against chunks, a different part of the state. Instead, one custom callTools node handles every tool call in a turn:

  1. Counts it against the budget, except update_checklist.

  2. Applies the URL rules to scrapes and links: vendor domain only, not scraped this run, and found during the run, in a search result or as a link on a chunk.

  3. Runs the tool, which checks its cache before Firecrawl.

  4. Adds new chunks to chunks, and logs the call to trail with its search results or chunk ids.

  5. Applies checklist operations, running proof of work for propose_answer and propose_negative.

  6. Returns one tool message per call, with the result and the current Next Action view.

Only this node and proof of work change the checklist, so "only code closes an item" holds by the shape of the graph.

Project layout

text
ground_truth/
  src/
    core/                          plain TypeScript, no LangChain, unit-tested
      checklist/types.ts           the checklist object
      checklist/ops.ts             add_item, set_note, propose_answer, ...
      checklist/views.ts           the Next Action, Draft and Reflection views
      checklist/seeds/module.ts    the four seeded items
      evidence/                    text normalisation and the verbatim quote check
      urlRules.ts
      budget.ts
    tools/                         LangChain tool wrappers: schemas and descriptions only
      domainSearch.ts
      scrapeUrl.ts
      docSearch.ts
      updateChecklist.ts
    cache/firecrawlCache.ts        raw Firecrawl answers on disk
    llm/models.ts                  the Bedrock chat model
    agents/moduleResearcher/
      state.ts
      graph.ts
      prompts.ts
      nodes/                       seed, nextAction, callTools, draft, proofOfWork
      reflection/graph.ts          the reflection subgraph
    cli/researchModule.ts
  tests/

Two pieces are copied from license_optimiser, not imported: the evidence text normalisation, with its fixes for <br> in table cells, backticks and callout labels, and the Firecrawl cache. The TypeScript, eslint and vitest configs are copied too, so the two projects match.

Build order

Each step works and can be shown before the next one starts.

  1. Model wrapper test. One script calls Opus 5.5 on Bedrock through ChatBedrockConverse and checks tool calling, effort, adaptive thinking and thinking blocks across a tool round trip. This settles the stack before anything is built on it.

  2. Core, no LLM. Checklist operations, views, the quote check and the URL rules, with unit tests. This is where correctness lives.

  3. Tools. Domain Searcher and URL Scraper behind the cache. The Vendor Documents Searcher can start as a stub, because pages scraped during a run aren't embedded yet.

  4. Graph with a fake model. LangChain's fake chat models replay scripted tool calls, so every route can be tested without paying for Opus: every item closed, budget spent, a reopened item, a negative close, reflection.

  5. Real runs on Salesforce. Five known modules and a few known non-modules. Read the evidence trails, then tune the prompts.

  6. Reflection subgraph, Postgres checkpointer and CLI.

  7. First eval. Clusters already tagged in the PoC should come out confirmed, and known non-modules should come out rejected. Track tool calls per module and the Unresolved rate: this is the baseline for checking that the checklist helps.