Pillars of AI governance.
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
flowchart TB
classDef teal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#000000;
subgraph prepareData ["Prepare data"]
direction TB
vdb["VectorDB"]
vm["View/Masks"]
tbl["Tables"]
end
subgraph buildAgents ["Build agents"]
direction TB
llm["LLM Selection"]
prompt["Prompt Eng."]
tools["Build/Bind Tools"]
end
subgraph evalAgents ["Evaluate agents"]
direction TB
judges["LLM judges"]
peer["Peer labeling"]
trace["Tracing"]
end
subgraph deployAgents ["Deploy agents"]
direction TB
serving["Agent serving"]
mlops["MLOps/LLMOps"]
lineage["Lineage"]
end
subgraph governAgents ["Govern agents (Unity Catalog + MLflow)"]
direction LR
lm["Lifecycle Management"]:::teal
rm["Risk Management"]:::teal
sec["Security"]:::teal
obs["Observability"]:::teal
end
prepareData --> buildAgents --> evalAgents --> deployAgents
deployAgents --> governAgents --> prepareDataLifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Unity Catalog
flowchart TD
classDef dashed fill:#f8fafc,stroke:#94a3b8,stroke-width:1.5px,stroke-dasharray: 4 4,color:#475569;
metastore["Metastore"]
%% Storage & Credentials (Metastore-level)
serviceCred["Service<br/>credential"]:::dashed
storageCred["Storage<br/>credential"]:::dashed
extLoc["External<br/>location"]:::dashed
extMeta["External<br/>metadata"]:::dashed
%% Core 3-Level Namespace
catalog["Catalog"]
schema[("Schema")]
table["Table"]
view["View"]
volume["Volume\n*unstructured storage"]
function["Function<br/>including models"]
%% Sharing & Collaboration
share["Share"]:::dashed
recipient["Recipient"]:::dashed
provider["Provider"]:::dashed
connection["Connection"]:::dashed
cleanRoom["Clean Room"]:::dashed
metastore -.-> serviceCred & storageCred & extLoc & extMeta
metastore --> catalog
metastore -.-> share & recipient & provider & connection & cleanRoom
catalog --> schema
schema --> table & view & volume & functionIdentity
flowchart TD
classDef acct fill:#ffffff,stroke:#1e293b,stroke-width:2px,color:#0f172a;
classDef item fill:#ffffff,stroke:#64748b,stroke-width:1.5px,color:#0f172a;
classDef grpBlue fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef grpRed fill:#fee2e2,stroke:#dc2626,stroke-width:1.5px,color:#7f1d1d;
classDef meta fill:#f0fdf4,stroke:#16a34a,stroke-width:1.5px,color:#14532d;
subgraph account ["Account"]
direction LR
subgraph acctAdmin ["Account Management"]
direction TB
billing["Billing and Usage"]:::item
acctAdmins["Account Administrators"]:::item
end
subgraph acctIdentities ["Users, Service Principals, Groups"]
direction TB
subgraph groupsRow [" "]
direction LR
g1["Group 1"]:::grpBlue
g2["Group 2"]:::grpBlue
g3["Group 3"]:::grpRed
end
end
end
subgraph deployments [" "]
direction LR
subgraph metastoreBox ["Metastore"]
metastore["Data & AI Assets<br/>(Catalogs, Schemas, Tables)"]:::meta
end
subgraph ws1 ["Workspace"]
ws1_g2["Group 2"]:::grpBlue
end
subgraph ws2 ["Workspace"]
direction LR
ws2_g1["Group 1"]:::grpBlue
ws2_g3["Group 3"]:::grpRed
end
end
account --> metastoreBox
account --> ws1
account --> ws2
g2 -.-> ws1_g2
g1 -.-> ws2_g1
g3 -.-> ws2_g3
metastoreBox -.- ws1
metastoreBox -.- ws2Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
flowchart LR
classDef user fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef agent fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f;
classDef sp fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a;
classDef res fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a;
subgraph autoMode ["Automatic Passthrough (System Auth)"]
direction TB
caller1["Caller / User"]:::user --> agent1["AI Agent"]:::agent
agent1 -->|"Agent's SP<br/>(Same identity)"| sp1["Agent Service Principal"]:::sp
sp1 --> res1[("Declared Resources<br/>& Tools")]:::res
end
subgraph oboMode ["On-Behalf-of-User (OBO)"]
direction TB
caller2["Caller / User"]:::user -->|"Caller token & scopes"| agent2["AI Agent"]:::agent
agent2 -->|"Caller's identity<br/>(User-level access)"| res2[("Sensitive User Data")]:::res
end
subgraph manualMode ["Manual Authentication"]
direction TB
secAdmin["Security Admin"]:::user -->|"Pre-provisions"| sp3["Central SP"]:::sp
caller3["Caller / User"]:::user --> agent3["AI Agent"]:::agent
agent3 -->|"Pre-configured credentials"| sp3
sp3 --> res3[("Enterprise Resources")]:::res
endAutomatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
MLflow
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
flowchart LR
classDef mlflow fill:#fff7ed,stroke:#c2410c,stroke-width:1.5px,color:#7c2d12;
classDef artifact fill:#f8fafc,stroke:#475569,stroke-width:1.5px,color:#0f172a;
classDef action fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef output fill:#f0fdf4,stroke:#16a34a,stroke-width:1.5px,color:#14532d;
subgraph experiment ["Experiment Tracking"]
direction TB
params["Prompts, Temperatures,<br/>Model Names"]:::artifact
metrics["Metrics & Output Quality"]:::artifact
code["Code Versions & Datasets"]:::artifact
end
subgraph tracing ["Tracing"]
direction TB
chain["Agent Execution Tree"]:::artifact
tools["Tool Calls & Retrieval Steps"]:::artifact
latency["Token Counts & Latency"]:::artifact
end
subgraph registry ["Model Registry"]
direction TB
staging["Staging"]:::action
production["Production"]:::action
archived["Archived"]:::action
staging --> production --> archived
end
subgraph evaluation ["Evaluation"]
direction TB
judge["LLM-as-a-Judge"]:::artifact
criteria["Relevance, Groundedness,<br/>Toxicity, Hallucination"]:::artifact
end
agent["AI Agent"]:::mlflow
agent --> experiment
agent --> tracing
agent --> evaluation
evaluation -->|"winner promoted"| registry
experiment -->|"best run registered"| registry
registry -->|"deploy to"| prod[("Production")]:::output
prod -->|"feedback traces"| tracingExperiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Pillars of AI governance.
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
flowchart TB
classDef teal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#000000;
subgraph prepareData ["Prepare data"]
direction TB
vdb["VectorDB"]
vm["View/Masks"]
tbl["Tables"]
end
subgraph buildAgents ["Build agents"]
direction TB
llm["LLM Selection"]
prompt["Prompt Eng."]
tools["Build/Bind Tools"]
end
subgraph evalAgents ["Evaluate agents"]
direction TB
judges["LLM judges"]
peer["Peer labeling"]
trace["Tracing"]
end
subgraph deployAgents ["Deploy agents"]
direction TB
serving["Agent serving"]
mlops["MLOps/LLMOps"]
lineage["Lineage"]
end
subgraph governAgents ["Govern agents (Unity Catalog + MLflow)"]
direction LR
lm["Lifecycle Management"]:::teal
rm["Risk Management"]:::teal
sec["Security"]:::teal
obs["Observability"]:::teal
end
prepareData --> buildAgents --> evalAgents --> deployAgents
deployAgents --> governAgents --> prepareDataLifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Unity Catalog
flowchart TD
classDef dashed fill:#f8fafc,stroke:#94a3b8,stroke-width:1.5px,stroke-dasharray: 4 4,color:#475569;
metastore["Metastore"]
%% Storage & Credentials (Metastore-level)
serviceCred["Service<br/>credential"]:::dashed
storageCred["Storage<br/>credential"]:::dashed
extLoc["External<br/>location"]:::dashed
extMeta["External<br/>metadata"]:::dashed
%% Core 3-Level Namespace
catalog["Catalog"]
schema[("Schema")]
table["Table"]
view["View"]
volume["Volume\n*unstructured storage"]
function["Function<br/>including models"]
%% Sharing & Collaboration
share["Share"]:::dashed
recipient["Recipient"]:::dashed
provider["Provider"]:::dashed
connection["Connection"]:::dashed
cleanRoom["Clean Room"]:::dashed
metastore -.-> serviceCred & storageCred & extLoc & extMeta
metastore --> catalog
metastore -.-> share & recipient & provider & connection & cleanRoom
catalog --> schema
schema --> table & view & volume & functionIdentity
flowchart TD
classDef acct fill:#ffffff,stroke:#1e293b,stroke-width:2px,color:#0f172a;
classDef item fill:#ffffff,stroke:#64748b,stroke-width:1.5px,color:#0f172a;
classDef grpBlue fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef grpRed fill:#fee2e2,stroke:#dc2626,stroke-width:1.5px,color:#7f1d1d;
classDef meta fill:#f0fdf4,stroke:#16a34a,stroke-width:1.5px,color:#14532d;
subgraph account ["Account"]
direction LR
subgraph acctAdmin ["Account Management"]
direction TB
billing["Billing and Usage"]:::item
acctAdmins["Account Administrators"]:::item
end
subgraph acctIdentities ["Users, Service Principals, Groups"]
direction TB
subgraph groupsRow [" "]
direction LR
g1["Group 1"]:::grpBlue
g2["Group 2"]:::grpBlue
g3["Group 3"]:::grpRed
end
end
end
subgraph deployments [" "]
direction LR
subgraph metastoreBox ["Metastore"]
metastore["Data & AI Assets<br/>(Catalogs, Schemas, Tables)"]:::meta
end
subgraph ws1 ["Workspace"]
ws1_g2["Group 2"]:::grpBlue
end
subgraph ws2 ["Workspace"]
direction LR
ws2_g1["Group 1"]:::grpBlue
ws2_g3["Group 3"]:::grpRed
end
end
account --> metastoreBox
account --> ws1
account --> ws2
g2 -.-> ws1_g2
g1 -.-> ws2_g1
g3 -.-> ws2_g3
metastoreBox -.- ws1
metastoreBox -.- ws2Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
flowchart LR
classDef user fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef agent fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f;
classDef sp fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a;
classDef res fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a;
subgraph autoMode ["Automatic Passthrough (System Auth)"]
direction TB
caller1["Caller / User"]:::user --> agent1["AI Agent"]:::agent
agent1 -->|"Agent's SP<br/>(Same identity)"| sp1["Agent Service Principal"]:::sp
sp1 --> res1[("Declared Resources<br/>& Tools")]:::res
end
subgraph oboMode ["On-Behalf-of-User (OBO)"]
direction TB
caller2["Caller / User"]:::user -->|"Caller token & scopes"| agent2["AI Agent"]:::agent
agent2 -->|"Caller's identity<br/>(User-level access)"| res2[("Sensitive User Data")]:::res
end
subgraph manualMode ["Manual Authentication"]
direction TB
secAdmin["Security Admin"]:::user -->|"Pre-provisions"| sp3["Central SP"]:::sp
caller3["Caller / User"]:::user --> agent3["AI Agent"]:::agent
agent3 -->|"Pre-configured credentials"| sp3
sp3 --> res3[("Enterprise Resources")]:::res
endAutomatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
MLflow
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
flowchart LR
classDef mlflow fill:#fff7ed,stroke:#c2410c,stroke-width:1.5px,color:#7c2d12;
classDef artifact fill:#f8fafc,stroke:#475569,stroke-width:1.5px,color:#0f172a;
classDef action fill:#e0e7ff,stroke:#4338ca,stroke-width:1.5px,color:#1e1b4b;
classDef output fill:#f0fdf4,stroke:#16a34a,stroke-width:1.5px,color:#14532d;
subgraph experiment ["Experiment Tracking"]
direction TB
params["Prompts, Temperatures,<br/>Model Names"]:::artifact
metrics["Metrics & Output Quality"]:::artifact
code["Code Versions & Datasets"]:::artifact
end
subgraph tracing ["Tracing"]
direction TB
chain["Agent Execution Tree"]:::artifact
tools["Tool Calls & Retrieval Steps"]:::artifact
latency["Token Counts & Latency"]:::artifact
end
subgraph registry ["Model Registry"]
direction TB
staging["Staging"]:::action
production["Production"]:::action
archived["Archived"]:::action
staging --> production --> archived
end
subgraph evaluation ["Evaluation"]
direction TB
judge["LLM-as-a-Judge"]:::artifact
criteria["Relevance, Groundedness,<br/>Toxicity, Hallucination"]:::artifact
end
agent["AI Agent"]:::mlflow
agent --> experiment
agent --> tracing
agent --> evaluation
evaluation -->|"winner promoted"| registry
experiment -->|"best run registered"| registry
registry -->|"deploy to"| prod[("Production")]:::output
prod -->|"feedback traces"| tracingExperiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Pillars of AI governance.
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Unity Catalog
Rendering diagram...
Rendering diagram...
Identity
No title
Rendering diagram...
Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Rendering diagram...
Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
MLflow
No title
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
Rendering diagram...
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
No title
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
Rendering diagram...
Rendering diagram...
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Pillars of AI governance.
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Whats wrong with Agents today?
Built as rapid POCs where governance is an afterthought
Prioritizing raw accuracy and stability while completely ignoring security controls
Granting overly broad, elevated permissions for convenience during early development
Operating without audit trails, making agent decisions and tool executions untraceable
Impact:
3x longer production deployment cycles to remediate governance and security debt
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Unity Catalog
Rendering diagram...
Rendering diagram...
Identity
No title
Rendering diagram...
Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Rendering diagram...
Databricks Identities
There are three types of Databricks identity:
Users: User identities recognized by Databricks and represented by email addresses.
Service principals: Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
Groups: Groups simplify identity management, making it easier to assign access to workspaces, data, and other securable objects.
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
MLflow
No title
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
Rendering diagram...
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
No title
MLflow is the observability and lifecycle layer of agent governance — the complement to Unity Catalog's data & access layer. Together they form the Govern agents ring in the pipeline above.
Rendering diagram...
Rendering diagram...
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
What is Agent Governance?
Data governance is the framework of polices, processes, and
standards that ensure data is accurate, secure and properly
managed throughout its lifecycle
Rendering diagram...
Lifecycle Management
Separation of Duties
Version Control, CI/CD Pipelines, Environment Management, Deployment Orchestration, Change Management
Risk Management
Defense in Depth
Data Quality Monitoring, PII Detection, Guardrails, Compliance, Model Validation
Security
Least Privilege Access
Authentication, Secrets Management, Access Control, Data Protection, Network Security
Observability
Audit Everything
Tracing, Logging, Monitoring, Lineage, Alerting
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Lifecycle Management
Can you safely promote changes through environments with proper review and rollback capabilities?
Risk Management
Do you have multiple layers of protection to catch issues before they impact production?
Security
Are all your data sources accessible only to authorized agents and users?
Observability
Can you trace every tool your agent used, when it ran, what data it accessed, and what it returned?
Administrative Roles
The following administrative roles manage Databricks identities:
Account Admins: Add users, service principals, and groups to the account, and assign admin roles.
Workspace Admins: Add users and service principals to the account; grant access to workspaces.
Group Managers: Manage group membership and assign the group manager role.
Service Principal Managers: Manage roles on service principals.
Agent Authentication Modes
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
No title
Rendering diagram...
Automatic Authentication Passthrough
System Authentication:
Agent service principal gets automatic access to declared resources
SAME identity regardless of who calls it
Managed credentials automatically
On-Behalf-of-User Authentication
Delegated Authentication:
Agent uses end user's credentials (CALLER's identity)
Good for user-specific sensitive data access
Requires specifying API scopes
Manual Authentication
Centralized Provisioning:
Pre-created service principals with specific permissions
What some orgs prefer for CENTRALIZED identity management
No on-the-fly service principal creation
Experiment Tracking
MLflow tracks every experiment run so you can compare and reproduce results:
What it logs: prompts, model names, temperatures, code versions, datasets, and output metrics.
Autologging works out of the box with LangChain, LlamaIndex, OpenAI, Anthropic, and HuggingFace — no boilerplate required.
Tracing
Multi-step agents are black boxes without tracing. MLflow Tracing gives granular visibility into every execution:
Records the full agent execution tree — chain-of-thought, tool calls, RAG retrievals.
Captures token counts, latency, and errors at each step.
OpenTelemetry compatible — stream traces into any production observability pipeline.
Operates asynchronously so production performance is never impacted.
Model Registry
Governance for agents means controlling what reaches production:
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Beyond models, the registry manages prompt templates — version-control your system prompts, compare which variant scored best in evaluation, and roll back instantly on regressions.
Staging
Candidate version under review and testing
Production
Live version serving user traffic
Archived
Retired version kept for audit and rollback
Evaluation
MLflow replaces manual QA with automated, repeatable evaluation:
LLM-as-a-Judge scores outputs on relevance, groundedness, toxicity, and hallucination rate.
Evaluation results are tied to the experiment run, so every registry promotion has a traceable score attached.
MLflow + Unity Catalog
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages
Data access
Tables, volumes, schemas, secrets
—
Model storage
Registers models as UC assets
Logs artifacts and promotes to UC
Lineage
Data lineage across pipelines
Experiment → run → model lineage
Observability
Audit logs of data access
Traces of agent tool calls and outputs
Lifecycle
Environment promotion via CI/CD
Staging → Production → Archived stages