Connecting your agentic system to Emet
This guide details the path from an Emet account to a verified agent in production. It covers what Emet checks and what you get back, connecting your data, what you build, the data you send, edge cases, going live, and handling errors.
At a glance
What Emet does for your agent
Your agent answers questions with figures, comparisons and trends drawn from its tools.
Emet examines each answer, recomputes its figures and calculations in deterministic code, and checks every claim against the evidence your agent actually received and, with your data connected, against the source itself. No model ever judges its own output. Every response your agent makes receives an Emet Accuracy Report: an assessment of its accuracy and completeness. Inaccurate figures or claims are logged, and Emet provides a clarification or correction based on the available evidence.
This way you find out which answers hold up, which ones do not, and exactly which claim, tool result, and model were responsible.
What you get back
Every checked answer produces an Emet Accuracy Report in the Emet Observability Dashboard, where everyone on your team can log in and review it at any time. Each report shows every claim in the answer with its verdict and reason, the evidence behind it, and the calculation behind every derived figure.
Each claim receives one of four verdicts, with its reason shown alongside it.
The Emet Observability Dashboard
An Accuracy Report covers one answer. The Emet Observability Dashboard covers all of them, across your company and by team, built from the same traces:
- Overview
Accuracy rate, hallucination rate and coverage (the share of claims Emet could check) over time. Accuracy is broken out by application, by model and by data source, with the reason behind every miss: incorrect value, misattributed figure, calculation error or ungrounded claim.
- Reports
Every Accuracy Report, filtered by status, application and team.
- Usage & cost
Agent runs, model calls, tool calls, tokens, estimated cost and run duration, by model and by tool.
- AI Analytics
Where your tokens go and where spend can be cut, ranked by monthly saving. Emet only recommends a change that keeps the hallucination rate flat.
The dashboard fills in from the moment you integrate. Already sending traces to another tracing backend? Keep it, and add Emet as a second export.
Example: right data, wrong reading
The question was “How did CTV perform against display last week?” The agent's SQL was correct and returned four rows:
| Channel | Week of | Conversions | CPA |
|---|---|---|---|
| CTV | 31 August | 1,248 | $50.00 |
| CTV | 7 September | 1,340 | $43.96 |
| Display | 31 August | 1,030 | $40.00 |
| Display | 7 September | 1,075 | $41.67 |
The Accuracy Report showed:
What is AI observability?
AI agents are non-deterministic: the same question can take a different route each time, so you can't test every behavior before launch. Traditional monitoring tells you an agent is running, what it cost and how many tokens it used. AI observability goes further and shows you what your agent actually did in production: what it was asked, which tools it called, what came back, and what it finally said. On its own, neither tells you whether the answer was right.
This works through instrumentation, which means adding a small amount of logging to your agent, typically through an SDK or an integration with your existing framework. After that, every run is captured automatically as structured data, organized in three levels:
- Conversation
The full multi-turn exchange with a user.
- Trace
A single turn, meaning one question, the agent's work, and its response.
- Span
One step within a trace, such as a sub-agent or a tool call and its result.
Because Emet sees the tool results your agent received at every step, and can query your connected data sources directly, it checks each claim in the final answer against both the evidence your agent saw and the source itself. Emet gives you both: what your agent did, and whether what it said is true.
What you build
Send each turn to Emet as one JSON document over HTTPS, using the open-source Emet SDK for Python with Pydantic AI, or our reference exporter for TypeScript (section 5). The document follows the OpenTelemetry GenAI conventions, so an agent that already records them maps onto it directly. Direct OpenTelemetry ingestion (OTLP) is available on request.
What you do not have to touch
- No proxy required.
Your model calls go straight to your provider, exactly as today.
- One exporter, at most.
The Emet SDK for Python with Pydantic AI, or one exporter for your stack, sends plain HTTPS and JSON, under your control.
- No inbound access to your agent.
Emet never calls into your application. Traces go one way, from your exporter to Emet. The only thing Emet connects to is the data source you choose to connect: read-only, scoped by you, revocable at once (section 3).
- No added latency.
Exports run in the background, after your user already has the answer.
- No production risk.
An export that fails is logged and dropped. It never reaches the request that served your user.
How it fits
Your agent works exactly as it does today. Emet only enters the picture after your user already has the answer, and reads your connected data only to check it.
Sequence: the user asks a question; the agent calls its tools with arguments and gets results; the agent answers the user. Only then does the agent export the turn to Emet and receive an acceptance. Emet runs read-only queries against your data, gets the source results, checks each claim, and produces the Accuracy Report.
The path to a first report
Six steps in order: 1. Set up, 2. Connect, 3. Instrument, 4. Export, 5. Check, 6. Roll out.
Steps 3 and 4 happen in your codebase.
Before you start
Two short lists: the answers your team needs before writing code, and the three things to create in the Emet dashboard.
Questions to answer first
Every one of these is about your own system, and each one maps to a field Emet needs.
tenant_refconversation_idIf your agent already records OpenTelemetry GenAI spans with content capture on, most of this data is already flowing, and section 4 shows how each attribute maps to the envelope. On Python with Pydantic AI, the Emet SDK reads those spans for you.
Set up the account
Create the team from Team in the sidebar. Register the application and mint its key under Applications.
- 01Create a team
Open Team. One for each part of your company whose work should be reported separately. Give each one the external ID your agent will send as
tenant_ref. - 02Register the application
Open Applications and add the agent system that will send traces. A name and a short description are enough.
- 03Mint a key
For the application, leaving the team blank, so each trace is attributed by its own
tenant_ref. The key is shown once; store it the way you store any other production credential.
One application per environment
Register a separate application for each environment that will send traces, such as “Nova Analyst (staging)” and “Nova Analyst (production)”, each with its own key. Trial traffic and real traffic stay apart on every report, rate limits are counted per application so a load test never throttles production, and revoking a staging key never touches production. Keys can carry an expiry date, rotate in pairs without downtime, and can be revoked at once.
What to hand your engineer
- A key for the trial application. Emet's API is at
https://app.emet.so. - The external ID of at least one team.
- The answers to the questions above.
Connecting your data
Your traces tell Emet what your agent saw. A data connection lets Emet check it against the source. With your warehouse connected, Emet runs its own read-only queries to confirm that the results your agent received match your data, that the query asked the right question of it, and that each claim in the answer holds up against the source, not only against the trace.
Emet connects to Databricks and other major data warehouses. You connect yours in the Emet dashboard, which walks you through it.
What the connection is
- Read-only.
Emet runs read queries only. It never writes, alters or deletes.
- Scoped by you.
Emet sees only the data you grant it access to.
- Revocable at once.
Remove Emet's access in your warehouse, and it stops immediately.
- Visible in your own logs.
Every Emet query runs through the connection you create, so it appears in your warehouse's query history alongside everything else.
How Emet learns your data
During onboarding, Emet maps your definitions: which tables hold which metrics, your attribution windows, fiscal calendar, currencies and time zones. That way its queries ask the same question your agent did, by your rules.
Before you connect
Capturing a turn
Emet takes each turn as one JSON document, the envelope, posted to a single endpoint. It carries the evidence: every model call and tool call in the turn, with what went in and what came out. Its fields follow the OpenTelemetry GenAI semantic conventions, so an agent already instrumented with them maps onto it field by field.
Sending JSON
Your exporter sends Emet one envelope per turn. It names whose work the turn was and carries the turn as a small tree of spans. Every model call and every tool call is its own span, and all of them hang off the turn: a turn with five tool calls across three model calls has nine spans. Every field and limit below is what the endpoint enforces today.
The request
POST /api/v1/traces/Authorization: Bearer, followed by your application keyThe envelope
tenant_refconversation_idtrace_idtrace_id adds to the same trace.ingestion_mode"sdk" when an exporter in your app posts the envelope: the Emet SDK, a reference exporter, or your own. "telemetry" when the envelope is built from OpenTelemetry spans outside your app.spansendtrue on the last request of a turn, so Emet starts checking at once. A request that carries the turn span has the same effect.semconv_version"emet-genai-1", the vocabulary on this page. An exporter that passes OpenTelemetry attributes through unchanged sends the GenAI conventions version its instrumentation emits.session_refenduser_refenduser.id. Emet matches it to a member of your organization by email; other values are stored but not shown. Set it from your server's signed-in identity, never from a value your users can edit, and send it on the first request of the turn: the first value Emet receives is the one it keeps.resourceRules the endpoint enforces
conversation_id,trace_idandspan_idaccept letters, digits and the characters_ . : -only, and an id made only of dots is rejected. They may be up to 255, 128 and 64 characters long.- Every timestamp carries a UTC offset, and no span ends before it starts.
- A
span_idappears once per request, and only one span per trace is the final output. - A request carries at most 1,000 spans and 32 MB.
attributesandresourcehold small metadata: 16 KiB each. - Text must not contain a NUL character, and numbers must be finite.
Span fields
span_idmodel.0 or tool.call_1.parent_span_idspan_id for everything else.name"chat" or "execute_tool get_spend".kindturn, model, tool, retrieval or memory for most agents.started_at, ended_atstatusok or error. Set error on a tool that failed.is_final_outputtrue on the one model span that produced the answer.model_name, provider, tokens, finish_reasontool_name, tool_call_idtool_call_id links the span to the model's call.error_messageerror.attributespayloadThe payload
The payload is the evidence, and it is the part of the envelope that matters most. Its keys depend on the span kind:
toolarguments, resultmodelinput_messages, output_messagesmodelsystem_instructions, tool_definitionsretrievalquery, documentsmemoryquery, recordsEach message is a role (user, assistant or tool) and a list of typed parts: text for anything the user or the model wrote, tool_call for the model asking for a tool (id, name, arguments), and tool_call_response for the tool's answer as the model saw it (id, response). Some frameworks, PydanticAI among them, send tool answers in a user message made only of tool_call_response parts; Emet skips those when it picks the turn's question.
A worked example
One question, one tool call, one answer: “What did we spend on paid search last week?”
The turn span carries timing only. Under it hang three spans: model.0, which calls a tool; tool.call_1, which carries the arguments and the result; and model.1, which writes the answer and is marked as the final output.
turnturnmodel.0modelget_spend for paid search, 7 to 13 September.tool.call_1toolmodel.1modelThe span below is the one that carries the evidence. The arguments say which question was asked of the data; the result is what came back.
If your agent emits OpenTelemetry
On Python with Pydantic AI, the Emet SDK builds the envelope from your spans in your own process, and you can skip this part. Otherwise, build it from your spans: each GenAI attribute below maps to a field in it. Direct OpenTelemetry ingestion (OTLP) is available on request; your Emet contact gives you the endpoint and setup.
gen_ai.input.messages, gen_ai.output.messagesinput_messages, output_messages on a model spangen_ai.system_instructions, gen_ai.tool.definitionssystem_instructions, tool_definitions on a model spangen_ai.tool.call.arguments, gen_ai.tool.call.resultarguments, result on a tool spangen_ai.retrieval.query.text, gen_ai.retrieval.documentsquery, documents on a retrieval spangen_ai.memory.query.text, gen_ai.memory.recordsquery, records on a memory spangen_ai.conversation.idconversation_idenduser.idenduser_ref, set to the user's email. Emet matches members by email, so if enduser.id holds an internal id, look up the email or leave the field out.What your spans need
- Content capture on.
Without it, messages, tool arguments and results are never recorded, and Emet has no evidence to check.
- The turn's team.
Each turn carries the team's external ID from section 2, so its report lands with the right team.
- The turn's conversation.
Set
gen_ai.conversation.id, so Emet can show a turn next to the ones around it. - Tool arguments as well as results.
A claim can only be traced back to the query that produced it when the arguments are there.
- Your system prompt stays private.
Standard instrumentation records it as text. To keep it private, hash or remove
gen_ai.system_instructionsbefore export, as the Emet SDK and the reference exporter do.
Building the exporter
The exporter builds the envelope from section 4. It is one function in your codebase: it turns a finished agent turn into the envelope and posts it.
Python with the Emet SDK
On Python with Pydantic AI, you do not build an exporter. The open-source Emet SDK, emet-sdk on PyPI (FUSEis/emet-py), reads the OpenTelemetry spans Pydantic AI already emits and sends each turn for you, applying the design rules below. It runs on Python 3.10 to 3.14, and Pydantic AI on Python 3.11 or later.
With a coding agent
emet-cli sets up the SDK with your coding agent, such as Claude Code or Cursor. You type the key into your own terminal, never into the agent's chat. login saves it to the app's .env, which the setup tells the agent not to read. To back that up, deny your agent read access to .env, for example with "Read(./.env)" under permissions.deny in Claude Code's .claude/settings.json. That rule covers Claude Code's file tools and shell commands such as cat, but not a script that opens the file itself; only Claude Code's sandbox blocks every process.
- 01Save your key.
In your own terminal, in the app's folder, run
uvx emet-cli@0.1.3 loginand paste a key for your trial application.uvxcomes with uv. - 02Hand the setup to your agent.
Tell your coding agent: “Follow https://github.com/FUSEis/emet-py/blob/v0.1.3/cli/AGENT_SETUP.md to set up Emet.” It installs the SDK, wires it into your code, shows you the diff, and asks you to send one question through the app.
- 03Check the turn.
Once the app has answered that one question, run
uvx emet-cli@0.1.3 check --last. It reads the most recent turn's result and says whether it reached Emet whole, and the fix if it did not.
By hand
Set EMET_APPLICATION_KEY in the app's environment. Without it, the app logs “Emet is off” once and runs as before. write_delivery_file saves each turn's result for emet-cli check --last, and writes only when its .emet folder exists, which emet-cli login creates. Setting up by hand, create it with mkdir .emet and add it to .gitignore. The SDK's README covers streaming responses, the delivery result and every option.
What the SDK handles for you
- Only spans started inside
turn()are sent. Your other agents, HTTP calls and database queries never leave the app. - It adds itself to your existing OpenTelemetry setup, so Logfire or any other exporter keeps receiving the same spans.
- Each turn is sent on a background thread after the answer, split across requests within Emet's limits (a single span over 32 MB is refused), with one retry on a timeout, a 5xx or a short 429. Nothing that goes wrong while tracing or sending raises into your app, and a turn without a team or conversation is skipped rather than failing the request.
- Known secret formats, such as provider API keys, tokens, private keys and passwords in URLs, are redacted from every string. Remove any other sensitive values from tool results yourself.
- The system prompt is sent as a SHA-256 hash, and files and images become their type, size and digest.
The rest of this section is for TypeScript and any other stack, where you build the exporter yourself.
Three steps, any stack
- 01Collect during the turn.
For each model call keep the messages in and out, the model name, token counts and timing. For each tool call keep the name, the arguments, the result and timing. Most agent frameworks already hold all of this in the run's result once the turn completes.
- 02Build the envelope when the turn completes.
One model span per model call and one tool span per tool call, all under one turn span. Mark the model span that produced the answer as the final output.
- 03Send it in the background.
After the answer has gone out, post the envelope with a short timeout.
Design rules
tenant_ref, the conversation and its history from your own authorization and storage, never from the request body.tenant_ref decides which team's reports a turn lands in, and history taken from the browser could plant evidence the model never saw.enduser_ref, from your server's session, and nowhere else. Send no credentials or connection strings.The reference exporter
For the Vercel AI SDK, we maintain a reference exporter that applies every rule above. Your team can adapt it rather than write one from scratch; we share the source files with you.
ai 6.0.116, strict type checkingresult.steps. The fields it reads mean the same in AI SDK 6 and 7.after(), which runs once the response has been sent.In a Next.js route, the only change is to note when each step finishes, then hand the finished run to the exporter inside after(), which runs once the response is out.
Test before you ship
Point the exporter at your trial application and run a handful of real questions, including one where a tool fails and one that takes several tool rounds. Each post should come back 202 with no capture gaps.
What the reference exporter handles
Edge cases
Real agents retry, stream, call tools in parallel and hand work to sub-agents. None of it needs special support from Emet, but each case has one right way to show up in the envelope.
error and what went wrong in error_message. Keep the arguments and leave the result out. A span with no result and no error status is reported as a capture gap, because Emet cannot tell a failure from a result that was never recorded.tool_call_id, all under the turn.agent for the sub-agent and point its model and tool spans at it. If the sub-agent's work is its own conversation with its own answer, export it as its own turn.trace_id.trace_id; order does not matter and duplicates are stored once. Emet starts checking as soon as a request carries the turn span or sets end, so send those only with the last request. Otherwise it checks 30 seconds after the last span arrives, and anything later is stored but not checked again.A failed tool call carries its status, the message and the arguments. No result, and no capture gap.
Going live
How to know the integration is right before it carries real traffic, and what to watch once it does.
Check the first turns
Send a handful of real questions through the trial application, including one that makes a tool fail, and check each one. The first four checks read the endpoint's response. If OpenTelemetry ingestion is enabled for your account, confirm your exporter logs each export as successful, then use the last three.
With the Emet SDK, send test questions one at a time and run uvx emet-cli@0.1.3 check --last after each, since it reads the most recent turn's result. It reads the SDK's delivery result and covers the first four checks at once: a turn is captured when every request was accepted, the turn found its team, nothing arrived late and no gaps were reported. Otherwise it says what went wrong and the fix.
202gaps in the responseunattributed in the responsefalselate in the responsefalseRoll out in stages
- 01Trial application.
Real questions, trial key, until every check above passes.
- 02Production application, part of the traffic.
A new application and key for production. Export a share of turns first if you want to watch volume on your side.
- 03All traffic.
Export every turn. Reports accumulate per turn and stay available in your dashboard.
What to monitor
202400), a key problem (403), or a burst over the rate limit (429).gapsunattributed: truelate: trueThe Emet SDK logs under the emet_sdk logger on the first turn and whenever the outcome changes: a warning when a turn is not captured, and an info line when it is, which Python hides by default. It also hands every turn's delivery result to the on_delivery callback you pass to configure(), for your own monitoring.
The rate limit is 10,000 requests an hour per application. If your production traffic will run above that, talk to us before you switch on step 3.
Keys over time
Keys rotate in pairs, so a rotation never needs a coordinated deploy: mint the new key, deploy it at your own pace, then revoke the old one. Revoking is immediate, which also makes it the fastest way to stop an export.
After a framework upgrade
Agent frameworks change how they record a turn between major versions. After upgrading yours, send a few turns through the trial application and repeat the checks above before the upgrade reaches production.
Handling errors
Every response Emet can send, what it means, and what your exporter should do about it. Branch on the status and the error body's code, never on the message text.
202gaps, unattributed and late, and log anything unexpected.400403, code not_authenticatedAuthorization: Bearer header. Any other scheme counts as none.Authorization: Bearer <key>.403, code authentication_failedBearer header whose key is unknown, expired, revoked or malformed (missing, or with extra parts), or whose Application or organization is deactivated.403, code permission_deniedresource names a different deployment.environment.name or none at all.deployment.environment.name in resource, or use that environment's key.413trace_id.429Retry-After header, then retry.5xxWith the Emet SDK
The SDK handles these for you. It retries once on a timeout, a 5xx, or a 429 whose Retry-After is 5 seconds or less, splits each turn across requests within the limits, and stops a turn at the first request Emet refuses. A single span over 32 MB cannot be split; Emet refuses it with 413, and the delivery result reports span_too_large. Each turn's outcome is in its delivery result, and uvx emet-cli@0.1.3 check --last explains it with the fix.
For a key pinned to an environment, the SDK sends deployment.environment.name from your OpenTelemetry resource. Set it with OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production, or use that environment's key.
What an error body contains
errorcodeinvalid, throttled or request_too_large.status_codedetails400, 403, 429400, the field errors, with each span's position in the spans list.limit_bytes413Below, the first span was sent with a timestamp that has no timezone. details points at span 0 and names the field.
Typical field errors
tenant_refspans, position 0, started_atspans, position 2, payload resulttrace_id_ . : - and may not be dots alonespansTroubleshooting
400 on a tool result or arguments400 on a timestampZ or an explicit offset.400 on an identifier202, but gaps lists tool spans202, but no report to opentenant_ref is hardcoded in the exporterunattributed: true on every turnlate: trueend, only with the last request of a turn.