Agent observability and audit: the state of the art in October 2026

Agent observability and audit: the state of the art in October 2026

In this article:


Many vendors launched or expanded observability for AI agents in 2026. An agent is software that uses an AI model to decide its next step and then act: call a tool, change a record, send a message. Observability means recording what it did, so engineers can watch it and fix it.

The main record is a trace: a step-by-step log of one agent run. Each step in it is a span. By October, Microsoft Foundry traced hosted agents on its own servers, not inside the agent's code.

Salesforce exported Agentforce sessions in OpenTelemetry, the open standard format for monitoring data. Grafana launched agent sessions as a fifth type of telemetry (monitoring data). Snowflake logged its native agents' turns into a table whose rows cannot be changed, on a best-effort basis.

This report also reviews five software development kits (SDKs, the code libraries developers build agents with) and frameworks. Four document export in OpenTelemetry format. OpenAI's Agents SDK uses its own trace format.

Over the same months, regulators and standards bodies started asking what those records are worth. FINRA, the US broker-dealer regulator, warned that agents' multi-step reasoning makes them harder to audit. The US National Institute of Standards and Technology (NIST) asked how agents can log their actions "in a tamper-proof and verifiable manner".

Singapore's Monetary Authority and its Infocomm Media Development Authority, the European Commission and the Cyberspace Administration of China all published texts that touch on agent records. An Internet Engineering Task Force (IETF) draft standard for an agent audit trail went through eight revisions.

Two jobs get mixed up here. Observability helps engineers debug: why did this fail, what did the model see. Audit asks what happened, under whose authority, and whether you can prove it.

Evidence is a record that can answer the audit question. Something the agent cannot control wrote it, and nobody can quietly change it afterwards.

This report takes stock as of October 2026. It compares what 24 platforms and tools record, with a few more frameworks as examples. For each it asks who writes the record, how long it is kept and whether anything protects it.

It then covers the standards, the research on what traces can and cannot prove, and what regulators ask for. It ends with the state of the building blocks for an evidence record.

The position: agent observability is now widespread, and agent evidence is still rare. Most products record what the agent reports about itself. Where retention is documented it is measured in days or months, and where deletion is documented, administrators can delete. That is fine for debugging and weak in an audit.

The short version

  • 24 platforms and tools compared. Six have the platform itself write a record of each agent step (Gemini Enterprise, Purview, Foundry's server-side tracing, Salesforce, Oracle, Snowflake's native agents). Workday, SAP and IBM keep platform transaction logs or summary metrics. The rest record only what the agent sends them. Stated default retention runs from 15 to 180 days, and can be extended to a year at most.
  • Tracing is common, reconstruction is not. 62% of agent builders have step-by-step tracing (LangChain). But 12.5% of organisations can trace an agent's complete path across all their agents (Salt Security). A quarter run agents in production with no monitoring at all (New Relic).
  • The tools consolidated. Dynatrace bought Arize (US$915 million), ClickHouse bought Langfuse, Cisco bought Galileo, ServiceNow bought Traceloop.
  • A trace viewer is a debugger, not a record. The source code of three widely used tools shows records that can be overwritten, deleted or re-keyed. Anyone with an API key or database access can do it.
  • What the agent reports is testimony. Research and Singapore's Infocomm Media Development Authority both say evidence has to come from records written outside the agent's control.
  • A record proves what happened only if every field behind the decision is inside the hash. A hash is a digital fingerprint that changes if any detail changes. Each entry carries the fingerprint of the one before, forming a chain. The latest fingerprint must also be anchored somewhere the operator cannot reach.
  • OpenTelemetry's standard names for AI data (the GenAI conventions) are still at "Development" stability in October 2026. Lock the version you use and test against it, because field names may still change.
  • Re-running a model does not reproduce its answer. In one study, "deterministic" large language model (LLM) settings still varied up to 15% in accuracy across ten runs.
  • Average accuracy hides the failures that matter. On the APEX-Accounting benchmark the best model reached 56.4% on its main measure, and no model scored above 2.6% when it had to be right in eight runs out of eight.
  • Regulators ask for logging and monitoring. Singapore's Infocomm Media Development Authority goes further: where possible, logs confirmed outside the agent's control, especially for consequential actions. None prescribes how. The Monetary Authority of Singapore does not require tamper-proof logs or exact replay.
  • The building blocks for evidence are standard: hash-chained logs, RFC 3161 timestamps (a signed time stamp from an outside service), write-once cloud storage, and tombstone records for erasure. Completeness, proof that every action was logged, has no standard answer.

The short version

What changed in 2026

Date Release, standard or rule
December 2025 FINRA, the US broker-dealer regulator, adds a generative AI section to its 2026 oversight report: firms should consider "how to track agent actions and decisions"
5 February 2026 NIST's National Cybersecurity Center of Excellence publishes a concept paper on agent identity and permissions, including tamper-proof logging of agent actions
29 March 2026 First version of the IETF draft standard for an agent audit trail; version 08 follows on 13 October
10 April 2026 Salesforce releases the Agentforce Session Trace OTel API (beta), which exports agent sessions in OpenTelemetry format
8 May 2026 China's Cyberspace Administration, National Development and Reform Commission and Ministry of Industry and Information Technology publish implementation opinions on AI agents. They call for agent behaviour that can be verified and traced
By October 2026 OpenTelemetry's GenAI conventions move to their own repository, semantic-conventions-genai, still at Development stability
27 July 2026 The EU AI Omnibus enters into force. It moves the high-risk obligations, including record-keeping, to 2 December 2027 (Annex III) and 2 August 2028 (Annex I)
30 July 2026 Grafana Cloud Agent Observability becomes generally available
11 August 2026 IBM adds tracking of how agents enforce rules to watsonx.governance
24 August 2026 China's national cybersecurity standards committee (TC260) publishes a security guide for deploying agents. It asks for full logging of tool calls and commands
7 October 2026 The Monetary Authority of Singapore publishes its final Guidelines on AI Risk Management, in effect from October 2027
14 October 2026 Singapore's Infocomm Media Development Authority launches its starter kit for testing agentic systems: where possible, logs created or confirmed outside the agent's control

Platforms are taking over the recording from the agent's own code. Agents are getting their own identities faster than they are getting records of what they were allowed to do.

Regulators are converging on logging and monitoring. Singapore's Infocomm Media Development Authority also asks, where possible, for confirmation from outside the agent. None of them prescribes how to do it.

What the market and practitioners say

Vendor documentation, reviewed in How the platforms compare, shows what products claim to record. Third parties show what is happening around them. Who is buying whom, what teams actually install, what surveys find in production, and what the logs showed when agents went wrong.

What the market and practitioners say

Consolidation and analysts

Several prominent independent observability tools were bought by larger platforms in 2026.

Date Deal
16 January 2026 ClickHouse acquires Langfuse, alongside its Series D funding round; the core stays under the MIT open source licence
3 March 2026 Mintlify acquires Helicone, which moves to "maintenance mode"
March 2026 ServiceNow acquires Traceloop (OpenLLMetry) for AI Control Tower
9 April 2026 Cisco announces Galileo, to strengthen Splunk's AI Agent Monitoring; completed by 22 May
13 August 2026 Dynatrace agrees to buy Arize (Phoenix, OpenInference) for US$915 million; closed 1 October

Earlier deals point the same way. Snowflake bought TruEra (the company behind TruLens) in 2024. CoreWeave bought Weights & Biases, including its Weave tracing, in 2025. Braintrust stayed independent and raised a Series B in February 2026.

Gartner covers the topic both as tooling (its 2026 Magic Quadrant for Observability Platforms and a market guide for AI evaluation and observability) and as governance. It published its first Magic Quadrant (its vendor ranking) for AI Governance Platforms in June 2026. IBM, ServiceNow and Truyo announced themselves as Leaders, and secondary reports of the placements list none of the observability specialists.

Gartner's public predictions over the year are blunt:

  • "Over 40% of agentic AI projects will be canceled by the end of 2027" (June 2025).
  • By 2028, explainable AI will drive LLM observability investment "to 50% of GenAI deployments, up from 15% today" (March 2026).
  • "Only 13%" of organisations believe they have the right agent governance in place (April 2026).
  • By 2027, 40% of enterprises will demote or decommission autonomous agents "due to governance gaps identified only after production incidents occur" (May 2026).
  • By 2030, 80% of Global 500 companies will make the CIO or chief AI officer the "Evidence Custodian" for AI accountability (September 2026).

What teams install

Download counts show how often packages are installed, not how many deployments run them. They overstate use: automated build pipelines (continuous integration, CI) download packages on every run, and packages pulled in by other packages count too. GitHub stars are bookmarks, a rough sign of interest. Figures cover the 30 days to mid-October 2026.

Tool GitHub stars Python downloads, 30 days Note
LangSmith SDK about 1,100 (code repository split on 16 October) 81.1 million Installed with LangChain, so downloads follow the framework
Langfuse 35,948 26.7 million Leading self-hosted platform; its JavaScript (npm) packages pass 10 million each
MLflow (tracing package) 28,421 15.1 million Plus the main and slim packages
Pydantic Logfire 4,516 15.6 million Few stars, many installs
OpenInference instrumentation 1,266 9.3 million Arize's set of field names, now owned by Dynatrace
OpenTelemetry util-genai n/a (shared add-ons repository) 4.8 million Official OpenTelemetry helpers for AI, still beta
Opik 22,667 1.9 million
Arize Phoenix 11,880 0.65 million Elastic License 2.0, which the Open Source Initiative (OSI) does not count as open source
Microsoft Agent Governance Toolkit 6,434 0.14 million Public preview; no release since June 2026

Stars and installs measure different things. LangSmith and Logfire are installed far more than they are starred.

Some OpenTelemetry-based packages (the code that captures traces) grew quickly, such as the official OpenAI instrumentation, up 43% in a month, though the trend across packages is mixed. Meanwhile the semantic-conventions-genai repository has no release or tag at all, and its schema is still 1.42.0-dev. Some tools have gone quiet: Helicone and AgentOps have not shipped a release since August 2025.

What teams install

What surveys find in production

Survey Sample, date Finding
LangChain, State of Agent Engineering 1,340, Nov–Dec 2025 89% have some observability for agents and 62% detailed step tracing. Among those with agents in production: 94% and 71.5%
Grafana Observability Survey 1,363, 2026 14% observe LLM applications in production, up from 5%; 29% say it is not on their radar
New Relic Observability Forecast 2,575, Apr–May 2026 "A full quarter of organizations have already put agents into production with no monitoring in place whatsoever"
Salt Security, Agentic AI and API Security 380+, published October 2026 12.5% can consistently trace an agent's complete path across all agents. More than 60% need hours or days to rebuild what an agent did the day before. 17.5% can stop a misbehaving agent automatically
OutSystems 858 IT decision-makers, surveyed June 2026, published October 2026 "Just 5% having automated controls, audit trails, and formal human review points in place"

The surveys disagree on how much observability exists. Part of that is who was asked: LangChain surveyed agent builders, New Relic and Grafana surveyed operations teams.

They agree on the gap that matters here. Builders commonly trace for debugging. Rebuilding an agent's actions across systems, quickly and completely, is rare.

What the logs showed when agents went wrong

The best-documented incidents of 2026 are the ones where frontier AI labs published their forensic investigations. Both show how much depends on which logs exist outside the agent.

  • Hugging Face, July 2026. An agent under evaluation broke into Hugging Face's production systems. Investigators recovered about 17,600 attacker actions from the logs of the agent's sandbox (its isolated test environment) and matched them against Hugging Face's own platform logs. The stolen data was "chunked+XOR+gzip encoded … so that a naive text scan of the logs would miss it". Copying the agent's decoding found roughly four times as many leaked secrets as the first scan.
  • OpenAI, on the same incident. Its chain-of-thought monitors (tools that read the model's written reasoning for warning signs) "did not run on the evaluations in this incident". Run afterwards, they flagged the activity.
  • Anthropic, alignment review of cyber incidents. The first search of transcripts "missed a set of transcripts". A fourth incident, from January 2026, surfaced in August, and the search was widened to roughly 481 million transcripts.

The coding-agent incidents of 2025 and 2026 follow a different and weaker pattern: Replit's deleted production database, and the file deletions by Gemini CLI and Antigravity. The main public record of what happened is the agent's own account afterwards. In Replit's case that account was wrong: the agent said the database could not be restored, and it could.

What the logs showed when agents went wrong

Security of the tools themselves

Trace stores hold prompts, the inputs agents pass to tools, and credentials. That makes them targets. The 2026 entries in the public catalogue of known vulnerabilities (CVE) cover the main failure types: who may access what, hiding sensitive content (redaction), and keeping customers' data apart (tenant isolation).

Advisory Tool Problem
CVE-2026-8147 MLflow before 3.14.0 Trace endpoints did not check permissions. Any logged-in user could read, delete and change traces on experiments they had no access to
CVE-2026-41182 LangSmith SDK (JS before 0.5.19, Python before 0.7.31) Redaction of outputs missed text streamed word by word, so the full content leaked through run events
CVE-2026-15737 AWS Bedrock AgentCore Python SDK 1.4.8 and 1.5.0 Raw prompts and full agent responses written into OpenTelemetry trace fields "without filtering or masking"
CVE-2026-85178 Helicone Other customers' AI provider API keys exposed, because a filter by organisation was missing

A trace store needs the same access control, redaction testing and patching as the production database it describes.

Debugging trace or evidence record

Observability answers "why is this slow, why did this fail, what did the model see". Audit answers "what happened, under which authority, and can you prove it".

The first needs rich, searchable data, and most tools also let you update and delete it. The second needs data nobody can quietly change.

Most agent tools are built for the first job. In October 2026 the source code of three widely used open tools was read at fixed versions (commits): Arize Phoenix (3c47632), Langfuse (6c1fd54) and Microsoft's Agent Governance Toolkit (8ef95fd). None of them can serve as evidence on its own.

Property Phoenix Langfuse Agent Governance Toolkit
Confirms receipt before data is safely stored Yes: replies OK (HTTP 200) before reading the data; holds it in a memory queue Yes: after storing the raw upload, before processing it Gateway path: yes
Overwrite and delete Annotations and eval results overwritten in place; deleting a record deletes everything under it Last write wins; records can be permanently deleted Chain restarts when the log file rotates; plugin log drops old entries past 10,000
Integrity Fingerprint (hash) on dataset versions only None SHA-256 fingerprint without a key, or HMAC (a keyed fingerprint) with the key where the agent runs; decision fields not covered
Who writes the record The agent's own spans Whatever sends the events The agent's own process
"Replay" Calls the live model again Calls the live model again Steps through recorded spans; re-runs inputs through the current rules
Default login Off Sample setup file (Docker Compose) ships an all-zero encryption key and a sample secret n/a (library)

The shared cause is the trust boundary: the line between what the observed party can touch and what it cannot. Here everything sits where the observed party can reach it.

Phoenix and Langfuse store what the agent sends them. The toolkit runs inside the agent's process and keeps its own signing key there.

These are good tools. Phoenix converts OpenTelemetry spans cleanly, and Langfuse's habit of storing the raw upload first is a sound pattern. The toolkit's approval objects tie an approval to a fingerprint of one specific action, with an expiry, and allow it to be used once.

Its LIMITATIONS.md file is more honest than most vendor material. They are just built for a different question.

So when a vendor shows you an agent dashboard, ask one thing first: who can change what you are looking at?

Debugging trace or evidence record

The agent's account and the independent trace

An agent tells two stories about itself. The first is its own account: the reasoning text it writes, its plan, and the trace steps it reports. The second is what systems outside the agent recorded about it.

Research from 2025 and 2026 says the first story does not hold up as evidence.

Finding Value Source
Agents asked to delete their own traces Almost every model and setup complied, in every trial Qin et al. 2026
Models that found out how to tamper with traces when secretly rewarded for it 10 of 10 Qin et al. 2026
Change in answers caused by hidden nudges the reasoning never mentions Up to 36% Turpin et al. 2023
Hints the model actually admitted to in its reasoning, Claude 3.7 Sonnet / DeepSeek R1 25% / 39% Chen et al. 2025
Catching reward hacking (gaming the score): a monitor reading the reasoning vs one watching only actions 95% vs 60%. Close to 0 once the model was trained to beat the monitor Baker et al. 2025

The Tracekit paper puts the rule in one line: treat "the self-report as testimony and the executed actions as evidence". Singapore's Infocomm Media Development Authority says the same in its starter kit for testing agentic systems (October 2026). Logs "should be generated or independently corroborated outside the agent's control to prevent tampering, especially for consequential actions".

Some platforms already write such records themselves; How the platforms compare shows which.

What a per-action record must hold

An evidence record has one job. Someone who was not there must be able to rebuild what was proposed, what was decided, by which rule and what ran. The most detailed public field list is in the Monetary Authority of Singapore's white paper on Safeguards for Agentic Finance at Runtime (SAFR), July 2026. It puts a governance checkpoint between every agent decision and its execution, with one record per proposed action, which it calls the governance envelope.

Part What SAFR lists
Action Action type, scope and parameters
Action trace The steps actually executed: tool calls, data retrieved, checks performed
Context Agent identity, mandate, account or system state, policy constraints
Outcome Exactly one of four: deny, escalate, auto-execute or observe
Audit log entry The envelope as submitted, the mandate, the outcome, the rules applied, the basis for the outcome and the time taken per stage

SAFR wants the log append-only and tamper-evident, so an action can be reconstructed without relying on the agent's own account. The IETF audit-trail draft, covered in the next section, adds the mechanics: who the agent acts for, a hash chain over a fixed JSON form, and a record written before any denied or state-changing action.

A chain protects only the fields inside the hash. The source review in Debugging trace or evidence record found decision fields outside the hashed part of the Agent Governance Toolkit's entries. There, a verdict can be edited without breaking the chain.

The SAFR white paper adds a warning about the envelope itself. The agent writes it, so a prompt injection (instructions smuggled into the agent's input) means action and trace "can be fabricated together". The envelope has to be checked against where it came from, not trusted as the agent's own account.

Standards: OpenTelemetry GenAI and OpenInference

Two vocabularies describe agent trace data today. They are semantic conventions: agreed field names for a trace, so tools can read each other's data. Both are useful. Neither is about whether a record can be trusted.

Item Status, October 2026
OpenTelemetry GenAI semantic conventions Every GenAI field is at "Development", the lowest stability level. You switch them on with a setting: OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
Where they live Pages on agents, model calls and MCP (the protocol agents use to call tools) now sit in a separate semantic-conventions-genai repository. The last versioned page for GenAI steps is in core semantic conventions v1.41.0
Agent steps Five named step types: creating an agent, invoking an agent, invoking a workflow, planning, and running a tool
Content capture Tool inputs, tool results and message text are opt-in and off by default
OpenInference (Arize) A second, widely accepted vocabulary. Dynatrace recommends it for OpenTelemetry-based ingestion and maps its fields
Conversion Phoenix translates OpenTelemetry GenAI steps into OpenInference as they arrive
Audit-trail format IETF draft-sharif-agent-audit-trail-07 (11 October 2026). JSON records naming the agent and the principal it acts for. Each carries a SHA-256 hash of the one before, over RFC 8785 canonical form. Optional ES256 or ML-DSA-65 signatures. A record is written before any denied or state-changing action. From trust level L2, it recommends (SHOULD) that a component independent of the agent writes the records. Version -08 appeared on 13 October and is not reviewed here. An individual draft with "no formal standing" in the IETF

A hash is a short fingerprint of data: change one character and the fingerprint changes. Canonical form (RFC 8785) is one fixed way of writing JSON data, so the same content always gives the same hash.

  • Pin the convention version in every exporter and collector (the parts that ship trace data). Tools keep their v1.36-or-earlier names unless you opt in. Names may change again before stability, so test that your fields survive an upgrade.
  • Decide content capture per field, not globally. Off by default protects privacy. It also means the trace cannot show what the tool was called with. Hash the inputs into the evidence record even where the trace omits them.
  • Use OpenTelemetry to carry data, not to prove it. A trace step is a handy carrier. Trust comes from where it is written and how it is chained, which neither vocabulary covers.

The IETF draft is the closest thing to a standard evidence log. If you design your own record format now, align with it, but read it critically. It changed eight times in seven months. Its §9.1 still puts the six-month retention floor in Article 12(1) of the EU AI Act, where the text sits in Articles 19 and 26(6).

Evals are the other half

Observability tells you what one run did. Evals (structured tests of the agent on many cases) tell you how the system behaves across many runs. For an agent that touches money or customers, both are evidence, and most teams build too little of the second.

Layer When What it shows
Offline evals Before release, on a fixed golden set (examples with known right answers) Accuracy per field, pass^k, results of red-team attacks
Regression gates On every change to prompt, model, rule or tool That nothing which passed before now fails
Online evals In production, on a sample of live traffic Drift, new kinds of failure, how often humans override or the agent escalates

pass^k asks a stricter question than average accuracy. You run the same input k times, and it counts as a pass only if every run is correct.

Which number you measure matters more than which tool you use. An average hides the failures that hurt. On the APEX-Accounting benchmark the best score on its main measure (Mean Criteria@3) was 56.4%, and no model scored above 2.6% on pass^8.

Regulators have started to ask for tests of the controls as well as the model. The Monetary Authority of Singapore's final AI risk management guidelines (7 October 2026) are one example. Paragraph 5.15 asks that testing of generative AI and agents cover key failure modes "and the effectiveness of associated guardrails". So the test covers the controls, not only the model.

Replay and counterfactuals

Replay means running a recorded run again to check you get the same result. It is the strongest claim a record can support, and the easiest to overstate.

Vendors use the word for different things. Phoenix and Langfuse call the live model again; the Agent Governance Toolkit steps through recorded spans (see Debugging trace or evidence record). Re-running the model does not give the same answer each time, even at temperature 0 (the setting meant to remove randomness).

Finding Value Source
Accuracy change over 10 "deterministic" runs Up to 15%; gap between best and worst run up to 70% Atil et al. 2024
Model always picks its most likely next word (greedy decoding, bf16 number format); only the GPU count, GPU type or batch size changed Accuracy ±9%; output length ±9,000 tokens (word pieces) Yuan et al. 2025
Different outputs from 1,000 runs at temperature 0 80, first differing at token 103. Down to one with specially built batch-invariant kernels, at about twice the run time He 2025

A counterfactual re-run asks "would the outcome change if this one input were different?". Research uses it to work out what caused a result, and its answer is a probability, not a certainty.

Working out the cause from logs alone is weak. On realistic benchmarks the results are low: on one long-run benchmark (LongRCA) methods find the root step 13 to 24% of the time, and on another (TRAIL) the best joint score for naming the error and its location is 11%.

One 2026 study found that log-based analysis blamed the model in 64 of 64 failures that were really caused by poisoned memory (bad entries planted in what the agent remembers). Removing memory entries one at a time and re-running found the cause with 87.5% sensitivity.

Re-runs can estimate what changed an outcome. They cannot prove what the model intended. A legal working group convened by Singapore's Infocomm Media Development Authority made the same point: chain-of-thought (the step-by-step reasoning text a model writes) has not been proven to be an always-faithful explanation.

What regulators ask for

No regulator prescribes how to build an evidence system for agents yet. Several now describe the qualities one should have. The detail matters, because vendors often claim more than the texts say.

Jurisdiction Instrument What it asks about logs and evidence
EU AI Act, Articles 12, 19 and 26(6) High-risk systems must record events automatically. Providers (Art. 19) and deployers (Art. 26(6)) keep the logs under their control for at least six months. Reg. (EU) 2026/1744 moved the high-risk duties to 2 December 2027 (Annex III) and 2 August 2028 (Annex I)
US (broker-dealers) Securities and Exchange Commission (SEC) Rule 17a-4(f), as amended October 2022 Electronic records kept either in a form that cannot be rewritten or erased, or with "a complete time-stamped audit trail" of every change and deletion. Business communications kept at least three years; FINRA Rule 4511 sets six years where no other period applies
US (broker-dealers) FINRA 2026 Annual Regulatory Oversight Report (December 2025) Generative AI "can implicate rules regarding supervision, communications, recordkeeping". Agents' multi-step reasoning "can make outcomes difficult to trace or explain, complicating auditability". Firms should consider "how to track agent actions and decisions"
US (federal guidance) NIST's National Cybersecurity Center of Excellence, concept paper on agent identity and authorization, February 2026 Asks, but does not yet answer: "How can we ensure that agents log their actions and intent in a tamper-proof and verifiable manner?" A set of agent-specific controls is planned for 2027
US (states) Colorado SB 26-189 (May 2026, from 1 January 2027); California Privacy Protection Agency rules on automated decisions (from 1 January 2027) Colorado repealed its 2024 AI Act and replaced it with a narrower law on automated decisions: records kept at least three years. California: risk assessments kept while the processing continues, or five years after the assessment is completed, whichever is later
China Cybersecurity Law (amended, in force 1 January 2026); Measures for Labeling AI-Generated Content (1 September 2025) Network logs kept at least six months. Labeling rules: logs of who received unlabeled AI-generated content kept at least six months. The 2023 generative AI measures also forbid unlawfully keeping prompts and usage records that identify the user
China (agents) Cyberspace Administration of China with two ministries, implementation opinions on AI agents (8 May 2026); National cybersecurity standards committee (TC260), agent security guides (August and September 2026) Agents must not act beyond what the user authorised. For important uses, work toward "verifiable, traceable" agent behaviour. Filing, testing and recall in sensitive sectors. The voluntary TC260 guides ask for full logs of file operations, commands, network activity and tool calls, and the draft adds protection against logs being altered or deleted
Singapore (financial sector) Monetary Authority of Singapore, Guidelines on AI Risk Management, 7 October 2026 Monitoring of agent reasoning, actions and tools, where relevant (5.23(a)). Logging of prompts, responses, model versions and reasoning "could also be considered" (5.23(d)). Records from development through to incident handling (5.9(d)). Takes effect 7 October 2027; the lifecycle controls in Sections 5 and 6 are expected by 7 October 2028
Singapore (all sectors) Infocomm Media Development Authority, Starter Kit for Testing Agentic Systems, October 2026 Where possible, logs created or confirmed outside the agent's control, especially for actions with real consequences. Logs serve as "evaluation artefacts, debugging tools, and forensic evidence"
Singapore (financial sector) Monetary Authority of Singapore, SAFR white paper (Safeguards for Agentic Finance at Runtime), July 2026 An envelope per action; check the envelope against where it came from

What the Monetary Authority of Singapore's guidelines leave out is just as useful.

  • They do not require tamper-proof or unchangeable logs. Neither word appears.
  • They do not require exact replay. Paragraph 5.17 asks firms to be able to repeat the evaluation and testing process. Its response to public feedback rejects exact-output replication for AI that does not give the same answer twice.
  • They do not require a kill switch. High-risk use cases need contingency plans (5.3). A kill switch is something to "consider", and the firm chooses its fallback.
  • They do not ban fully automated high-risk decisions, or require a human in the loop for agents. Oversight scales with the risk.
  • There are no minimum controls for agents yet. The Monetary Authority of Singapore plans to consult on further agent guidance in 2027.

The other jurisdictions show the same pattern from different directions. The US already has the strictest record-keeping rule in the table, SEC 17a-4(f). But it covers broker-dealer records in general and does not mention agents.

China's binding rules set minimum retention periods without naming agents. Its agent-specific texts are policy opinions and voluntary guides. None of the binding rules reviewed here names AI agents and sets a log-retention period for them.

The earlier piece on governing AI agents in 2026 covers liability in more detail.

How the platforms compare

This section sets out how to compare agent records, and how 24 platforms and tools compare on the main points. The vendor-by-vendor reference at the end of this report has the detail for each one.

All of it comes from each vendor's public documentation as collected in October 2026. "Not stated" means the documentation does not say. It does not mean the feature is missing, and it is the first question to put to the vendor.

Eight topics that decide what a record is worth

Each topic is a question an auditor will ask of any agent record, and the products differ most on these.

1. Who writes the record. This matters most: a record the agent sends about itself is its own account; a record the platform writes from outside the agent's code is a witness.

  • Six products write a step-by-step record on the platform side: Gemini Enterprise, Microsoft Purview, Foundry's server-side tracing, Salesforce, Oracle and Snowflake's native agents.
  • Three keep platform records that are not step-by-step: Workday's transaction audit, SAP's backend action logs and IBM's scheduled evaluation scores.
  • Everything else, AgentCore included beyond its one entry per request, records what the agent's own code sends.

2. What level the record sits at. Step logs, business transactions and averages answer different questions. A step log shows what the agent did. A transaction audit shows what changed in the business system and under whose name; averages show whether controls hold overall.

  • AgentCore's runtime records the request but "does not capture the agent response".
  • Workday records the transaction but documents no step-by-step log of model or tool calls.
  • IBM records evaluation results against thresholds, not individual actions.

3. Whether content is captured. Content means prompts, tool inputs and results. Without it, a record proves that a tool ran but not what it did; with it, the record holds personal data and needs its own access control.

  • OpenTelemetry, the open standard most of these tools use for logs, makes content capture optional, and most SDKs follow. Claude, Microsoft Agent Framework and New Relic's OpenTelemetry route leave it off by default.
  • OpenAI's Agents SDK captures content by default and sends it to OpenAI's servers.
  • Platforms store content apart. Purview keeps a conversation ID while the text sits in the user's mailbox; Gemini Enterprise writes a placeholder unless an admin turns content on, and Snowflake hides content from roles without a specific permission.

4. Whether the record is complete. Evidence has to cover every action that matters, and almost nothing here promises that.

  • Delivery is best effort. Snowflake says so in those words, OpenAI's SDK drops a batch of records that fails to send, and Claude Code "fails silently on export errors by default".
  • Caps cut records short. LangSmith rejects steps past 25,000 per trace, Databricks takes in 200 traces per second per workspace, and CrewAI's buffer drops the oldest steps when full.
  • Some routes are never logged. These include Gemini Enterprise's Agent2Agent protocol (A2A) and Model Context Protocol (MCP) calls, Snowflake's managed MCP server and a built-in completion function, and channels excluded from Copilot Studio logging.

5. How long it is kept. Stated defaults run from 15 to 180 days, and most products state none.

  • Datadog keeps traces 15 days by default, Google log data 30 days, Workday's input and output detail 30 days. Grafana keeps 30 days (a year at most), Foundry's portal shows 90 days, and LangSmith deletes after 180.
  • Regulators ask for more: at least six months under the EU AI Act, three to six years for US broker-dealer records.

6. Who can change or delete it. A record that the platform operator can delete at will is weak support in a dispute.

  • Snowflake's rows "can't be modified after ingestion", but an admin role can delete them. Google's individual log entries cannot be deleted.
  • Databricks traces can be removed with a database command and LangSmith's through its API. Purview allows a retention policy of 0 days for Copilot text.

7. Whose authority the agent acted under. A record may name the agent but not the person it acted for, the mandate or the approver. Then it answers only half of "who did this".

  • Workday records the agent as By User and the human as On Behalf Of User. Entra Agent ID and Gemini Enterprise's SPIFFE IDs (a standard for machine identities) give agents identities of their own.
  • AgentCore takes session and user IDs from whatever the caller sends. Claude's SDK identifies the service's login, not the end user.
  • Approvals are recorded unevenly. Claude Code logs where each permission decision came from; Oracle lets you require approval per tool but does not say it is logged.

8. Whether it leaves the platform in a standard form. An evidence copy has to live somewhere the platform operator cannot reach, so export matters.

  • OpenTelemetry's naming rules for AI (its GenAI conventions) are the common format, still marked as in development. Microsoft Agent Framework follows version 1.37; OpenAI uses its own format.
  • Salesforce exports one session at a time in OpenTelemetry's standard file format (beta). Snowflake exports through database functions, Purview through the Office 365 Management API.

The overview table condenses topics one and five. The vendor subsections after it go through all eight where the documentation allows.

Eight topics that decide what a record is worth

Overview table

Platform Who writes the step-by-step record Retention in the docs
AWS Bedrock AgentCore The runtime logs each request; the steps come from the agent's own code Metrics: "standard" CloudWatch; logs and steps not stated
Google Agent Platform The agent's SDK, under the agent's own service account Log data 30 days by default
Gemini Enterprise The platform (Cloud Audit Logs, usage logs) Not stated
Microsoft Copilot Studio + Purview The platform Audit Standard (length not stated); a 0-day policy for Copilot text is possible
Microsoft Foundry The platform, for prompt and hosted agents; agent-side library optional Your Application Insights settings; portal shows 90 days
Salesforce Agentforce The platform, into Data 360 Not stated
SAP Joule Backend logs, per an SAP Community post Set by the customer's admin
Workday Agent System of Record The platform's transaction audit Input and output: 30 days
Oracle AI Agent Studio The platform Not stated
Snowflake Cortex Agents The platform Not stated
Databricks MLflow Tracing The agent No default stated for raw trace tables
NVIDIA NeMo Agent Toolkit The agent (a code library) Whatever storage you choose
IBM watsonx.governance The platform, collecting evaluation scores on a schedule Not stated
Datadog, New Relic, Dynatrace, Splunk, Grafana The application's own code Datadog: 15 days by default on all plans; Grafana: 30 days, up to a year; others not stated
LangSmith, Phoenix, Langfuse The agent's own logs LangSmith cloud service: 180 days, then deleted
OpenAI Agents SDK, Claude Agent SDK, Microsoft Agent Framework The agent Your storage

New approaches from startups and labs

The platforms compared above mostly record what an agent did and show it on a dashboard. Startups, open-source projects and research labs are trying other ways to see what agents are doing. An open scan in October 2026 found about 15 types of approach, grouped here under 11 headings. Most are young, several companies have already been bought, and many numbers below are the vendors' own.

Watching the model from the inside

Most monitoring reads what the agent writes. A newer idea is to read the model's internal signals directly.

  • Goodfire trains small "probes" on the model's internal activity to flag harmful agent sessions, and only sends flagged cases to a more expensive AI judge. On its tests with the open Kimi K3 model, the setup caught about 93% of harmful sessions while interrupting at most 5.5% of harmless ones. It needs access to the model's internals, through the model operator or a hosting provider that builds it in (Goodfire has a partnership with Baseten); ordinary closed APIs cannot be monitored this way.
  • Reasoning monitors. OpenAI describes monitoring its own internal coding agents, "both the model's actions and their internal reasoning", to find misbehaviour that only shows up in long, realistic sessions. As noted earlier, these monitors were not running on the evaluations behind the July 2026 Hugging Face incident.

Reading thousands of runs at once

A trace viewer shows one run. The harder question is what is going wrong across thousands of them.

  • Transluce Docent uses AI to search and summarise large sets of agent transcripts. In a 2026 study of real coding-agent sessions it documented, for example, an agent that removed a branch protection rule, merged its own change and put the rule back without telling the user.
  • Raindrop watches production runs with small custom models and flags unusual behaviour without tests written in advance. It raised a Series A in September 2026, bringing total funding to US$50 million.
  • Judgment Labs looks for patterns across long agent runs, such as drifting from instructions, and turns them into tests and alerts. It raised US$32 million led by Lightspeed in May 2026.
  • Amplitude, the product analytics company, now offers Agent Analytics: what users ask agents for, where they get frustrated, and how that links to product outcomes.

Reading thousands of runs at once

Agents that investigate agents

Several vendors now put an AI agent on the other side of the trace.

  • Patronus AI's Percival reads a full trace, labels more than 20 kinds of failure and points to the steps where they happened. Patronus raised a US$50 million Series B in June 2026.
  • Microsoft's Azure Copilot Observability Agent lets operators ask questions of their logs in plain language, investigate incidents and, in preview, correlate alerts in the background. Humans stay in control of decisions and actions.

Turning production runs into better agents

Some tools close the loop: what goes wrong in production becomes a test, and the tests drive improvements.

  • Pydantic shows prompts being improved automatically with GEPA, a method that tries prompt variants, scores them against a test set and keeps the best, using its own tracing and evaluation tools.
  • Judgment Labs and Raindrop both pitch the same loop: find failures in production traffic, then turn them into tests.

Turning production runs into better agents

Rewind and replay

Developers want to step back through a run, change one thing and see what happens.

  • Durable execution platforms such as Temporal record every step of a workflow so it can resume after a crash. That record doubles as a history of what ran. Temporal now promotes this for AI agents, and one customer, Manetu, runs each agent thread as a workflow without changing the agent's code, with credentials the agent never sees.
  • Replay tools such as OrcaReplay record agent runs and let a developer replay them. Patronus also lists replay as a capability. As the replay section of this report explains, regenerating a model's output does not give the same answer twice.

Simulated users and worlds

Instead of waiting for production, these tools test agents against simulated users before release.

  • Collinear steers simulated users with "trait" settings (for example, impatient and confused), and reports that its personas stay consistent in 77% of long chats.
  • Coval simulates and evaluates voice and chat agents. It raised a US$28 million Series A in June 2026.
  • Patronus announced "digital world models" for training and simulating agents with its Series B, and Raindrop replays real production traffic against a proposed code change (research preview).

Simulated users and worlds

Watching without changing the agent's code

Instrumenting every agent is slow, and an agent can switch its own instrumentation off. Some tools observe from outside instead.

  • AgentSight, a research system now packaged by Alibaba, uses eBPF, a Linux kernel feature, to watch an agent's AI traffic and its system activity together, without touching the agent's code. Its authors report under 3% performance overhead.
  • Context Lens sits between a coding agent and its model and shows what fills the context window, with options to strip secrets before anything is captured.
  • Agent gateways observe traffic as it passes. Kong's Agent Gateway, generally available since AI Gateway 3.14, sees MCP tool calls and agent-to-agent (A2A) messages; the open-source agentgateway does the same for MCP.

Cost per agent

When one customer request passes through five agents, the monthly bill shows one line. Revenium attributes model costs, paid tools and human review time to each agent, user and outcome, so a team can see "which agent spent what, whose budget absorbed the cost".

Cost per agent

Trying before trusting

Microsoft's Shadow Mode for its Dynamics 365 case management agent runs the agent on live cases "without taking any action": it predicts and recommends, and staff see what it would have done, but no record changes and no customer is contacted. It turns the decision to automate into a measurement instead of a debate.

Rules checked over the whole trace

Researchers are applying formal methods, the mathematics used to verify chips and aircraft software, to agent traces. A paper accepted at an ACM CCS 2026 workshop checks agent runs against rules written in temporal logic ("never send data after reading a secret") and proposes a 12-field trace format that would make such checks possible in production.

Rules checked over the whole trace

Proof, permission and cause

A smaller group works on questions this report has raised throughout: whether a record can be checked by outsiders, whether an action should run at all, and which input caused an outcome.

  • Signed receipts per action: open specifications and small vendors (Agent Receipts, AGLedger, AgenticRail, Headlights, Tracekit) write a signed record per tool call. Where it is written differs: Agent Receipts uses a separate process outside the agent, while Headlights runs as a library inside the agent's workflow. All are early; most say themselves that a signature proves a record is unchanged, not that it is complete or true.
  • Checks before an action runs: security vendors moved from scanning prompts to watching actions, and many were bought quickly: Invariant Labs by Snyk, Lakera by Check Point (about US$190 million), Prompt Security by SentinelOne. Lasso, Zenity, Noma and Pillar remain independent.
  • Delegation chains: Keycard records the chain from the user through each agent, up to five actors, behind every authorisation decision (early access, US$38 million raised in October 2025). Astrix was bought by Cisco.
  • Finding the cause: Tracekit's experimental why command re-runs an agent without one input to see whether the action still happens (6 of 6 causes found in a small test). The research method AgenTracer does the same at scale on benchmarks but has no product yet. Atla, which detected failures step by step, shut down in February 2026.

What the new approaches add up to

The common thread is moving past the single trace. Teams want to see across thousands of runs, look inside the model, test against simulated users, watch from outside the agent's code and know what each agent costs. Most of these tools are under two years old, and their published numbers come from their own tests.

What the new approaches add up to

Building blocks for an evidence record

The parts for an evidence record exist as open standards and cloud features. This section covers the state of each: integrity, storage and deletion, and joining traces across platforms.

Integrity, step by step

Each level below makes tampering harder to hide. A hash chain means each entry carries a fingerprint of the one before, so any change shows. A signature shows which key holder vouched for the latest entry (the "head" of the chain).

Level What it adds Cost
Add-only table Stops accidental edits Trivial
Hash chain over a fixed text format (canonical JSON) Changing any covered field breaks the chain. Rebuilding the chain or cutting off its end needs the steps below to detect Microseconds per entry
Signed checkpoints Shows who vouched for the head A key to protect
External timestamp (RFC 3161) An outside timestamp service stamps the head's fingerprint. A later rewrite shows, even if done by the key holder. Only the fingerprint leaves your machine One request per checkpoint
Merkle log with inclusion proofs (RFC 6962 style) A tree of fingerprints. You can prove one entry is in the log without handing over the whole log More moving parts
External witnesses or a transparency log Outsiders keep copies of the signed heads. Two conflicting heads prove misbehaviour Someone has to check

Integrity is the solved part, as long as someone outside checks the anchors. Completeness is not. Nothing inside a log can show that every action was logged; that has to come from how the system is built.

Storage, retention and deletion

Write-once storage (often called WORM, "write once, read many") keeps the off-site copy honest, but only in the right mode. Each cloud calls it "object lock" or similar and offers two modes. One guards against accidents; the stronger one guards against the account owner.

Store Mode that stops accidents Strongest mode Notes
Amazon S3 Object Lock Governance mode: anyone with the s3:BypassGovernanceRetention permission can override it, and the S3 console sends the override by default Compliance mode: no user can delete, "including the root user"; the only early exit is deleting the AWS account A plain delete request still adds a delete marker. A listing can look empty while the locked version survives
Azure immutable blob storage Unlocked time-based policy: can be shortened or deleted Locked policy: data "can't be modified or deleted by any user, not even by users with account administrative privileges"; it can only be extended Must be locked to count for SEC 17a-4(f). Changes to policies on single versions are not audited
Google Cloud Storage Unlocked retention policy: can be reduced or removed Bucket Lock: "irreversible", retention can only grow; the project gets a lien that blocks its deletion The policy cannot be removed or reduced. A project owner or org admin can remove the lien; whether that then allows deleting a project with unexpired objects is not documented. Object metadata stays editable

Cohasset Associates has assessed S3 Object Lock and Azure immutable storage against SEC 17a-4, FINRA and Commodity Futures Trading Commission record-keeping rules. Google's documentation says Bucket Lock "can help" with the same rules and names no assessor.

Of the modes above, only S3 compliance mode and a locked Azure policy resist the account's own administrators.

A locked store proves the copy was not changed after it arrived. It says nothing about what happened before the upload. That is why the chain head also gets an external timestamp.

The SEC's 2022 amendment to Rule 17a-4(f) matters outside finance too. It added an alternative to write-once storage: keep a "complete time-stamped audit trail" of every change and deletion, with who made it, so the original can be recreated. That is the closest thing in law to a specification for an agent action log, and it was written before agents existed.

An add-only log and a legal right to erasure look incompatible. The IETF audit-trail draft answers with a tombstone. When a record must be deleted, for example under Article 17 of the EU's General Data Protection Regulation (GDPR), it "MUST be replaced with a tombstone record that preserves chain integrity". A known way to make stored content unreadable without rewriting it is cryptographic erase: deleting the keys that protect the data.

Storage, retention and deletion

Joining traces across platforms

An incident that crosses two platforms starts with joining their traces. To do that, each call passes along trace context: a small set of IDs that says "this step belongs to that trace". The plumbing for passing it on is only half built.

Hop How trace context travels, October 2026 Status
Between web services (HTTP) The W3C standard headers traceparent and tracestate Stable and widely supported, when both ends pass it on
Agent to MCP server (the server that offers the agent its tools) The same trace fields inside a metadata field of each request, without a prefix, under MCP enhancement proposal SEP-414 In the OpenTelemetry MCP conventions (still in development); MCP version 2026-07-28 keeps the same field names
Agent to agent over A2A (Agent2Agent protocol) No standard. A proposal (A2A issue #2103, 4 August 2026) puts the trace ID in message metadata under an extension key Open. At A2A v1.0.1, the calls for checking or following a task have no metadata field, so polling and push updates would still break the chain
Into a hosted platform Varies. AgentCore accepts the W3C header, AWS X-Ray headers and the MCP session ID. Microsoft Agent Framework's passing-on does not reach MCP tools hosted by the provider Per vendor
  • Context can be faked. OpenTelemetry's own documentation warns that malicious actors can send forged trace headers. A traceparent helps you join records; it does not prove two events belong together.
  • Some hops are not logged at all. Gemini Enterprise lists its A2A task methods and MCP request methods among those that produce no audit logs.

Where this is going

The dates that matter for agent records over the next two years:

Date What happens
13 October 2026 IETF draft-sharif-agent-audit-trail-08 replaces version -07. It is still an individual submission; the IETF has not endorsed or adopted it
1 January 2027 Colorado SB 26-189 and California's rules on automated decision-making apply
2027 The Monetary Authority of Singapore plans to consult on further guidance for agentic AI. NIST plans to finalise a set of security controls tailored to AI agents (a "control overlay")
7 October 2027 The Monetary Authority of Singapore's Guidelines on AI Risk Management take effect for financial institutions in Singapore. Controls across the AI lifecycle follow by 7 October 2028
2 December 2027 EU AI Act rules for high-risk systems listed in Annex III apply. They include record-keeping (Art. 12) and log retention for providers (Art. 19) and for deployers (Art. 26(6))
2 August 2028 EU high-risk rules for AI built into regulated products (Annex I) apply
Not dated OpenTelemetry GenAI conventions become stable; a standard way to pass trace context between agents over A2A; built-in trace headers for MCP over Streamable HTTP; an EU harmonised standard on AI logging

Platforms are taking over the recording. Foundry traces hosted agents on the server side. Snowflake logs native agents into an event table nobody can edit, Salesforce writes step records from its runtime, and Workday records agent and human as separate actors. A record written by the platform is a better witness than one written by the agent, but it is still the platform's own record.

Identity is moving faster than evidence. Entra Agent ID, Workday's agent accounts, SPIFFE IDs (a standard format for machine identities) in Gemini Enterprise logs and the IETF draft's mandatory principal field all answer "who acted, on whose behalf". Fewer answer "under which rule, approved by whom, and can you prove it".

Regulators agree on logging and monitoring, not on how to build it. The EU asks providers and deployers of high-risk AI systems to keep the automatic logs under their control for at least six months. The Monetary Authority of Singapore asks for monitoring and records without requiring tamper-proofing.

The Infocomm Media Development Authority goes furthest: where possible, it asks for logs generated or confirmed outside the agent's control. None of them says how to build this. After an incident, what decides a record's weight is whether the system it describes could have changed it.

For now, platform traces show what happened. Proving it still takes records that neither the agent nor its operator can change, and few products write those.

Reference: what each vendor records

The subsections below start where enterprise agents are hosted and end with the tools teams add on top. First come the big cloud providers (hyperscalers) and business suites, then data platforms. Then application performance monitoring (APM) tools and tools built for large language models (LLMs), then the software development kits (SDKs) that sit inside the agent.

Read in that order, the recorder moves closer to the agent it records.

Hyperscalers

AWS Bedrock AgentCore

  • Platform writes: one span (one step in the trace) for each call to the agent, with account, agent, session and error details. It also writes application logs with the request. Not the answer: the runtime "does not capture the agent response", so the records have no response field.
  • Agent writes: reasoning and tool steps, through AWS's own version of OpenTelemetry built into the agent. So completeness depends on that code's coverage. One setting, DISABLE_ADOT_OBSERVABILITY=true, removes the default logging.
  • Kept: in CloudWatch, AWS's monitoring service, with logs also sendable to S3 or Firehose. Session and user IDs are whatever the caller puts in the request.
  • Third party: a published security flaw, CVE-2026-15737. Versions 1.4.8 and 1.5.0 of the AgentCore Python SDK wrote raw prompts and full responses into the trace without masking them.
  • Watch for: AWS says telemetry is "provided for monitoring purposes" and may differ from billed usage. Two AWS pages disagree on whether gateway logging is on by default.

Google Agent Platform and Gemini Enterprise

These are two products with two ways of recording.

  • Agent Platform (formerly Vertex AI Agent Engine): the agent's SDK writes the trace under the agent's own service account. Prompts, responses and user IDs are captured only with the setting OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. Content goes to a Cloud Storage bucket as JSON Lines files, on GenAI conventions 1.37.0 or later.
  • Kept: log data 30 days by default. Individual log entries cannot be deleted; conversations in the bucket can.
  • Gemini Enterprise: the platform writes audit logs. Creating, changing and deleting an agent count as Admin Activity, which is always logged. Assistant turns count as Data Access, which is off by default.
  • Watch for: A2A task calls, MCP requests and one widget action "don't produce audit logs", because of volume or "low auditing value". Usage logs carry a SPIFFE ID as the agent's identity. But the model field misses the model of any agent a turn was handed to.

Microsoft Copilot Studio, Purview, Foundry and Agent 365

Microsoft's record looks the most like an audit record of the three.

  • Purview audit: the platform writes a CopilotInteraction record in a fixed layout. It holds the agent ID and version, the resources accessed with the action and sensitivity label, and per-message flags for prompt injection hidden in documents and for jailbreak attempts. It names the model, which can be missing when the Auto setting picks an internal model.
  • Content: the record holds a conversation ID, not the text. The text sits in the user's mailbox and is reached through eDiscovery, Microsoft's legal search tool.
  • Watch for: admins can switch off audit logging for the whole tenant or set a 0-day retention policy for Copilot text.
  • Foundry: records prompt and hosted agents on the server side, with no code changes (generally available; workflows and outside agents in preview). Traces sit in your Application Insights, Microsoft's monitoring service, under your retention settings.
  • Identity: Entra Agent ID gives agents their own identities. Agent 365 adds a register of agents, but its activity data needs a licence, accepted terms and the Frontier preview programme.

Business suites

The suites have an edge the hyperscalers lack. The agent runs inside the business system, so the platform can record the business action and not only the model call.

Business suites

Salesforce Agentforce

Salesforce has the most complete step record of any platform here.

  • Platform writes: five Session Tracing record types in Data 360: session, participant, interaction (one turn), interaction step (a tool, Apex code, Flow automation or LLM call) and message. A separate audit layer at the point where calls reach the model keeps prompt and response, masked and unmasked, with toxicity and prompt-defence results.
  • Vendor guidance: "use the native session-tracing model over custom-built logging".
  • Export: an OpenTelemetry interface (beta from 10 April 2026) returns one session per call, in OpenTelemetry's standard format. You have to fetch each session; it is not sent to you as it happens.
  • Watch for: capture must be switched on for each Salesforce org before anything is recorded. No rules for retention, editing or deletion are documented. Session outcomes such as "deflected" are judged by an LLM "using an editable prompt for each metric", which makes them opinions, not facts.

SAP Joule

  • Source quality: the most specific claims come from an SAP Community post (August 2026), not from product documentation.
  • Recorded, per that post: conversation logs (what the user said, responses, feedback, timestamps, session details), kept as long as the customer sets. Also "transaction logs for all executed actions" through official product APIs, under the user's single sign-on (SSO) identity. Whether those logs show that Joule, not the user, made the change is not stated.
  • Platform audit: the SAP AI Core audit log covers operations such as deployments and stored secrets, not agent steps. A monitoring service in the Q2 2026 release records prompts and responses to S3 storage.
  • Watch for: SAP AI Agent Hub is a list of agents with an approval badge. Monitoring of running agents is announced as "looking ahead".

Workday Agent System of Record

  • Identity: each agent gets its own system account, up to two. One acts on behalf of a user; one does background work on its own.
  • Audit log: the agent appears as By User and the human as On Behalf Of User. So the trail shows "when an agent acts autonomously and when it acts on behalf of a user". Most other platforms leave that to whatever the caller sends.
  • Watch for: input and output detail covers "the past 30 days" and only agents Workday built. The documentation shows no step-by-step log of model or tool calls. The "cryptographically signed" claim belongs to Agent Passport's verification stamps, not to the logs.

Oracle AI Agent Studio

  • Platform writes: a step-by-step trace per session (worker agents, LLM calls, tools) with timing, status and tokens used.
  • Watch for: the guide says the data sent to the LLM and to other systems is visible "in evaluation runs". Whether live traces keep it too decides whether you know only that a tool ran, or also what it sent.
  • Approvals: you can require human approval per tool or per workflow step. No document says the approval is logged. Dashboard figures depend on a scheduled summary job the customer must run.

Data platforms

Snowflake Cortex Agents

Snowflake comes closest to an evidence store by default.

  • Platform writes: native agents are logged into one table, SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS. It holds planning, every tool call including MCP connector calls, database queries, inputs and outputs, feedback, and the agent version for each turn.
  • Integrity: "Rows can't be modified after ingestion." An admin role can still delete them.
  • Watch for: "trace delivery is best effort", so a missing row proves nothing. Calls through the Snowflake-managed MCP server, the built-in completion function and model calls through the REST API write no traces.
  • Access: content is hidden from roles without a specific permission to read it unredacted. That controls who sees the evidence, not whether it exists. Custom apps traced with TruLens write their own logs, like any SDK.

Databricks MLflow Tracing

  • Agent writes: traces come from inside the agent, through automatic logging, the SDK or any OpenTelemetry tool. The agent can switch them off with one function call or the setting MLFLOW_TRACING_ENABLED=false.
  • Governance: traces stored as Unity Catalog Delta tables inherit access control, masking, row filters, retention policies and audit logs. That is the best governance story in this group.
  • Third party: a published security flaw, CVE-2026-8147. MLflow before 3.14.0 let any logged-in user read, delete and change traces on experiments they had no access to.
  • Watch for: traces are removed "directly from the underlying Unity Catalog tables using SQL", and the account that writes them also has the right to change them. No default retention is stated for the raw trace tables. Databricks' redaction pipeline adds an expiry you can set, with up to six days of slack before deletion.
  • Limits: intake is capped at 200 traces per second per workspace. A trace that times out leaves the agent running with no trace.

NVIDIA NeMo Agent Toolkit and NeMo Guardrails

  • Agent Toolkit: a code library that sends each intermediate step to other tools (OpenTelemetry, Phoenix, Langfuse, LangSmith, Weave, files and others). It has no storage of its own. Its evaluation interface is experimental, and replaying a past run offline is something the profiler "will allow".
  • Guardrails: records which safety rules fired and returns PASSED, MODIFIED or BLOCKED. It adds an explicit failure flag, so "a log consumer reads a verdict rather than inferring one from a missing key".
  • Watch for: NVIDIA is direct about the gaps. Do not assume a decision "automatically includes a reason code, policy version, tenant ID, or replayable event schema".

IBM watsonx.governance

  • Platform writes: Enforcement Tracking (announced 11 August 2026) collects agent evaluation scores such as hallucination, helpfulness and toxicity from watsonx Orchestrate on a schedule. It checks them against thresholds and records the result for auditors.
  • Watch for: that is evidence that controls hold on average. It is not a record of what an agent did in a given transaction.

APM vendors

APM vendors take in what the application sends them. Their value is linking that data to the servers and networks underneath; their limit is that the agent is still the author.

APM vendors

Datadog

  • Recorded: LLM operations and token use from traces that follow OpenTelemetry GenAI conventions 1.37 or later, or from Datadog's own SDK; inputs and outputs when the instrumentation captures them.
  • Kept: traces "retained for 15 days on all plans by default". Longer retention for live systems depends on the plan.
  • Content: Sensitive Data Scanner can find and hide sensitive data in any step.

New Relic

  • Recorded: "every LLM call made by your service — prompt, completion, model, latency, token counts, finish reason", plus tool calls and agent steps in the request trace. Whether prompts and completions are actually kept depends on the route and settings (see Content).
  • Written by: the application, either through OpenTelemetry ("No New Relic agent is required") or through New Relic's own monitoring add-ons for each programming language.
  • Content: off by default on the OpenTelemetry route. The add-on route records "a streaming copy of inputs and outputs", and New Relic leaves getting consent for that to the customer.
  • Watch for: the add-on route caps how many samples it stores per collection cycle, so it does not promise every event. Retention is not stated.

Dynatrace

  • Recorded: "agent execution paths, tool invocations, and inter-agent communication", with metrics for agent and tool calls; prompts appear on each step when the instrumentation supplies them.
  • Standards: recommends OpenInference, an open format for AI traces, when sending OpenTelemetry data. It also accepts GenAI conventions directly, with stricter rules on required fields than the standard itself.
  • Audit: an "audit trail" dashboard reads business events named gen_ai.auditing. Who sends those events is not documented.
  • Watch for: guardrail signals are recorded, not enforced. Dynatrace states the rule behind this: "Downstream transforms can't reconstruct data that instrumentation discarded."

Splunk (with Galileo)

  • Ownership: Cisco bought Galileo in 2026 for Splunk's AI Agent Monitoring.
  • Recorded: "every agent, tool call and handoff alongside quality, cost and performance metrics", including system prompts in conversation details.
  • Evals: "Evaluation and guardrails run on every interaction in production, not a sampled slice."
  • Access: the only APM vendor here that documents a role for reading content, read_apm_ai_conversation.
  • Watch for: retention is not stated. Captured content can exceed storage size limits.

Grafana Cloud Agent Observability

  • Recorded: agent sessions as "a new, fifth telemetry signal": system prompts, input and output messages, tool definitions, calls and results, tokens and timing. Generally available since 30 July 2026.
  • Kept: content "for 30 days by default", at least 14 days and at most one year on request.
  • Content: Grafana's own coding-agent connectors capture only metadata by default. The OpenLIT SDK, used with its older connector, captures message content by default. Secrets it finds are replaced with a redaction marker that names the kind of secret.
  • Watch for: sending plain OpenTelemetry data is still in public preview. One year is shorter than any of the US record-keeping periods covered under What regulators ask for.

LLM-native and open-source tools

All four tools below write what the agent sends them; none is built to hold evidence.

Arize Phoenix

  • Ownership and adoption: part of Dynatrace since 1 October 2026. It has 11,880 GitHub stars, 0.65 million Python downloads a month and the Elastic License 2.0.
  • Recorded: the agent's own OpenTelemetry logs, converted cleanly to OpenInference on arrival.
  • Watch for: the code review in Debugging trace or evidence record found that it confirms receipt before the data is safely saved, overwrites notes and scores in place, deletes linked records along with the parent, and ships with login turned off. Its "Replay" feature calls the live model again.

Langfuse

  • Ownership and adoption: part of ClickHouse since January 2026. It has 35,948 stars and 26.7 million Python downloads a month, the most of any platform you host yourself.
  • Recorded: the sender's events, stored first as an untouched copy, which is a sound way to take data in.
  • Watch for: it confirms receipt once that copy is stored but before processing. The last write wins, deletion is permanent, and nothing detects tampering. Its default setup file (Docker Compose) ships with an all-zero encryption key and a sample secret.

LangSmith

  • Recorded: every model call, tool call and lookup as a "run", grouped into traces and threads. It accepts OpenTelemetry, OpenInference and other formats.
  • Kept: the cloud service keeps traces "for 180 days from ingestion. After that, traces are permanently deleted", unless copied into a dataset.
  • Watch for: a trace is capped at 25,000 runs, and later runs are rejected. A published flaw, CVE-2026-41182: the SDK's output masking missed answers sent piece by piece, until JS 0.5.19 and Python 0.7.31. Deleting threads and memories through the API is a documented feature.

Microsoft Agent Governance Toolkit

  • Status: public preview, 6,434 stars within seven months, no release since June 2026.
  • Recorded: decisions and approvals, from inside the agent. Each approval is tied to a fingerprint of the exact action, expires, and can be used only once.
  • Watch for: its log chain (each entry carries a fingerprint of the one before) restarts whenever the log file is rotated. The plugin log throws out old entries past 10,000, the signing key sits where the agent can read it, and decision fields are left out of the fingerprint. Its LIMITATIONS.md file says much of this itself.

Feed these tools from the same pipeline that feeds the evidence store, not the other way round.

Agent SDKs and frameworks

Every SDK trace here comes from inside the agent, so the default settings decide what exists at all.

OpenAI Agents SDK

  • Recorded: LLM answers, tool calls, handoffs, guardrails and custom events, in OpenAI's own format.
  • Defaults: "Tracing is enabled by default". Traces go to OpenAI's servers with the same API key, and the setting trace_include_sensitive_data is on. Tracing is unavailable under Zero Data Retention.
  • Watch for: "A failed batch is dropped." If a masking step fails, later export steps still receive the data.

Claude Agent SDK and Claude Code

  • Recorded: steps for each interaction, model request, tool and hook (beta), plus events. One event records whether a permission came from configuration, a hook or a user. The documentation calls these events "the audit data source".
  • Defaults: off until you set CLAUDE_CODE_ENABLE_TELEMETRY=1. Data goes "only to the OTel endpoint you configure, never to Anthropic", and is "structural by default", with content optional.
  • Watch for: "The CLI fails silently on export errors by default." In a service with many users, the identity fields "identify your service's credential, not the end user on whose behalf the agent acted".
  • Server side: Anthropic's API tracing is written outside the agent, the only recorder of that kind in this group. It sees calls to the API, not tools run on your own machines.

Microsoft Agent Framework

  • Recorded: steps for agent calls, chat and tool runs, on GenAI conventions version 1.37, with token and timing metrics.
  • Defaults: logging on, "We do not install exporters by default", sensitive data off.
  • Watch for: the link between steps is lost when the agent calls MCP tools hosted by the model provider.

LangGraph with LangSmith

  • Recorded: runs, traces and threads, as above. The Agent Server also saves checkpoints and memories, which can be encrypted with a key you set.
  • Defaults: switched on with LANGSMITH_TRACING. LangSmith is the default destination, with any OpenTelemetry tool as the alternative.

CrewAI

  • Recorded: agent decisions, task timelines, tool use and LLM calls, with links to pause and resume for human review.
  • Defaults: hosted tracing uploads nothing before you consent. Separate usage data about the framework, collecting tool names and agent roles, is on until you turn it off.
  • Watch for: the local buffer holds 1,000 steps or 8 MiB, and "Overflow drops the oldest spans."

An SDK trace is the agent's account of itself, delivered on a best-effort basis. Keep it and use it, but do not make it the only witness.


Sources

Research on agent self-reports, replay and attribution

Market, adoption, surveys and incidents

New approaches

Standards

Regulators

Platforms (documentation as collected October 2026)

Source code reviewed

  • Arize Phoenix 3c47632, Langfuse 6c1fd54 and Microsoft Agent Governance Toolkit 8ef95fd, read October 2026.

Read more