awesome-agent-observability

Awesome Agent Observability

A neutral list of resources for observing AI agents in development and production: standards, instrumentation, frameworks, platforms, gateways, evaluation, cost, safety signals, and on-chain agents and payments, with an OpenTelemetry focus.

The list covers every kind of agent, from coding and customer support agents to enterprise workflows, research and agents that make payments, and every place they run, from cloud providers and enterprise vendors to open source projects. It holds 441 entries.

Open source and commercial entries are listed side by side, in alphabetical order inside each section, never by preference or payment. There is no sponsored placement. Inclusion says nothing about quality, and the list takes no side between competing projects. Disclosure: the maintainer of this list also builds hashspan, which is listed under On-chain agents and payments like any other entry, and reported some of the issues listed under Findings. Entries submitted by people who work on the project carry a vendor_affiliation field in their data file.

Legend

Short tags after an entry:

• OTel - sends or receives OpenTelemetry data (OTLP), or emits telemetry through the OpenTelemetry API, according to the linked evidence.
• GenAI semconv, OpenInference - follows these semantic conventions.
• OSS - the license of the repository root is OSI-approved. Open-core - also shown when the repository has a source-available or enterprise directory such as ee/. Source-available - the root license is not OSI-approved (for example ELv2, BSL or SSPL). Proprietary - closed source or a hosted product.
• Self-hosted, SaaS - can run on your own infrastructure, or is offered as a hosted service.
• Open, Fixed in X - state of a finding.

A missing tag means the point was not confirmed, not that the answer is no. Every entry is a data file in entries/ with more fields: the evidence link for OTel support, the license, the kinds of agents it is specific to (agent_types), and repository metadata refreshed by a script (stars, archived, last push, last release). The same data is in entries.json and llms.txt for programs and agents.

Use the data

The list is also data, for scripts and agents:

• entries.json: every entry with every field, plus the categories.
• llms.txt: the list as plain text for language models.

Examples with jq:

# Self-hosted open source tools that send or receive OTLP
jq -r '.entries[] | select(.historical_section | not) | select(.otel and (.tags | index("self-hosted")) and (.tags | index("oss"))) | .name' entries.json

# Python instrumentation that follows the OpenTelemetry GenAI semantic conventions
jq -r '.entries[] | select(.category == "instrumentation" and (.tags | index("python")) and (.tags | index("genai-semconv"))) | "\(.name) \(.url)"' entries.json

# How each agent framework emits telemetry, with the page it was read from
jq -r '.entries[] | select(.category == "frameworks") | "\(.name): \(.telemetry) \(.telemetry_docs // "")"' entries.json

A missing or unknown value means the point was not confirmed, not that the answer is no.

Contents

• Standards and specifications
• Instrumentation
• Agent frameworks
• Agent protocols and tool servers
• Collectors and pipelines
• Platforms
  • LLM and agent observability platforms
  • General APM with GenAI support
  • Trace backends and UIs
  • Cloud provider offerings
  • Model serving
• Gateways and proxies
• Evaluation and testing
• Cost and usage
• Safety signals
• On-chain agents and payments
  • Transaction tracing
  • Payment protocols
  • RPC and node observability
  • Wallets and smart accounts
  • Local and test chains
• Findings
• Benchmarks and datasets
• Learning
  • Articles and guides
  • Books
  • Talks and videos
  • Papers
  • Courses
  • Case studies
  • Newsletters and podcasts
• Community
• Related lists
• Archived and historical

Standards and specifications

Semantic conventions, wire protocols and context propagation standards that agent telemetry is built on.

• A2A distributed tracing proposal - Open proposal in the A2A repository for semantic conventions and W3C Trace Context propagation between delegating agents.
• A2A Protocol - Specification of the Agent2Agent protocol for communication between agents, relevant to propagating context between them. OSS
• A2A protocol documentation - Documentation site of the Agent2Agent protocol.
• GenAI semantic conventions for agentic systems (issue 35) - Tracking issue for agentic system conventions in OpenTelemetry GenAI.
• MCP specification release candidate 2026-07-28 - Announcement of the MCP release candidate, which includes trace context propagation (SEP-414) and OpenTelemetry-compatible telemetry.
• Model Context Protocol specification - Specification of the protocol that connects agents to tools and data sources.
• OpenInference specification - Semantic conventions for tracing LLM, retriever, tool and agent steps on top of OpenTelemetry. (Source) OpenInference OSS
• OpenTelemetry GenAI agent spans - Semantic conventions for spans that describe agent creation and invocation.
• OpenTelemetry GenAI Anthropic conventions - Provider-specific attributes for Anthropic calls within the GenAI conventions.
• OpenTelemetry GenAI attribute registry - Reference list of the gen_ai.* attribute names and their types.
• OpenTelemetry GenAI events - Semantic conventions for events that carry prompts and completions.
• OpenTelemetry GenAI MCP conventions - Semantic conventions for Model Context Protocol operations.
• OpenTelemetry GenAI metrics - Semantic conventions for token usage and operation duration metrics.
• OpenTelemetry GenAI OpenAI conventions - Provider-specific attributes for OpenAI calls within the GenAI conventions.
• OpenTelemetry GenAI semantic conventions - Entry page of the conventions for generative AI spans, metrics and events.
• OpenTelemetry GenAI spans - Semantic conventions for spans that describe model calls.
• OpenTelemetry JSON-RPC conventions - Semantic conventions for JSON-RPC calls, the transport of MCP and of blockchain nodes.
• OpenTelemetry Protocol (OTLP) - Specification of the wire protocol for traces, metrics and logs. OTel
• OpenTelemetry semantic conventions repository - Source repository of all semantic conventions, including the GenAI ones. OSS
• OpenTelemetry semantic-conventions-genai repository - Repository of the OpenTelemetry semantic conventions for generative AI: spans, metrics, events, MCP and provider conventions.
• OpenTelemetry Weaver - Tool for defining, validating and evolving telemetry schemas built on semantic conventions, with a live check of OTLP data. OSS
• SEP-414: OpenTelemetry trace context in MCP - MCP proposal that defines traceparent, tracestate and baggage keys in the request metadata for W3C Trace Context propagation.
• W3C Baggage - Specification for propagating application key-value pairs across service boundaries.
• W3C Trace Context - Specification of the traceparent and tracestate headers used to correlate distributed traces.
• W3C Trace Context Level 2 - Newer level of the Trace Context specification.

Back to top

Instrumentation

Libraries and SDKs that produce telemetry from LLM clients, agent frameworks and services.

• Arize OTel Python helper - Python helper that registers OpenTelemetry export to Arize with OpenInference. OTel OpenInference OSS
• AWS Distro for OpenTelemetry Python instrumentation - Python auto-instrumentation distribution from AWS with generative AI instrumentation. OTel GenAI semconv OSS
• Axiom AI SDK - TypeScript SDK from Axiom for instrumenting and evaluating AI applications. OSS
• Elastic Distribution of OpenTelemetry Python - Python distribution of OpenTelemetry maintained by Elastic. OTel OSS
• Galileo Python SDK - Python package that sends traces of LLM applications and agents to a Galileo log stream. OSS
• HoneyHive Python SDK - Python SDK for HoneyHive tracing built on OpenTelemetry. OTel OpenInference
• Langfuse JS/TS SDK - JavaScript client for sending traces to Langfuse, built on OpenTelemetry. OTel OSS
• Langfuse Python SDK - Python client and decorators for sending traces to Langfuse, built on OpenTelemetry. OTel OSS
• Logfire - SDK that sends OpenTelemetry traces and metrics, with integrations for LLM clients and agent frameworks. OTel OSS
• Maxim Python SDK - Python SDK for Maxim tracing and evaluation. OSS
• Microsoft.Extensions.AI libraries - Documents .NET AI abstractions that include OpenTelemetry middleware for chat clients. (Source) OTel OSS
• Monocle - Python library that traces GenAI apps and agents with automatic instrumentation and exports spans over OTLP, with an optional GenAI semantic conventions mode. OTel GenAI semconv OSS
• New Relic OpenTelemetry examples - Examples for sending OpenTelemetry data to New Relic. OTel OSS
• OpenInference - Instrumentation packages for LLM clients and agent frameworks that follow the OpenInference conventions. OTel OpenInference OSS
• OpenLIT - SDK and tooling that instruments LLM calls and agents and exports OpenTelemetry data. OTel GenAI semconv OSS
• OpenLLMetry - Python instrumentation for LLM clients, vector databases and frameworks on top of OpenTelemetry. OTel OSS
• OpenLLMetry for Go - Go counterpart of OpenLLMetry. OTel OSS
• OpenLLMetry for JavaScript - JavaScript and TypeScript counterpart of OpenLLMetry. OTel OSS
• OpenLLMetry Ruby - Ruby sibling of OpenLLMetry for tracing LLM applications. OTel OSS
• OpenTelemetry .NET - OpenTelemetry SDK and API for .NET. OTel OSS
• OpenTelemetry contrib GenAI utilities for JavaScript - Shared helpers for GenAI instrumentations in the JavaScript contrib repository. OTel GenAI semconv OSS
• OpenTelemetry contrib OpenAI instrumentation for JavaScript - Instrumentation of the OpenAI client library for Node.js. OTel GenAI semconv OSS
• OpenTelemetry eBPF instrumentation - Zero-code instrumentation of HTTP and gRPC services through eBPF. OTel OSS
• OpenTelemetry GenAI OpenAI Agents instrumentation for Python - Instrumentation of the OpenAI Agents SDK following the GenAI conventions; replaces the deprecated contrib package openai-agents-v2. OTel GenAI semconv OSS
• OpenTelemetry GenAI OpenAI instrumentation for Python - Instrumentation of the OpenAI client library following the GenAI conventions; replaces the deprecated contrib package openai-v2. OTel GenAI semconv OSS
• OpenTelemetry Go - OpenTelemetry SDK and API for Go. OTel OSS
• OpenTelemetry Java - OpenTelemetry SDK and API for Java. OTel OSS
• OpenTelemetry Java instrumentation - Java agent and libraries that instrument applications without code changes. OTel OSS
• OpenTelemetry Java instrumentation: OpenAI - Java instrumentation folder for OpenAI client libraries. OSS
• OpenTelemetry JavaScript - OpenTelemetry SDK and API for JavaScript and TypeScript. OTel OSS
• OpenTelemetry JavaScript contrib - Community instrumentations for the JavaScript SDK. OTel OSS
• OpenTelemetry Python - OpenTelemetry SDK and API for Python. OTel OSS
• OpenTelemetry Python contrib - Community instrumentations for the Python SDK, including a GenAI directory. OTel OSS
• OpenTelemetry Python GenAI instrumentations - Repository of the OpenTelemetry instrumentation packages for generative AI client libraries and agent frameworks in Python. GenAI semconv OSS
• OpenTelemetry Python GenAI utilities - Shared utilities for Python GenAI instrumentations. OSS
• opentelemetry-hooks - Hooks that record activity of several coding agents as OpenTelemetry data. OTel GenAI semconv
• Splunk OpenTelemetry Python distribution - Splunk distribution of OpenTelemetry Python, used to instrument agent applications for Splunk Observability Cloud. OTel OSS

Back to top

Agent frameworks

Frameworks for building agents, with a note on how each one emits telemetry.

• AG2 - Multi-agent framework whose classic API derives from AutoGen, with OpenTelemetry tracing that follows the GenAI agent span conventions. (Telemetry) OTel GenAI semconv OSS
• AgentScope - Multi-agent framework in Python with OpenTelemetry-based tracing sent to a collector or compatible backend. (Telemetry) OTel OSS
• Agno - Python framework for agents and multi-agent teams that traces runs with OpenTelemetry through the OpenInference instrumentation. (Telemetry) OTel OpenInference OSS
• AutoGen - Microsoft multi-agent framework whose tracing is built on OpenTelemetry and follows its semantic conventions for agents and tools. (Telemetry) OTel GenAI semconv OSS
• BeeAI Framework - Framework for building agents in Python and TypeScript, traced in Python through an OpenInference instrumentor. (Docs) OTel OpenInference OSS
• Claude Code monitoring - Documentation of the OpenTelemetry metrics and events that the Claude Code terminal agent exports. OTel Proprietary
• Cline - Open source coding agent for the editor, with telemetry and OpenTelemetry export documented for enterprise setups. (Docs) OTel OSS
• CrewAI - Multi-agent framework in Python whose built-in tracing exports OpenTelemetry spans, with integrations for other tracing tools. (Telemetry) OTel OSS
• Dify - Platform for agentic workflows with integrations to tracing backends such as Opik, Langfuse and Arize Phoenix. Source-available Self-hosted
• DSPy - Python framework for programming language model pipelines, with a callback interface for custom logging and tracing through MLflow autolog. (Telemetry) OSS
• Embabel - JVM agent framework whose user guide documents observation of its services through Micrometer. (Docs) OSS
• Gemini CLI telemetry - Documentation of the OpenTelemetry traces, metrics and logs of the Gemini CLI terminal agent, with GenAI conventions. (Source) OTel GenAI semconv OSS
• Genkit - Framework for AI-powered applications that collects traces and metrics through OpenTelemetry instrumentation. (Telemetry) OTel OSS
• GitHub Copilot OpenTelemetry monitoring - GitHub documentation on sending OpenTelemetry data from Copilot clients to an OTLP-compatible backend. OTel Proprietary
• Google Agent Development Kit for JavaScript - Agent framework for JavaScript and TypeScript with an OpenTelemetry telemetry setup in its core package. (Telemetry) OTel OSS
• Google Agent Development Kit for Python - Agent framework in Python whose traces implement the OpenTelemetry semantic conventions for GenAI. (Telemetry) OTel GenAI semconv OSS
• GPT Researcher - Autonomous research agent whose README documents tracing of its workflows with LangSmith. OSS
• Haystack - Framework for pipelines and agents with tracing through OpenTelemetry, Datadog, Langfuse and other backends. (Docs) OTel OSS
• Inngest AgentKit - TypeScript toolkit for agent networks with durable execution. OSS
• kagent - Kubernetes-native framework for running agents, with OpenTelemetry tracing of agent, model and tool execution. (Telemetry) OTel OSS
• LangChain - Framework for LLM applications whose callbacks send traces to LangSmith and other tools. (Telemetry) OSS
• LangChain.js - JavaScript and TypeScript version of LangChain. (Telemetry) OSS
• LangChain4j - Java library for LLM applications with observability listeners that follow the GenAI conventions, and Micrometer metrics. (Docs) GenAI semconv OSS
• LangGraph - Library for stateful agent graphs; traces go to LangSmith. (Telemetry) OSS
• LiveKit Agents - Framework for voice agents whose documentation describes OpenTelemetry traces per session, exportable to any OTLP backend. (Source) OTel OSS
• LlamaIndex - Data and agent framework with its own instrumentation module and integrations that include OpenTelemetry. (Telemetry) OSS
• Mastra - TypeScript agent framework with its own tracing and exporters for Langfuse, Datadog and OpenTelemetry-compatible platforms. (Telemetry) OTel OSS Open-core
• Microsoft Agent Framework - Microsoft framework for building agents in .NET and Python that emits traces, logs and metrics per the OpenTelemetry GenAI conventions. (Telemetry) OTel GenAI semconv OSS
• NVIDIA NeMo Agent Toolkit - Open source library for connecting and profiling teams of agents, with OpenTelemetry-based observability. (Docs) OTel OSS
• Open WebUI - Self-hosted chat interface for language models with documented OpenTelemetry trace and metric export over OTLP. (Docs) OTel Source-available Self-hosted
• OpenAI Agents SDK for JavaScript - OpenAI agent SDK for TypeScript with built-in tracing in its own trace format and processors for other destinations. (Telemetry) OSS
• OpenAI Agents SDK for Python - OpenAI agent SDK; tracing uses its own trace format, with processors that can export elsewhere. (Telemetry) OSS
• OpenAI Codex telemetry - Documentation of the opt-in OpenTelemetry log export of the Codex terminal coding agent. (Source) OTel OSS
• OpenClaw - Personal AI assistant whose Gateway exports metrics, traces and logs over OTLP/HTTP through a diagnostics plugin. (Docs) OTel OSS
• Pipecat - Framework for voice and multimodal agents with OpenTelemetry tracing of conversational pipelines. (Docs) OTel OSS
• Pydantic AI - Python agent framework that emits OpenTelemetry data per the GenAI conventions and documents Logfire as one backend. (Telemetry) OTel GenAI semconv OSS
• Quarkus LangChain4j - Quarkus extension for LangChain4j, with an observability guide that covers tracing and metrics. (Docs) OTel GenAI semconv OSS
• Rasa - Framework for text and voice assistants whose documentation covers OpenTelemetry tracing of messages, actions and LLM calls. (Docs) OTel OSS
• Rig - Rust library for LLM applications with tracing that is compatible with OpenTelemetry. OTel GenAI semconv OSS
• Semantic Kernel - Microsoft SDK for agents that emits logs, metrics and traces following the OpenTelemetry semantic conventions. (Telemetry) OTel GenAI semconv OSS
• smolagents - Hugging Face library for code-writing agents, traced with OpenTelemetry through the OpenInference instrumentation. (Telemetry) OTel OpenInference OSS
• Spring AI - Spring framework for AI applications, with Micrometer observations that follow the GenAI conventions. (Docs) GenAI semconv OSS
• Strands Agents SDK - Agent SDK from AWS for Python and TypeScript whose traces use OpenTelemetry and the GenAI attribute names. (Telemetry) OTel GenAI semconv OSS
• Temporal Python SDK - Python SDK for durable workflows, with OpenTelemetry tracing for workflows that often run long-lived agents. (Docs) OTel OSS
• Temporal TypeScript SDK - TypeScript SDK for durable workflows, with OpenTelemetry interceptors for tracing. OTel OSS
• Vercel AI SDK - TypeScript toolkit whose telemetry option emits OpenTelemetry spans with GenAI attributes. (Telemetry) OTel GenAI semconv OSS
• VoltAgent - TypeScript agent framework with tracing in its own VoltOps observability platform. (Telemetry) OSS

Back to top

Agent protocols and tool servers

SDKs and servers for agent protocols such as MCP and A2A, and how they carry traces.

• A2A enterprise features - Page of the A2A documentation that covers tracing, monitoring and auditing for agents that use the protocol.
• A2A Java SDK - Official Java SDK for the A2A protocol, with an OpenTelemetry module for tracing and context propagation in servers and clients. OTel OSS
• A2A Python SDK - Python library for building A2A clients and servers, with optional OpenTelemetry tracing. OTel OSS
• AG-UI - Agent-user interaction protocol with Python, TypeScript and .NET SDKs, where the .NET SDK emits OpenTelemetry spans for each run. OTel OSS
• Datadog MCP server - Managed MCP server that lets agents query logs, metrics, traces and incidents in Datadog. (Source) OSS SaaS
• FastMCP - Python framework for MCP servers and clients with documented OpenTelemetry tracing. (Docs) OTel OSS
• Last9 MCP server - MCP server that lets agents query Last9 telemetry such as logs, traces and metrics. OSS
• mcp-grafana - MCP server that gives agents access to Grafana dashboards, data sources and alerts. OSS
• mcp-otel - Library that reads W3C Trace Context from the MCP request metadata and emits OpenTelemetry spans for MCP servers. OTel OSS
• Model Context Protocol C# SDK - Official C# SDK for MCP clients and servers, with a sample that configures OpenTelemetry for MCP operations. OTel OSS
• Model Context Protocol Python SDK - Python SDK for MCP servers and clients. OSS
• Model Context Protocol TypeScript SDK - TypeScript SDK for MCP servers and clients. OSS
• OpenTelemetry MCP server (Traceloop) - MCP server that lets agents search and read traces stored in Jaeger, Tempo or Traceloop. OSS
• SigNoz MCP server - MCP server that lets agents query SigNoz telemetry. OSS

Back to top

Collectors and pipelines

Collector distributions and components that receive, process, redact and route agent telemetry.

• Bindplane OpenTelemetry Collector - Bindplane build of the OpenTelemetry Collector that can be managed remotely through the OpAMP protocol. OTel OSS
• Dynatrace OpenTelemetry Collector - Collector distribution for sending OpenTelemetry data to Dynatrace. OTel OSS Self-hosted
• Fluent Bit - Telemetry agent that collects and routes logs, metrics and traces, with plugins that read and write OpenTelemetry data. OTel OSS
• Grafana Alloy - Grafana distribution of the OpenTelemetry Collector that also runs Prometheus-style pipelines for metrics, logs, traces and profiles. OTel OSS
• OpenTelemetry Collector - Vendor-neutral agent for receiving, processing and exporting telemetry. OTel OSS Self-hosted
• OpenTelemetry Collector Contrib - Community receivers, processors and exporters for the Collector. OTel OSS Self-hosted
• OpenTelemetry Operations for Python (Google Cloud exporters) - OpenTelemetry Python exporters for Cloud Monitoring and Cloud Trace. OTel OSS
• OpenTelemetry redaction processor - Collector processor that removes attributes not on an allowed list and masks values that match a blocked list. OSS
• OpenTelemetry span metrics connector - Collector connector that aggregates spans into request, error and duration metrics. OSS
• OpenTelemetry tail sampling processor - Collector processor that decides which traces to keep after seeing whole traces, by policies such as errors or latency. OSS
• OpenTelemetry transform processor - Collector processor that modifies telemetry with OpenTelemetry Transformation Language statements. OSS
• Refinery - Sampling proxy that buffers all spans of a trace and then keeps or drops the trace as a whole. OSS
• Vector - Observability data pipeline for logs, metrics and traces, with an OpenTelemetry source and sink. OTel OSS

Back to top

Platforms

Products and backends that store, query and visualize agent telemetry.

LLM and agent observability platforms

Platforms built for tracing, evaluating and monitoring LLM applications and agents.

• Agenta - Workspace for building, evaluating and observing LLM applications. OSS Open-core Self-hosted
• AgentOps - Python SDK and platform for session replay, cost and tracing of agents. OTel OSS
• Arize AX - Hosted platform from Arize for tracing and evaluating LLM applications. OTel OpenInference Proprietary SaaS
• Arize Phoenix - Self-hostable tracing and evaluation tool that ingests OpenInference traces over OTLP. OTel OpenInference Source-available Self-hosted SaaS
• Braintrust - Hosted platform for logging traces, evaluating and monitoring LLM applications, with an open source JavaScript SDK. (Source) Proprietary SaaS
• Databricks: MLflow tracing overview - Documents MLflow tracing on Databricks for generative AI applications and agents. OTel Proprietary SaaS
• Evidently - Framework to evaluate and monitor LLM apps and ML systems, with a trace store and viewer that receives OpenTelemetry traces over OTLP. (Docs) OTel OSS Self-hosted SaaS
• Fiddler documentation - Documentation of Fiddler, a platform for monitoring and governing agents and models. Proprietary SaaS
• Galileo documentation - Documentation of Galileo, a platform for evaluating and monitoring agents, with OpenTelemetry ingest. OTel Proprietary SaaS
• Helicone - Platform for logging requests, cost and sessions of LLM applications. OSS Self-hosted SaaS
• HoneyHive documentation - Documentation of HoneyHive, a platform for tracing and evaluating agents with OpenTelemetry support. OTel Proprietary SaaS
• Laminar - Self-hostable agent observability platform with an OpenTelemetry-native SDK that instruments common LLM frameworks. OTel OSS Self-hosted SaaS
• Langfuse - Self-hostable platform for traces, prompts and evaluations, with an OpenTelemetry endpoint. OTel OSS Open-core Self-hosted SaaS
• LangSmith - Hosted platform from LangChain for tracing and evaluating LLM applications, with OpenTelemetry trace ingestion. OTel Proprietary SaaS
• Langtrace - Tool that records OpenTelemetry traces of LLM calls, vector database queries and framework operations in LLM applications. (Docs) OTel OSS Self-hosted SaaS
• LangWatch - Platform for tracing, evaluating and simulating agents. OTel OSS Open-core Self-hosted SaaS
• Latitude - Agent observability platform that groups failing traces into tracked signals and replays fixes against them. OTel OSS Self-hosted
• Lunary - Hosted platform documentation for monitoring LLM applications, with a JavaScript SDK. (Source) SaaS
• Maxim documentation - Documentation of Maxim, a platform for evaluating and observing agents. Proprietary SaaS
• MLflow Tracing - Tracing feature of MLflow for LLM applications and agents, compatible with OpenTelemetry and the GenAI conventions. (Source) OTel GenAI semconv OSS Self-hosted SaaS
• Opik - Platform for tracing, evaluating and monitoring LLM applications and agents. OTel OSS Self-hosted SaaS
• Snowflake AI Observability with Cortex - Documents Snowflake tracing and evaluation of generative AI applications. Proprietary SaaS
• Traceloop - Hosted platform from the OpenLLMetry maintainers for monitoring LLM applications. OTel Proprietary SaaS
• Weights & Biases Weave - Toolkit for tracing and evaluating LLM applications. OSS

General APM with GenAI support

Application performance monitoring products with features for LLM and agent workloads.

• Axiom LLM observability - Documents Axiom tracing of LLM and agent applications with OpenTelemetry. OTel GenAI semconv Proprietary SaaS
• Coralogix AI observability - Product page for Coralogix monitoring of AI applications and agents. (Docs) OTel GenAI semconv Proprietary SaaS
• Datadog LLM Observability - Hosted product for tracing and monitoring LLM applications and agents, which also accepts OpenTelemetry GenAI instrumentation. OTel GenAI semconv OpenInference Proprietary SaaS
• Dynatrace AI observability - Hosted product documentation for observing AI workloads, with OpenTelemetry ingestion through OpenInference or GenAI conventions. OTel OpenInference GenAI semconv Proprietary SaaS
• Elastic LLM observability - Elastic Observability features for monitoring LLM applications, with OpenTelemetry support through the Elastic distributions (EDOT). OTel Self-hosted SaaS
• GenAI Observability for Splunk - Splunkbase app with dashboards for tokens, latency, cost and errors of generative AI workloads. Proprietary Self-hosted SaaS
• Grafana Cloud AI Observability - Documents Grafana Cloud AI observability dashboards built on OpenLIT instrumentation and OpenTelemetry data. OTel Proprietary SaaS
• Grafana Cloud Claude Code integration - Documents the prebuilt Grafana Cloud integration that receives coding agent telemetry over OTLP. OTel Proprietary SaaS
• Honeycomb - Hosted observability product that ingests OpenTelemetry data. OTel Proprietary SaaS
• Honeycomb AI and LLM observability - Product page for Honeycomb tracing of LLM applications and agents. (Docs) OTel GenAI semconv Proprietary SaaS
• IBM Instana Observability - Product page of IBM Instana, an APM platform with OpenTelemetry ingest and generative AI monitoring. OTel Proprietary Self-hosted SaaS
• Last9 documentation - Documentation of Last9, an observability platform that ingests OpenTelemetry data. OTel Proprietary SaaS
• New Relic AI monitoring - Hosted product documentation for monitoring AI applications. Proprietary SaaS
• Salesforce Agentforce Observability - Product page for observability of Agentforce agents. Proprietary SaaS
• Sentry AI agent monitoring - Hosted product documentation for monitoring agent runs and tool calls. Proprietary SaaS
• SigNoz - OpenTelemetry-native observability platform with guides for monitoring LLM applications. OTel OSS Open-core Self-hosted SaaS
• Splunk AI Agent Monitoring - Documents Splunk Observability Cloud views for AI agents, built on OpenTelemetry GenAI data. OTel GenAI semconv Proprietary SaaS

Trace backends and UIs

General backends and front ends for traces, metrics and logs that accept agent telemetry.

• Claude Code Grafana dashboard (Application Insights) - Community Grafana dashboard that reads coding agent telemetry from Azure Application Insights. OTel
• Claude Code Metrics (Prometheus) Grafana dashboard - Community Grafana dashboard for coding agent usage metrics on Prometheus-compatible backends. OTel
• claude-code-metrics-prometheus - Grafana dashboard that reads OpenTelemetry metrics of a coding agent from Prometheus-compatible stores. OTel OSS
• claude-otel - Configuration for sending a coding agent’s OpenTelemetry metrics to Grafana Cloud, with a dashboard of cost and tokens per session. OTel
• Grafana - Dashboard and query front end used with Tempo, Prometheus and Loki. OSS Self-hosted SaaS
• Grafana Tempo - Distributed tracing backend that accepts OTLP. (Source) OTel OSS Self-hosted SaaS
• Jaeger - Open source distributed tracing backend that accepts OTLP. OTel OSS Self-hosted
• OpenObserve - Self-hostable backend that stores and queries logs, metrics and traces, with native OTLP ingestion. OTel OSS Self-hosted SaaS
• Prometheus - Metrics database commonly used for GenAI metrics. OSS Self-hosted
• Uptrace - Self-hostable APM that collects data with OpenTelemetry and keeps traces, metrics and logs in ClickHouse. OTel OSS Self-hosted

Cloud provider offerings

Observability features of the agent runtimes and AI platforms of cloud providers.

• Amazon Bedrock AgentCore observability - Observability of agents run on Amazon Bedrock AgentCore, with telemetry in an OpenTelemetry-compatible format. Proprietary SaaS
• Amazon Bedrock AgentCore Observability, configure - Describes how to configure observability for AgentCore runtimes and for agents hosted outside AgentCore. OTel Proprietary SaaS
• Amazon Bedrock AgentCore Observability, get started - Explains how to enable AgentCore observability for agents and send their telemetry to CloudWatch. OTel Proprietary SaaS
• Amazon CloudWatch generative AI observability - Documents CloudWatch views for latency, usage and errors, and prompt tracing across models and agents. Proprietary SaaS
• AWS Distro for OpenTelemetry - AWS-supported distribution of OpenTelemetry components that sends traces and metrics to AWS services and other backends. (Source) OTel OSS
• Azure Monitor: enable OpenTelemetry in Application Insights - Explains the Azure Monitor OpenTelemetry distro that sends application telemetry to Application Insights. OTel Proprietary SaaS
• Azure Monitor: monitor AI agents with Application Insights - Describes the Application Insights agents view for performance, token usage and errors of agents. OTel Proprietary SaaS
• Google Cloud Observability for AI agent developers - Overview of instrumenting agents built with frameworks such as ADK and LangGraph and viewing them in Cloud Observability. OTel Proprietary SaaS
• Google Cloud Trace overview - Describes Cloud Trace, the distributed tracing backend that accepts OpenTelemetry data from agents on Google Cloud. OTel Proprietary SaaS
• Google Cloud: trace an agent (Agent Engine runtime) - Explains how to enable Cloud Trace on deployed agents and read traces of tool calls and model requests. OTel Proprietary SaaS
• Microsoft Copilot Studio analytics overview - Documents analytics and monitoring views for agents built in Copilot Studio. Proprietary SaaS
• Microsoft Foundry tracing - Tracing of agents and applications in Microsoft Foundry (formerly Azure AI Foundry), built on OpenTelemetry. Proprietary SaaS

Model serving

Inference and serving engines and the telemetry they expose, such as Prometheus metrics.

• BentoML - Python framework for serving models and AI services, with documented OpenTelemetry tracing and OTLP export. (Docs) OTel OSS
• KServe - Kubernetes inference platform with a Prometheus monitoring guide and an OpenTelemetry-based JSON log profile. (Docs) OSS
• llama.cpp server - HTTP server of llama.cpp with an optional Prometheus-compatible metrics endpoint enabled by a command-line flag. (Source) OSS
• llm-d - Kubernetes inference stack whose operations guide covers Prometheus metrics, Grafana dashboards and OpenTelemetry distributed tracing. (Docs) OSS
• LMDeploy - Toolkit for deploying and serving language models that exposes Prometheus metrics, with a bundled Grafana setup. (Docs) OSS
• NVIDIA Dynamo - Distributed inference serving framework whose Kubernetes guide covers Prometheus metrics and an OTLP collector endpoint for traces. (Docs) OTel OSS
• SGLang - Serving framework for language models, with Prometheus production metrics documented. (Docs) OSS
• Triton Inference Server - Inference server that exposes Prometheus metrics and can emit request traces through OpenTelemetry. (Docs) OTel OSS
• vLLM - Inference and serving engine for language models, with Prometheus production metrics documented. (Docs) OSS

Back to top

Gateways and proxies

Gateways and proxies for model and agent traffic that record or export telemetry.

• Agent Router - Control plane for AI and agent traffic built on Envoy Gateway, formerly named Envoy AI Gateway. (Docs) GenAI semconv OpenInference OSS Self-hosted
• agentgateway - Proxy for agent, MCP and LLM traffic, with tracing and metrics documented. (Docs) OTel OSS
• Azure API Management AI gateway capabilities - Lists API Management features for LLM endpoints, including token metrics, logging and monitoring. Proprietary SaaS
• Bifrost - AI gateway that also works as an MCP gateway, with Prometheus metrics, distributed tracing and logging. OSS
• Cloudflare AI Gateway - Hosted gateway whose documentation covers request logging, analytics, costs and an OpenTelemetry integration. Proprietary SaaS
• Databricks AI Gateway - Documents the Databricks gateway for governing model, MCP and coding agent traffic. Proprietary SaaS
• Envoy Gateway - Gateway project that the Envoy AI Gateway builds on. OSS Self-hosted
• Helicone AI Gateway - Open source gateway for routing LLM requests, with OpenTelemetry support for logs, metrics and traces. OTel OSS Self-hosted SaaS
• IBM ContextForge - Gateway that fronts MCP, A2A and REST tools, with documented tracing and metrics. (Docs) OTel OSS Self-hosted
• Kong AI Gateway - AI traffic features of Kong Gateway, with the AI Proxy plugin documented separately. (Docs) Self-hosted SaaS
• LiteLLM - Python SDK and proxy server with one API for many model providers, including an OpenTelemetry callback for traces. (Source) OTel OSS Open-core Self-hosted SaaS
• OpenRouter broadcast - Hosted router that can forward traces of requests to observability tools, including an OpenTelemetry Collector. OTel Proprietary SaaS
• Portkey AI Gateway - Gateway that routes requests to many providers, with request logging in the hosted product. (Docs) OSS Self-hosted SaaS
• Traceloop Hub - Rust gateway in front of LLM providers that emits OpenTelemetry traces and Prometheus metrics for each request. OTel OSS
• Vercel AI Gateway - Hosted gateway for model access with usage and cost reporting. Proprietary SaaS

Back to top

Evaluation and testing

Evaluation, testing and simulation tools with a documented link to traces or observability.

• Braintrust autoevals - Library of evaluation scorers that works with Braintrust logging. OSS
• DeepEval - Evaluation framework with tracing of LLM applications and an OpenTelemetry integration. (Docs) OSS
• Inspect AI - Evaluation framework for models and agents, with trace logging for debugging evaluation runs. (Docs) OSS
• Opik evaluation - Documentation of the evaluation features that run on traced data in Opik.
• promptfoo - Test and red-team tool for prompts and agents that reads OpenTelemetry (OTLP) traces of the application under test. (Docs) OTel OSS
• Ragas - Python library of evaluation metrics for LLM applications that can score traces collected in tools such as Langfuse and Opik. (Docs) OSS
• TruLens - Library for evaluating and tracking LLM applications with instrumented traces, described as OpenTelemetry-native. (Docs) OSS

Back to top

Cost and usage

Token accounting, budgets and cost attribution for agents.

• Azure API Management llm-emit-token-metric policy - Policy reference for emitting token usage metrics of LLM calls passing through API Management.
• ccusage - Command-line tool that reports token usage and cost of coding agents from local data. OSS
• FinOps for AI overview - FinOps Foundation guide to managing token costs and resource usage of language models and generative AI.
• genai-prices - Library and price data to calculate the price of calls to LLM inference APIs. OSS
• Helicone cost tracking - Guide to monitoring LLM spend and unit economics with Helicone.
• Langfuse token and cost tracking - Documentation of how Langfuse records usage and cost of generations, with model prices that can be overridden.
• LangSmith cost tracking - Documentation of token and cost tracking on LangSmith traces.
• LiteLLM budgets and rate limits - Documentation of budgets and rate limits per user, key and team in the LiteLLM proxy.
• LiteLLM Prometheus metrics - Documents Prometheus metrics for spend, tokens and latency exposed by the LiteLLM proxy.
• LiteLLM spend tracking - Documentation of spend tracking by key, user and team in the LiteLLM proxy.
• OpenClaw Monitor - Dashboard that monitors OpenClaw AI agents, tracking token usage, sessions and 7-day trends across models. OSS Self-hosted
• OpenLIT LLM cost tracking - Documentation of spend per model, user and agent run in OpenLIT, with managed model pricing and backfill of missing costs.
• OpenMeter - Metering and usage-based billing infrastructure that aggregates usage events for AI, API and DevOps products. OSS Self-hosted SaaS
• Phoenix cost tracking - Documentation of cost tracking on traces in Arize Phoenix.
• Tokscale - Command-line tool and dashboard that track token usage and cost across several coding agents. OSS

Back to top

Safety signals

Guardrails and runtime monitoring that emit telemetry, limited to their observability side.

• Agent Governance Toolkit - Policy enforcement and trust scoring for agents that emits spans and metrics for policy evaluations and violations. OTel OSS
• Amazon Bedrock Guardrails CloudWatch metrics - Documentation of CloudWatch metrics for guardrail invocations, interventions, latency and errors.
• Arthur Engine - Monitoring and governance engine that ingests OpenInference traces, runs evaluations on live traces and enforces guardrails. OTel OpenInference OSS Self-hosted
• Google Cloud Model Armor - Google Cloud service that screens prompts and responses of AI applications, with a monitoring dashboard. Proprietary SaaS
• Guardrails AI - Python framework for validating inputs and outputs of language models, with guard invocations traced through an OpenInference instrumentor. OTel OpenInference OSS
• IBM watsonx.governance - Product page for continuous monitoring and governance of AI systems for compliance and operational risk. Proprietary SaaS
• Langfuse security and guardrails - Guide to monitoring and evaluating prompt injection, PII leakage and harmful content with Langfuse and guardrail libraries.
• NeMo Guardrails - Toolkit for adding guardrails to LLM applications, with OpenTelemetry tracing of guardrail execution. (Docs) OTel GenAI semconv OSS
• OpenGuardrails - Specification and reference integrations for portable agent safety verdicts, with a mapping to OpenTelemetry concepts. OSS
• OpenLIT guardrails - Documentation of OpenLIT guardrails for prompt injection, sensitive topics and topic restriction, traced with OpenTelemetry.
• Pydantic Logfire agent governance - Product page on spend policies, data loss prevention and audit of agent actions based on traces. Proprietary SaaS

Back to top

On-chain agents and payments

Tracing, protocols and infrastructure for agents that send transactions or make payments.

Transaction tracing

Tools and documentation for tracing, replaying and debugging on-chain transactions, including those sent by agents and services.

• cast run (Foundry) - Foundry command that re-executes a mined transaction against forked chain state and shows the resulting call trace, with an optional debugger. OSS
• Geth EVM tracing - Documentation of the go-ethereum tracers that replay a transaction on the node and report each step the EVM ran.
• hashspan - OpenTelemetry tracing for on-chain transactions sent by AI agents and other services, with adapters for viem, the CDP SDK, x402 and MPP. OTel OSS
• Tenderly Debugger - Hosted tool for inspecting a failed or past EVM transaction: its call trace, contract state at each step and the matching source code. Proprietary SaaS

Payment protocols

Protocols and SDKs for payments made by agents, such as x402, MPP, AP2 and ACP.

• a2a-x402 - Extension that combines the A2A protocol with x402 payments. OSS
• Agent Payments Protocol (AP2) documentation - Documentation site of AP2, an open protocol for agent-initiated payments.
• Agentic Commerce Protocol - Specification of how a buyer, the buyer’s agent and a merchant exchange checkout requests to complete a purchase. OSS
• AP2 - Agent Payments Protocol specification for payments initiated by agents. OSS
• Mastercard agentic commerce guide - Mastercard developer guide on agentic commerce flows and recommended practices.
• MPP - Machine Payments Protocol for payments between machines and agents. (Source) OSS
• mppx - TypeScript SDK for the Machine Payments Protocol. OSS
• OpenAI commerce developer documentation - Developer documentation for commerce integrations with conversational agents.
• Stripe agentic commerce documentation - Stripe documentation of agentic commerce flows and protocol support.
• Universal Commerce Protocol - Specification for interoperable commerce between agents and merchants. (Source) OSS
• Visa Trusted Agent Protocol - Visa developer documentation for establishing trust between agents and merchants. (Source) Source-available
• x402 - Open protocol for payments over HTTP 402 responses. (Docs) OSS

RPC and node observability

RPC interfaces, proxies and nodes, and the telemetry they expose.

• Erigon dashboard setup - Documentation of Prometheus metrics export and a Grafana dashboard for monitoring an Erigon node. (Source)
• eRPC - Fault-tolerant RPC proxy for EVM chains with caching, a monitoring setup and tracing exported over OTLP. (Docs) OTel OSS Self-hosted
• Ethereum execution APIs - Specification of the JSON-RPC interface of Ethereum execution clients.
• Geth metrics - Documentation of the metrics of the go-ethereum client, including a Prometheus-formatted endpoint. (Source)
• Reth - Ethereum execution client with Prometheus metrics and OTLP trace export through a command-line flag. (Docs) OTel OSS

Wallets and smart accounts

Wallet SDKs, client libraries and smart account infrastructure used by agents.

• Alto - ERC-4337 bundler that relays user operations sent by smart accounts. OSS
• Coinbase AgentKit - Toolkit that gives AI agents wallets and on-chain actions. OSS
• Coinbase CDP SDK - SDK for Coinbase Developer Platform wallets and APIs. OSS
• Privy agent wallets - Documentation of wallets for agents with a policy engine that limits tokens, chains, transfer values and recipients.
• Turnkey agentic wallets - Documentation of scoped signing authority for agents with policies, spending caps and multi-party approval. (Source) OSS
• viem - TypeScript library for Ethereum clients and transactions. OSS

Local and test chains

Local nodes and toolkits for testing transactions without a public network.

• Anvil - Local Ethereum node of Foundry for testing transactions without a public network. OSS
• ethereum-package (Kurtosis) - Kurtosis package that starts a private multi-client Ethereum testnet on Docker or Kubernetes for testing and development. OSS
• Foundry - Toolkit for Ethereum development that includes Anvil and Cast. OSS
• Hardhat simulated networks - Documentation of the in-process blockchains that Hardhat starts on the local machine when tests or scripts run. (Source) OSS

Back to top

Findings

Compatibility notes and bugs that affect agent telemetry, each with public evidence such as an upstream issue.

• Anvil gave concurrent transactions from one account the same nonce - Concurrent eth_sendTransaction calls from one unlocked Anvil account got the same nonce, which breaks parallel agent tests. (Notes) Fixed in Foundry nightly builds after 1.8.5
• EDR OP chain type reports an L1 data fee of 0 - The OP chain type of EDR set the L1Block fee scalars at the wrong storage offsets, so simulated receipts reported an L1 data fee of 0. (Notes) Fixed in @nomicfoundation/edr 0.24.0
• eRPC OTLP/HTTP tracing exports nothing when the endpoint has a scheme - With tracing over OTLP/HTTP, eRPC exports no spans when the endpoint includes a scheme, the form its documentation asks for. Open
• MLflow OTLP trace ingest drops the span kind - Spans sent to the OTLP endpoint of MLflow lose their kind, so CLIENT and SERVER spans read back as INTERNAL. Open
• viem concurrent receipt waits shared the first call options - Concurrent viem receipt waits for one hash on one client all ran with the timeout and confirmations of the first call. (Notes) Fixed in viem 2.57.0
• viem receipt wait missed replacements of already dropped transactions - The viem receipt wait never detects a replacement when the original transaction left the pool before its first lookup, and times out. Open
• viem receipt wait took a transaction for its own replacement - The viem receipt wait treated the awaited transaction as its own replacement when the node returned it before its receipt, and failed. (Notes) Fixed in viem 2.57.3

Back to top

Benchmarks and datasets

Agent trace datasets and observability benchmarks.

• AgentErrorBench (AgentDebug) - Annotated failure trajectories from ALFWorld, GAIA and WebShop with an error taxonomy and a debugging pipeline. OSS
• AgentRx dataset (Hugging Face) - Annotated failed agent trajectories used to evaluate step localization and failure attribution. (Source)
• AIOpsLab - Framework to design, run and evaluate autonomous AIOps agents on microservice faults with telemetry as input. OSS
• CatchBench - Benchmark for auditing agent failures from the declared configuration, a growing prefix of a run and the finished trace. OSS
• MAST-Data (Hugging Face) - Annotated traces of multi-agent systems used for failure taxonomy research.
• OpenRCA - Benchmark of root cause localization for software failures from logs, metrics and traces. OSS
• SWE-smith trajectories (Hugging Face) - Dataset of recorded trajectories of a coding agent solving software engineering tasks. OSS
• TRAIL dataset (Hugging Face) - Annotated agent execution traces with error labels, stored as OpenTelemetry spans. OSS
• Who_and_When (Hugging Face) - Dataset of multi-agent logs with annotated failure agents and steps.

Back to top

Learning

Articles, books, talks, papers, courses and case studies on observing agents.

Articles and guides

Blog posts, official guides and demo applications.

• Agenta guide to LLM observability with OpenTelemetry - Vendor guide on instrumenting LLM applications with OpenTelemetry.
• Agno tracing - Documentation of OpenTelemetry tracing of Agno agents and teams.
• AI Agent Observability - Evolving Standards and Best Practices - OpenTelemetry blog post on observing agents and the standards in progress.
• Amazon Bedrock AgentCore samples - Sample agents and notebooks for Amazon Bedrock AgentCore, including observability examples. OSS
• AutoGen core OpenTelemetry - Documentation of the OpenTelemetry support of the AutoGen core runtime.
• AutoGen tracing and observability - Documentation of OpenTelemetry tracing for AutoGen AgentChat applications.
• AWS blog: build trustworthy agents with AgentCore Observability - AWS Machine Learning blog post on using AgentCore Observability to monitor and debug agents.
• CrewAI observability - Official documentation listing tracing integrations for CrewAI.
• Dify: integrate with Alibaba Cloud Monitor - Documents the Dify setting that exports application traces to Alibaba Cloud Monitor through OpenTelemetry.
• Dynatrace expands AI coding agent monitoring - Dynatrace announcement of monitoring for several coding agents.
• Google ADK observability - Official documentation on observability for ADK agents.
• Google Cloud blog: Google Cloud and OpenTelemetry GenAI conventions - Google Cloud blog post on its work with the OpenTelemetry GenAI conventions.
• Govern and redact traces in Unity Catalog - Databricks documentation on governing access to MLflow traces and redacting them in Unity Catalog.
• Greptime: how OpenTelemetry traces LLM calls, agent reasoning and MCP tools - Blog post summarizing the GenAI semantic conventions for LLM, agent and MCP spans.
• Hamel Husain: a field guide to rapidly improving AI products - Practitioner guide on error analysis of logged traces for AI products.
• How to redact PII from LLM telemetry - Guide to redacting PII from LLM traces with pipelines and an OpenTelemetry Collector while keeping traces debuggable.
• How to redact sensitive prompts in GenAI OpenTelemetry traces - Guide to removing PII and credentials from GenAI traces before they reach the observability backend.
• IONOS: monitor a coding agent with the Monitoring Service - IONOS tutorial for exporting coding agent token and cost metrics to its monitoring service.
• LangGraph observability - Documentation of the tracing options for LangGraph applications, mostly through LangSmith.
• LangSmith: trace voice agents - Vendor blog on tracing voice agent frameworks in LangSmith.
• LangWatch OpenClaw integration docs - Documents sending traces of a personal assistant agent to LangWatch over OpenTelemetry.
• LangWatch: instrumenting an OpenClaw assistant with OpenTelemetry - Vendor guide to exporting OpenTelemetry GenAI traces from a self-hosted personal assistant agent.
• LlamaIndex observability - Documentation of the tracing options of LlamaIndex, including OpenTelemetry-based integrations.
• Mastra observability - Official documentation on tracing Mastra agents and workflows.
• Maxim: agent observability guide - Vendor guide on monitoring and evaluating production agents.
• Microsoft Agent Framework observability - Documentation of how Microsoft Agent Framework emits traces, logs and metrics with the OpenTelemetry GenAI conventions.
• Microsoft Tech Community: Observability for the Age of Generative AI - Azure observability blog post on agent dashboards, AI-aware traces and attribute-based trace search.
• Mintive: OpenTelemetry standards for agent monitoring - Vendor blog on OpenTelemetry standards for agent monitoring.
• MLflow tracing integrations - Official list of libraries that MLflow can trace automatically.
• MLflow: tracing OpenClaw - MLflow blog on tracing a personal assistant agent with a self-hosted MLflow server.
• Monitor agent usage with OpenTelemetry (VS Code) - Documentation of the OpenTelemetry traces, metrics and events of agent mode in VS Code, following the GenAI conventions.
• Monitoring AI commerce agents (LogicMonitor) - Vendor guide on monitoring shopping agents, with attention to infrastructure and agent decision paths.
• Nango: OpenTelemetry for AI agent observability - Blog post on tracing agent tool calls and API integrations with OpenTelemetry.
• Observability and audit trails for Agent Builder in Elastic - Article on reading Agent Builder traces in Elastic and writing an approval record for agent actions.
• OpenAI Agents SDK tracing - Official documentation on the trace and span model of the SDK.
• OpenTelemetry blog: generative AI - OpenTelemetry blog post introducing the generative AI work.
• OpenTelemetry blog: LLM observability - OpenTelemetry blog post on observing LLM applications.
• OpenTelemetry context propagation - Explains how trace context moves between services and processes, which applies to multi-agent and tool calls.
• OpenTelemetry demo - Multi-service demo application that shows traces, metrics and logs end to end. OTel OSS
• OpenTelemetry GenAI conventions are not stable yet (DEV Community) - Community post on the stability status of the GenAI conventions in 2026.
• OpenTelemetry traces concept - Official introduction to spans and traces.
• Opik: observability and alerts for OpenClaw - Vendor post on tracing LLM calls, tool runs and memory recall of a personal assistant agent with an Opik plugin.
• Opik: Pipecat integration - Documents conversation and turn level tracing of Pipecat voice agents in Opik.
• Pydantic AI with Logfire - Official documentation on debugging and monitoring Pydantic AI agents.
• Semantic Kernel observability - Official documentation on logs, metrics and traces for Semantic Kernel applications.
• smolagents: inspecting runs with OpenTelemetry - Tutorial on instrumenting smolagents runs with OpenTelemetry and viewing them.
• Splunk blog: what is agentic observability - Vendor explainer on agentic observability and AI-assisted operations.
• Traceloop OpenLLMetry blog introduction - Original announcement of OpenLLMetry, extending OpenTelemetry to LLM applications.
• Uptrace: AI agent observability explained - Vendor explainer of agent observability concepts and standards.
• Uptrace: OpenTelemetry for AI systems - Vendor implementation guide for OpenTelemetry instrumentation of LLM and agent systems.
• Vellum: AI observability for agents - Vendor practical guide to observing agents in production.
• Vercel AI SDK telemetry - Official documentation of the telemetry settings of the AI SDK.
• Weights and Biases: AI agent observability guide - Vendor guide to tracing AI agents.

Books

Books on agent or LLM observability, and on the observability and OpenTelemetry practice they build on.

• AI Engineering (book) - Book by Chip Huyen on building applications with foundation models, with chapters on evaluation and on monitoring and observability.
• Building Applications with AI Agents (book) - Book by Michael Albada on designing agent systems, with chapters on validation, measurement and monitoring in production.
• Learning OpenTelemetry (book) - Book by Ted Young and Austin Parker on adopting OpenTelemetry, the telemetry layer most agent tracing builds on.
• Observability Engineering (book) - Book by Charity Majors, Liz Fong-Jones and George Miranda on observability practice for production systems.
• Observability for Large Language Models (report) - Short report by Phillip Carter on observability for LLM applications with OpenTelemetry.

Talks and videos

Conference sessions, videos and podcast episodes with a public page.

• KubeCon EU 2025: deep dive to AI agent observability - Session on tracing and monitoring AI agents with OpenTelemetry, by IBM and Langtrace speakers. (Docs)
• KubeCon India 2026: what did my agent do? - Session on observability and accountability for AI agents with OpenTelemetry and GenAI conventions.
• KubeCon talk: AI/ML observability with OpenTelemetry and Fluent Bit - KubeCon session recording by Celalettin Calis on AI and ML observability with OpenTelemetry and Fluent Bit.
• KubeCon talk: End-to-end observability for LLM inference: from token to GPU - KubeCon session recording by speakers from DaoCloud on observability of LLM inference.
• KubeCon talk: Exploring observability with MCP servers - KubeCon session recording by Tiffany Jernigan of Grafana Labs on observability with MCP servers.
• KubeCon talk: GenAI observability: keeping GenAI honest without oversharing - KubeCon session recording by Liudmila Molkova of Grafana Labs on GenAI observability and sensitive content.
• KubeCon talk: LLM observability with Open Data Hub and vLLM - KubeCon lightning talk recording by Twinkll Sisodia on LLM observability with Open Data Hub and vLLM.
• KubeCon talk: OpenTelemetry for autonomous AI agents - KubeCon session recording by Abdel Fane on applying OpenTelemetry to autonomous AI agents.
• KubeCon talk: Optimizing LLM performance in Kubernetes with OpenTelemetry - KubeCon session recording by Ashok Chandrasekar and Liudmila Molkova on OpenTelemetry for LLM serving on Kubernetes.
• KubeCon talk: Sponsored keynote: OpenSearch, open observability for agentic AI - KubeCon sponsored keynote recording by Anirudha Jadhav on open observability for agentic AI with OpenSearch.
• KubeCon talk: Why are your AI’s decisions hard to explain: trace every decision - KubeCon session recording by Dhiraj Kumar Jain and Vikash Agrawal on tracing the decisions of AI systems.
• OpenClaw observability and monitoring with OpenTelemetry and SigNoz - Video on collecting traces, metrics and logs from OpenClaw with OpenTelemetry and viewing them in SigNoz.
• OpenLLMetry is all you need (AI Engineer talk) - Conference talk on instrumenting LLM applications with OpenLLMetry.
• Watching AI workers: OpenSearch observability for OpenClaw and Hermes-agent - Conference talk by Stefano Pampaloni on observing OpenClaw and Hermes-agent runs with OpenSearch.

Papers

Research papers on agent tracing, evaluation and monitoring.

• A multi-dataset benchmark for LLM agents in microservice failure diagnosis - Benchmark of agents diagnosing failures in microservice systems.
• A survey on AgentOps - Survey of categories, challenges and directions of AgentOps.
• AEGIS: a pre-execution firewall and audit layer for AI agents - Proposes a layer that checks tool calls before execution and keeps an audit log.
• Agent Flight Recorder: tamper-evident audit trails for tool-using agents - Proposes tamper-evident audit trails for agents, with on-chain anchoring.
• AgentDebugX: toolkit for failure observability, attribution and recovery - Presents an open-source toolkit for failure observability and attribution in LLM agents.
• AgentGuard: execution guardrails from anomalous coding-agent trajectories - Studies learning guardrails from anomalous trajectories of coding agents.
• Agentic observability: automated alert triage for Adobe e-commerce - Industry paper on agent-based alert triage for an e-commerce platform.
• AgentLens: production-assessed trajectory reviews for coding agent evaluation - Studies reviewing production trajectories of coding agents for evaluation.
• AgentOps: enabling observability of LLM agents - Proposes a taxonomy of artifacts and data to trace across the lifecycle of LLM agents.
• AgentTrace: a structured logging framework for agent observability - Proposes a structured logging framework for agent systems that can integrate with OpenTelemetry.
• AIOpsLab: a framework to evaluate AI agents for autonomous clouds - Framework for evaluating agents that operate cloud systems.
• Annotrace: a workbench for step-level annotation of agent traces - Presents a workbench for annotating individual steps in agent traces.
• Beyond the mandate: a security analysis of the Agent Payments Protocol - Systematic security analysis of AP2.
• Detecting safety violations across many agent traces - Studies detection of safety violations by analyzing many agent traces.
• Failure as a process: an anatomy of CLI coding agent trajectories - Analyzes failure patterns in trajectories of command line coding agents.
• Five attacks on the x402 agentic payment protocol - Describes attacks on the x402 payment protocol.
• From agent traces to trust: evidence tracing and execution provenance - Survey of evidence tracing and execution provenance for LLM agents.
• Functionality-grounded benchmark for web agents in e-commerce - Benchmark of web agents on e-commerce functionality.
• Governance-aware agent telemetry for closed-loop enforcement - Proposes agent telemetry that supports governance enforcement in multi-agent systems.
• Observability and fault injection for LLM multi-agent systems - Studies observability and fault injection for multi-agent systems used in software engineering.
• Observability for delegated execution in agentic AI systems - Discusses observability needs when agents delegate execution to other agents or tools.
• Observing, analyzing and optimizing agentic AI systems - Describes automation for observing, analyzing and optimizing agentic AI systems.
• PrefixGuard: online failure warning and trace-grounded diagnosis for LLM agents - Studies online failure warning from partial agent traces.
• ProbGuard: proactive runtime monitoring for LLM agent safety - Proposes probabilistic runtime monitoring to predict unsafe agent behavior.
• ShopGym: simulation and benchmarking of e-commerce web agents - Framework for simulating and benchmarking e-commerce agents.
• SoK: security of autonomous LLM agents in agentic commerce - Systematization of security issues of agents in commerce settings.
• TelemetrySuffBench: is agent telemetry sufficient for failure-origin diagnosis? - Benchmark that tests whether agent telemetry suffices to find where failures originate.
• Traccia: an OpenTelemetry-based governance platform for AI systems - Describes a governance platform built on OpenTelemetry data.
• TRAIL: trace reasoning and agentic issue localization - Introduces an annotated benchmark of agent traces for locating errors.
• WebMall: a multi-shop benchmark for web agents - Benchmark of web agents performing tasks across several online shops.
• Which agent causes task failures and when? - Studies automated failure attribution in multi-agent systems and releases a dataset.
• Whispers of wealth: red-teaming the Agent Payments Protocol via prompt injection - Security analysis of AP2 flows under prompt injection.
• Why do multi-agent LLM systems fail? - Analyzes failure modes of multi-agent systems and releases annotated traces.
• Willful disobedience: automatically detecting failures in agentic traces - Studies automatic detection of failures in agent traces.

Courses

Courses and workshops.

• Evaluating AI Agents (DeepLearning.AI) - Short course on adding tracing to an agent and evaluating its components, taught with Arize.
• Getting Started with OpenTelemetry (LFS148) - Free online Linux Foundation course covering manual and automatic OpenTelemetry instrumentation and Collector pipelines.
• Hugging Face Agents Course - Free course on building agents; has a bonus unit on agent observability and evaluation. (Source)

Case studies

Public write-ups of how organizations observe agents in production, by industry.

• Anthropic engineering: how we built our multi-agent research system - Engineering write-up of a multi-agent research system that describes production tracing for debugging.
• Elastic: AI agent observability with OTel, OpenLIT and Elastic - Elastic blog on monitoring web-browsing agents with OpenTelemetry, OpenLIT and Elastic.
• Elastic: GenAI observability for customer support - Engineering account of monitoring a customer support assistant built on Elastic.
• Honeycomb for financial services - Vendor page on observability requirements in financial services, including retention for audit.
• Pydantic case study: Evergreen.ai - Case study of a financial planning agent team using tracing and evaluations in production.

Newsletters and podcasts

Regular publications on the topic.

• Code RED #31: beyond hype, the real impact of AI on observability - Live episode on agentic observability and OpenTelemetry support for AI.
• Dash0 Code RED podcast - Podcast about observability and OpenTelemetry, with episodes on AI and observability.
• Datadog OpenTelemetry news, May 2026 - Monthly roundup of OpenTelemetry developments, including GenAI conventions releases.
• o11y.news - Weekly newsletter linking to observability and OpenTelemetry reading.
• O11ycast - Podcast on observability with episodes on AI systems and OpenTelemetry.
• OpenObservability Talks podcast - Podcast on open source observability with an episode on OpenTelemetry for GenAI.
• OpenObservability Talks: OpenTelemetry for GenAI and the OpenLLMetry project - Podcast episode (S6E06) with the creator of OpenLLMetry on GenAI conventions and tracing.

Back to top

Community

Working groups, meetings and channels where agent observability is discussed.

• A2A protocol discussions - GitHub Discussions forum of the Agent2Agent protocol repository.
• CNCF Slack channel otel-genai-instrumentation-wg - Slack channel of the OpenTelemetry GenAI instrumentation working group in the CNCF workspace.
• CNCF: Observability Day at KubeCon North America 2026 - CNCF blog post announcing Observability Day on 2026-11-09, with AI and agent observability on the program.
• Model Context Protocol discussions - GitHub Discussions forum of the Model Context Protocol specification repository.
• OpenTelemetry community - Page of the OpenTelemetry project on how to join its working groups, meetings and chat channels.
• OpenTelemetry GenAI conventions issues - Issue tracker of the repository that holds the OpenTelemetry GenAI semantic conventions.
• OpenTelemetry GenAI project page (community repo) - Describes the OpenTelemetry GenAI project, its scope, owners and meeting information.
• OpenTelemetry GenAI SIG meeting notes - Shared meeting notes of the OpenTelemetry GenAI semantic conventions and instrumentation SIG.
• OpenTelemetry YouTube channel - Official OpenTelemetry channel with recordings of project meetings and talks.

Back to top

Related lists

Other curated lists that cover parts of this topic.

• awesome-agent-observability (anhermon) - Link list on agent observability, grouped into standards, platforms, trace backends, evaluation, guardrails, gateways, MCP and frameworks.
• awesome-agent-observability (danielt69) - Link list on agent observability with hands-on guides, platform comparisons, evaluation, guardrails, red teaming and benchmarks.
• awesome-ai-sre - List of AI agents, production debugging, incident management, AIOps and chaos engineering tools for site reliability work.
• awesome-harness-engineering - Curated list on AI agent harness engineering with tools, patterns and evaluations.
• awesome-llm-observability (ContextJet-ai) - Curated list of resources for observability of LLM applications.
• Awesome-LLMOps - Curated list of LLMOps tools.
• awesome-observability - Curated list of observability resources, covering tools, practices and standards.
• awesome-opentelemetry - List of OpenTelemetry learning material, books, videos, tracing backends and instrumentation by language.

Back to top

Archived and historical

Entries kept for reference: archived upstream, superseded, or historical.

- [OpenTelemetry contrib Vertex AI instrumentation for Python](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-vertexai) - Deprecated OpenTelemetry instrumentation package for the Vertex AI client library in Python. `OSS` Kept because: Deprecated in the GenAI package migration of opentelemetry-python-contrib, with no replacement package planned (README, 2026-10-20).

Back to top

Contributing

Contributions are welcome. Read CONTRIBUTING.md first: one entry per pull request, a neutral one-sentence description, and public evidence for every OTel claim. Submissions may be prepared with AI tools; a person submits them and answers the review.

License

To the extent possible under law, the contributors have waived all copyright and related rights to this work.

This site is open source. Improve this page.