LLM observability and debugging

All topics · Alphabetical index

Concepts

• Execution traces; model spans; tool spans; handoff traces
• Prompt versions; context versions; retrieved-document IDs
• Token usage; cost attribution; budget tracking
• Time to first token; end-to-end latency; tool latency
• Tool failures; loop counts; repeated actions; timeout rates
• Error taxonomies; trace replay; production-to-eval datasets
• Sensitive-data redaction; access-controlled telemetry
• Prompt failures; context failures; retrieval failures; model failures; application failures

Further reading

Reference links for this topic.

• OpenAI agent engineering resources