Skip to main content

Annotate Traces

Annotations in Agenta let you enrich the traces created by your LLM applications. You can add scores, comments, expected answers and other metrics to help evaluate your application's performance.

Google Colaboratory

Open in Google Colaboratory

What You Can Do With Annotations​

  • Collect user feedback on LLM responses
  • Run custom evaluation workflows
  • Measure application performance in real-time

Understanding How Annotations Work​

Annotations link to specific points (spans) in your application traces. You can link them to:

  • Parent spans (like an agent invocation)
  • Child spans (like a single LLM call within a larger process)

Each annotation can include:

  • Numbers (scores, ratings)
  • Categories (labels, classifications)
  • Text (comments, reasoning)

Annotation Structure​

Here's what an annotation looks like:

{
    "data": {
        "outputs": {
            "score": 80,
            "normalized_score": 0.8,
            "reasoning": "The answer is mainly correct"
        }
    },
    "references": {
        "evaluator": {
            "slug": "my_evaluator"
        }
    },
    "links": {
        "invocation": {
            "trace_id": "47cb4a13b42a47bab2198509543ff14d",
            "span_id": "1a9130f6ad398e55"
        }
    },
    "metadata": {
        "someattr": "somevalue"
    }
}

An annotation has four main parts:

  1. Data: The actual evaluation content (scores, comments)
  2. References: Which evaluator to use (will be created automatically if it doesn't exist)
  3. Links: Which trace and span you're annotating
  4. Metadata (optional): Any extra information you want to include

What You Can Include in Annotations​

Annotations are flexible - you can include whatever makes sense for your evaluation:

// Just a simple score
{"score": 3}

// A score with explanation
{"score": 3, "comment": "The response is not grounded"}

// Multiple metrics with reference information
{"score": 3, "normalized_score": 0.5, "comment": "The response is not grounded", "expected_answer": "The capital of France is Paris"}

You can use numbers, text, booleans, or categorical values.

Connecting Annotations to Traces​

Each annotation links to a specific span in your trace using a trace_id and span_id. These IDs follow the OpenTelemetry format.

warning

Agenta doesn't check if the trace and span IDs exist. Make sure they're valid before sending.

Working with Evaluators​

Every annotation must reference an Evaluator, which:

  • Defines what your annotation should contain
  • Provides context for interpreting the annotation

If you reference an evaluator that doesn't exist yet, Agenta will automatically create one based on your first annotation.

You can add multiple annotations from the same evaluator to a single span, which is useful for collecting feedback from different sources.

Using Metadata​

The metadata field lets you add extra information to your annotation. Use it to store things like user IDs, session information, or other context.

Creating Annotations​

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

# Annotation data
annotation_data = {
    "annotation": {
        "data": {
            "outputs": {
                "score": 40,
                "reasoning": "Response contains some inaccuracies",
                "labels": ["Partially Correct", "Needs Improvement"]
            }
        },
        "references": {
            "evaluator": {
                "slug": "content_quality"
            }
        },
        "links": {
            "invocation": {
                "trace_id": "47cb4a13b42a47bab2198509543ff14d",
                "span_id": "1a9130f6ad398e55"
            }
        }
    }
}

# Make the API request
response = requests.post(
    f"{base_url}/api/annotations/",
    headers=headers,
    json=annotation_data
)

# Process the response
if response.status_code == 200:
    print("Annotation created successfully")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)
info

If the evaluator doesn't exist yet, Agenta will create it automatically based on your first annotation.

Understanding the Response​

When you create an annotation, you'll receive a response that includes the annotation data and its ID:

{
  "annotation": {
    "created_at": "2025-05-13T16:54:52.245664Z",
    "created_by_id": "019315dc-a332-7ba5-a426-d079c43ab776",
    "span_id": "a1713ea8f59291f6",
    "trace_id": "bb39070937f74f169b60dfc420729ad3",
    "kind": "custom",
    "source": "api",
    "data": {
      "outputs": {
        "score": 0,
        "reasoning": "The answer is totally false.",
        "expected_answer": "The capital of France is Paris",
        "normalized_score": 0
      }
    },
    "metadata": {
      "user_id": "miamia"
    },
    "references": {
      "evaluator": {
        "id": "0196ca64-9cbf-7ee3-b507-2b559dd13090",
        "slug": "main-scorer"
      }
    },
    "links": {
      "invocation": {
        "span_id": "09383e1124b264d0",
        "trace_id": "5d52a529b48135fd58790f8aa3e33fb8"
      }
    }
  }
}
info

Annotations themselves are stored as traces with their own trace_id and span_id, which is different from the invocation they're linked to.

Viewing Annotations in the UI​

You can see all annotations for a trace in the Agenta UI. Open any trace and check the Annotations tab to see detailed information. The right sidebar shows average metrics for each evaluator.

Querying Annotations​

You can query annotations in several ways:

1. Query by annotation ID:

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

# Query by annotation ID
query_data = {
    "annotation": {
        "trace_id": "bb39070937f74f169b60dfc420729ad3",
        "span_id": "a1713ea8f59291f6"
    }
}

response = requests.post(
    f"{base_url}/api/annotations/query",
    headers=headers,
    json=query_data
)

if response.status_code == 200:
    print("Query successful")
    print(response.json())
else:
    print(f"Error: {response.status_code}")
    print(response.text)

2. Query all annotations for an invocation:

# Query all annotations for an invocation
query_data = {
    "annotation": {
        "links": {
            "invocation": {
                "trace_id": "5d52a529b48135fd58790f8aa3e33fb8",
                "span_id": "09383e1124b264d0"
            }
        }
    }
}

response = requests.post(
    f"{base_url}/api/annotations/query",
    headers=headers,
    json=query_data
)

Removing Annotations​

import requests

base_url = "https://cloud.agenta.ai"
headers = {
    "Content-Type": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY"
}

# Define the trace_id and span_id of the annotation to delete
trace_id = "bb39070937f74f169b60dfc420729ad3"
span_id = "a1713ea8f59291f6"

# Make the delete request
response = requests.delete(
    f"{base_url}/api/annotations/{trace_id}/{span_id}",
    headers=headers
)

# Process the response
if response.status_code == 200:
    print("Annotation deleted successfully")
else:
    print(f"Error: {response.status_code}")
    print(response.text)