feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK by yahya-mouman · Pull Request #12318 · DataDog/dd-trace-java

yahya-mouman · 2026-08-27T14:01:13Z

Summary

Adds annotateAgentManifest(LLMObs.AgentManifest) to the Java LLMObs SDK, allowing users to manually declare an agent span's configuration (name, instructions, model, model_settings, tools). Mirrors the Python implementation (DataDog/dd-trace-py#19771) and follows the annotatePrompt pattern from #12161.

Changes

LLMObsTags.java — adds AGENT_MANIFEST = "agent_manifest" constant
LLMObs.java — adds two public immutable builder classes:
- LLMObs.AgentTool — represents a single tool (name, optional description, optional parameters)
- LLMObs.AgentManifest — builder with name, instructions, model, model_settings, tools
LLMObsSpan.java — adds default void annotateAgentManifest(LLMObs.AgentManifest) (no-op default for backwards compat)
NoOpLLMObsSpan.java — explicit @Override no-op
DDLLMObsSpan.java — real implementation: validates span kind (agent only), builds manifest map, stores as _ml_obs_tag.agent_manifest
LLMObsSpanMapper.java — adds agent_manifest to TAGS_FOR_REMAPPING; serializes to meta.agent_manifest as a msgpack map

Behaviour

LLMObs.AgentManifest manifest = LLMObs.AgentManifest.builder()
    .name("travel_desk")
    .instructions("Book travel for the user.")
    .model("gpt-4o")
    .modelSettings(Map.of("temperature", 0.7))
    .tools(List.of(LLMObs.AgentTool.from("get_weather", "Look up weather", null)))
    .build();

agentSpan.annotateAgentManifest(manifest);

Only applies to agent span kind; other span kinds emit a log warning and no-op
Calling twice on the same span overwrites the previous manifest (no merge)
framework is set to "AgentObs SDK" automatically by the SDK
name defaults to the span name if not provided
model_settings keys are forwarded as-is (no allowlist in this initial PR)
Null manifest is silently ignored

Test plan

./gradlew :dd-trace-api:test --tests "datadog.trace.api.llmobs.LLMObsTest" — 27 tests pass (4 new)
./gradlew :dd-java-agent:agent-llmobs:test --tests "datadog.trace.llmobs.domain.DDLLMObsSpanTest" — 39 tests pass (7 new)
./gradlew :dd-trace-core:test --tests "datadog.trace.llmobs.writer.ddintake.LLMObsSpanMapperTest" — 18 tests pass (2 new)

🤖 Generated with Claude Code

- Add AgentManifest immutable value class with Builder pattern in LLMObs.java - Add AgentTool immutable value class in LLMObs.java - Add AGENT_MANIFEST constant to LLMObsTags - Add annotateAgentManifest() default method to LLMObsSpan interface - Add comprehensive builder tests for AgentManifest and AgentTool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Stores agent manifest fields (name, instructions, model, model_settings, tools) as an internal tag `_ml_obs_tag.agent_manifest`. Only applies to agent spans; warns and no-ops on other span kinds. Null manifest is silently ignored. Tools with null/empty names are skipped with a warning. A second call overwrites the previous manifest. Framework field "AgentObs SDK" is always added when any manifest fields are present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add AGENT_MANIFEST_KEY byte constant, include the tag in TAGS_FOR_REMAPPING, and handle it in the meta serialization loop as a msgpack map. Add two tests: one verifying all manifest fields appear in meta, one verifying the tag does not leak into the tags list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ncybul · 2026-08-27T16:51:14Z

@codex

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5f56f11d6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

ncybul

A couple minor suggestions but overall looks good to me! Would be nice to see a manual verification of setting the manifest via the SDK.

ncybul · 2026-08-27T17:51:51Z

+   *
+   * <p>Build via {@link AgentManifest#builder()} and pass to {@link
+   * LLMObsSpan#annotateAgentManifest(AgentManifest)}. Only applied on agent spans; ignored on other
+   * span kinds. A subsequent call on the same span overwrites the previous manifest.


This is different from the Python implementation, right? I think in Python, we were merging the two manifests together whenever possible. It's probably best that we align the implementations and choose one approach.

Personally, I would lean towards merging the fields whenever possible.

Yes we can do that. It'll help the async distributed trace cases too,

Switched to merge semantics in 71284a3. The second call now overlays non-null/non-empty fields on top of the existing manifest, name/instructions/model use new-value-wins with fallback, model_settings shallow-merges (new keys win on conflicts), and tools replace only when the new call provides a non-empty list.

i think the docstring still needs to be updated here from overwrites → merges

dd-octo-sts · 2026-08-28T11:08:53Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

datadog-prod-us1-6

More details

The default API method keeps compatibility. The implementation accepts manifests only for agent spans and writes them to meta.agent_manifest.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit b5f56f1 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

…test hardening - Change annotateAgentManifest to merge successive calls instead of overwriting: name/instructions/model use new-value-wins logic with fallbacks; model_settings shallow-merges; tools replace only when caller provides a non-empty list. - Change MANUAL_FRAMEWORK constant from "AgentObs SDK" to "manual". - Update all test assertions that expected "AgentObs SDK" to "manual". - Replace overwrite test with merge-semantics test; add model_settings merge test. - Tests for post-finish no-op and empty-tools-list were already present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sabrenner

one main comment a couple nits otherwise lgtm!

sabrenner · 2026-08-28T17:27:29Z

+      }
+    }
+    // instructions
+    if (manifest.getInstructions() != null && !manifest.getInstructions().isEmpty()) {


nit - can we follow the same pattern above of getting

String manifestInstructions = manifet.getInstructions();

so that we only do access once? same thing with the other gets below

sabrenner · 2026-08-28T17:29:55Z

  }
+
+  /** A tool declared in an agent manifest. */
+  public static final class AgentTool {


can we re-use the existing ToolDefinition class? I know it has schema vs description, etc., but wondering if we would be able to do that to consolidate types so the API wasn't as packed

It would mean we would need to add code in the processor to accept this. As I understand the processor expects that tool format in the manifest. Is it needed enough to require us to propagate the change ?

sabrenner · 2026-08-28T17:34:43Z

+      base.put("model", manifest.getModel());
+    }
+    // model_settings: shallow merge
+    if (manifest.getModelSettings() != null && !manifest.getModelSettings().isEmpty()) {


should we have an allowlist for model settings here? could we potentially be adding sensitive information?

I thought about adding an allowlist however I don't want to block users from adding any non supported configs. like agent related settings example : retry_number or framework specific agentic settings

…domain/DDLLMObsSpan.java Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

…nce; update docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

yahya-mouman and others added 4 commits August 27, 2026 13:36

feat(llmobs): add annotateAgentManifest default method to LLMObsSpan

ff0f77c

yahya-mouman added the type: feature Enhancements and improvements label Aug 27, 2026

yahya-mouman requested review from ncybul and sabrenner August 27, 2026 14:15

This comment has been minimized.

Sign in to view

chatgpt-codex-connector Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/domain/DDLLMObsSpan.java Outdated

ncybul approved these changes Aug 27, 2026

View reviewed changes

yahya-mouman marked this pull request as ready for review August 28, 2026 11:08

yahya-mouman requested a review from a team as a code owner August 28, 2026 11:08

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Aug 28, 2026

datadog-prod-us1-6 Bot reviewed Aug 28, 2026

View reviewed changes

yahya-mouman mentioned this pull request Aug 28, 2026

feat(llmobs): accept an agent manifest manually DataDog/dd-trace-py#19771

Open

sabrenner reviewed Aug 28, 2026

View reviewed changes

yahya-mouman and others added 2 commits August 28, 2026 19:43

Update dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/…

a0e864c

…domain/DDLLMObsSpan.java Co-authored-by: Sam Brenner <106700075+sabrenner@users.noreply.github.com>

Extract manifest getter calls to local variables; fix spanName refere…

df4a559

…nce; update docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sabrenner approved these changes Aug 28, 2026

View reviewed changes

Uh oh!

Conversation

yahya-mouman commented Aug 27, 2026

Summary

Changes

Behaviour

Test plan

Uh oh!

This comment has been minimized.

ncybul commented Aug 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yahya-mouman Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

sabrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yahya-mouman Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yahya-mouman Aug 28, 2026 •

edited

Loading

yahya-mouman Aug 28, 2026 •

edited

Loading