Skip to content

Commit 171d78e

Browse files
authored
feat(tailsamplingprocessor): Add tracing for ConsumeTraces and samplingTick (#45646)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> # Description This PR adds distributed tracing instrumentation to the tailsamplingprocessor to provide visibility into trace processing behavior and policy evaluation. ## What's Changed - Trace spans for key operations: - `tailsampling.ConsumeTraces`: Tracks incoming traces/spans count - `tailsampling.samplingPolicyOnTick`: Records batch processing metrics and policy evaluation results - Optimized for minimal allocations: - Pre-allocated policy name attributes to reduce per-decision allocations - Added String() method to samplingpolicy.Decision enum ## Why This Matters Tracing provides critical insights into the async nature of tail sampling decisions, making it significantly easier to debug issues like: - Which policies are making sampling decisions - Policy evaluation latency and errors - Trace batching and decision timing <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> ## Link to tracking issue Fixes #43931 <!--Describe what testing was performed and which tests were added.--> # Testing I have run my own collector with 17 policies, and a testing app sending data to try to match some of them. [See my config here](https://gist.github.com/jdvr/740b8a4d83d5fdcc9219601e8a2ad31f) ## Resulting traces ### `tailsampling.ConsumeTraces` shows full data journey, from receiver. <img width="1678" height="454" alt="image" src="https://github.com/user-attachments/assets/01f54409-17fb-422a-8fb4-5a29e8582871" /> ### `tailsampling.samplingPolicyOnTick` shows executed policies and sampling decision information <img width="1676" height="545" alt="image" src="https://github.com/user-attachments/assets/1783b0a6-14a8-4718-93a6-fdc6fb9ff112" /> <img width="1680" height="367" alt="image" src="https://github.com/user-attachments/assets/608bec00-69a5-47e5-bda6-dddb05969bd7" /> # Benchmarks I have run benchmark test and the new code doesn't produce any significant change <details> <summary>bench_before.txt</summary> ```shell goos: darwin goarch: arm64 pkg: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor cpu: Apple M1 BenchmarkSampling-8 55839 19188 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 64318 18691 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 63302 18951 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 63372 18894 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 64154 18808 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 64200 18687 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 63379 18854 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 63357 18933 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 64212 18686 ns/op 0 B/op 0 allocs/op BenchmarkSampling-8 63412 18927 ns/op 0 B/op 0 allocs/op BenchmarkProcessorThroughput-8 866 1210287 ns/op 231.94 MB/s 4688599 B/op 36616 allocs/op BenchmarkProcessorThroughput-8 897 1211507 ns/op 231.71 MB/s 4687953 B/op 36606 allocs/op BenchmarkProcessorThroughput-8 916 1217058 ns/op 230.65 MB/s 4687569 B/op 36602 allocs/op BenchmarkProcessorThroughput-8 927 1219294 ns/op 230.23 MB/s 4689993 B/op 36638 allocs/op BenchmarkProcessorThroughput-8 926 1212671 ns/op 231.49 MB/s 4687731 B/op 36603 allocs/op BenchmarkProcessorThroughput-8 853 1215764 ns/op 230.90 MB/s 4687635 B/op 36602 allocs/op BenchmarkProcessorThroughput-8 874 1267715 ns/op 221.43 MB/s 4689856 B/op 36638 allocs/op BenchmarkProcessorThroughput-8 885 1228161 ns/op 228.57 MB/s 4688816 B/op 36624 allocs/op BenchmarkProcessorThroughput-8 854 1214659 ns/op 231.11 MB/s 4689043 B/op 36625 allocs/op BenchmarkProcessorThroughput-8 894 1226793 ns/op 228.82 MB/s 4687590 B/op 36605 allocs/op PASS ok github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor 46.393s ``` </details> <details> <summary>bench_after.txt</summary> ```shell ``` </details> Comparation using [benchstat](golang.org/x/perf/cmd/benchstat) ``` ❯ benchstat bench_before.txt bench_after.txt goos: darwin goarch: arm64 pkg: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor cpu: Apple M1 │ bench_before.txt │ bench_after.txt │ │ sec/op │ sec/op vs base │ Sampling-8 18.87µ ± 1% 18.84µ ± 3% ~ (p=0.838 n=10) ProcessorThroughput-8 1.216m ± 1% 1.314m ± 1% +8.02% (p=0.000 n=10) geomean 151.5µ 157.3µ +3.83% │ bench_before.txt │ bench_after.txt │ │ B/op │ B/op vs base │ Sampling-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ ProcessorThroughput-8 4.471Mi ± 0% 4.471Mi ± 0% ~ (p=0.315 n=10) geomean ² +0.00% ² ¹ all samples are equal ² summaries must be >0 to compute geomean │ bench_before.txt │ bench_after.txt │ │ allocs/op │ allocs/op vs base │ Sampling-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ ProcessorThroughput-8 36.61k ± 0% 36.61k ± 0% ~ (p=0.305 n=10) geomean ² +0.00% ² ¹ all samples are equal ² summaries must be >0 to compute geomean │ bench_before.txt │ bench_after.txt │ │ B/s │ B/s vs base │ ProcessorThroughput-8 220.1Mi ± 1% 203.7Mi ± 1% -7.42% (p=0.000 n=10) ``` # Benchmark Results ### BenchmarkProcessorThroughput | Metric | Before | After | Change | |--------|--------|-------|--------| | **Latency** | 1.216ms | 1.314ms | +8.02% | | **Memory** | 4.471 MiB | 4.471 MiB | ~0% | | **Allocations** | 36.61k | 36.61k | ~0% | | **Throughput** | 220.1 MB/s | 203.7 MB/s | -7.42% | ### BenchmarkSampling | Metric | Before | After | Change | |--------|--------|-------|--------| | **Latency** | 18.87µs | 18.84µs | -0.16% | | **Memory** | 0 KB | 0 KB | ~0% | | **Allocations** | 0 | 0 | ~0% | <!--Describe the documentation added.--> #### Documentation <!--Please delete paragraphs that you did not use before submitting.-->
1 parent 3ad470c commit 171d78e

6 files changed

Lines changed: 358 additions & 10 deletions

File tree

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
7+
component: processor/tail_sampling
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: "Add distributed tracing instrumentation to `tailsamplingprocessor` to provide visibility into trace processing behavior and policy evaluation"
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [43931]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext: |
19+
Adds trace spans for key operations:
20+
- `tailsampling.ConsumeTraces`: Tracks incoming traces/spans count
21+
- `tailsampling.samplingPolicyOnTick`: Records batch processing metrics and policy evaluation results
22+
23+
# If your change doesn't affect end users or the exported elements of any package,
24+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
25+
# Optional: The change log or logs in which this entry should be included.
26+
# e.g. '[user]' or '[user, api]'
27+
# Include 'user' if the change is relevant to end users.
28+
# Include 'api' if there is a change to a library API.
29+
# Default: '[user]'
30+
change_logs: [user]
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
// Copyright The OpenTelemetry Authors
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
package telemetry // import "github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/internal/telemetry"
5+
6+
import (
7+
"go.opentelemetry.io/otel/attribute"
8+
9+
"github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/pkg/samplingpolicy"
10+
)
11+
12+
// DecisionAttributes holds pre-allocated attribute.KeyValue for all decision types.
13+
// It provides O(1) access to pre-computed attributes using array indexing instead of map lookups,
14+
// eliminating runtime allocations during trace processing. The struct is immutable after construction
15+
// and safe for concurrent access.
16+
type DecisionAttributes struct {
17+
key attribute.Key // The attribute key used for all decisions
18+
attrs [8]attribute.KeyValue // Array indexed by Decision value (0-7)
19+
unknownAttr attribute.KeyValue // Returned for invalid decisions
20+
}
21+
22+
// NewDecisionAttributes creates a DecisionAttributes instance with pre-computed KeyValue pairs
23+
// for all 8 decision types defined in samplingpolicy.Decision.
24+
// Parameters:
25+
// - attrKey: The attribute key to use for all decision attributes (e.g., "decision.final", "policy.decision")
26+
//
27+
// Example:
28+
//
29+
// spanDecisionAttrs := NewDecisionAttributes(attribute.Key("decision.final"))
30+
// attr := spanDecisionAttrs.Get(samplingpolicy.Sampled) // Returns pre-allocated KeyValue
31+
func NewDecisionAttributes(attrKey attribute.Key) *DecisionAttributes {
32+
da := &DecisionAttributes{
33+
key: attrKey,
34+
}
35+
36+
// Pre-compute all 8 decision attributes
37+
da.attrs[samplingpolicy.Unspecified] = attrKey.String(samplingpolicy.Unspecified.String())
38+
da.attrs[samplingpolicy.Pending] = attrKey.String(samplingpolicy.Pending.String())
39+
da.attrs[samplingpolicy.Sampled] = attrKey.String(samplingpolicy.Sampled.String())
40+
da.attrs[samplingpolicy.NotSampled] = attrKey.String(samplingpolicy.NotSampled.String())
41+
da.attrs[samplingpolicy.Dropped] = attrKey.String(samplingpolicy.Dropped.String())
42+
da.attrs[samplingpolicy.Error] = attrKey.String(samplingpolicy.Error.String())
43+
//nolint:staticcheck // InvertSampled is deprecated but we need to support existing decision values
44+
da.attrs[samplingpolicy.InvertSampled] = attrKey.String(samplingpolicy.InvertSampled.String())
45+
//nolint:staticcheck // InvertNotSampled is deprecated but we need to support existing decision values
46+
da.attrs[samplingpolicy.InvertNotSampled] = attrKey.String(samplingpolicy.InvertNotSampled.String())
47+
48+
// Pre-compute the unknown attribute for out-of-bounds decisions
49+
da.unknownAttr = attrKey.String("missing.preallocation")
50+
51+
return da
52+
}
53+
54+
// Get returns the pre-allocated KeyValue for the given decision.
55+
func (d *DecisionAttributes) Get(decision samplingpolicy.Decision) attribute.KeyValue {
56+
// Bounds check: valid decisions are 0-7 (inclusive)
57+
if decision < 0 || decision >= samplingpolicy.Decision(len(d.attrs)) {
58+
return d.unknownAttr
59+
}
60+
return d.attrs[decision]
61+
}
Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
// Copyright The OpenTelemetry Authors
2+
// SPDX-License-Identifier: Apache-2.0
3+
4+
package telemetry
5+
6+
import (
7+
"sync"
8+
"testing"
9+
10+
"github.com/stretchr/testify/assert"
11+
"github.com/stretchr/testify/require"
12+
"go.opentelemetry.io/otel/attribute"
13+
14+
"github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor/pkg/samplingpolicy"
15+
)
16+
17+
func TestNewDecisionAttributes_AllDecisions(t *testing.T) {
18+
attrKey := attribute.Key("decision.final")
19+
da := NewDecisionAttributes(attrKey)
20+
21+
require.NotNil(t, da)
22+
23+
// Test all 8 valid decisions
24+
testCases := []struct {
25+
decision samplingpolicy.Decision
26+
expectedValue string
27+
}{
28+
{samplingpolicy.Unspecified, "unspecified"},
29+
{samplingpolicy.Pending, "pending"},
30+
{samplingpolicy.Sampled, "sampled"},
31+
{samplingpolicy.NotSampled, "not_sampled"},
32+
{samplingpolicy.Dropped, "dropped"},
33+
{samplingpolicy.Error, "error"},
34+
//nolint:staticcheck // Testing deprecated but still supported values
35+
{samplingpolicy.InvertSampled, "invert_sampled"},
36+
//nolint:staticcheck // Testing deprecated but still supported values
37+
{samplingpolicy.InvertNotSampled, "invert_not_sampled"},
38+
}
39+
40+
for _, tc := range testCases {
41+
t.Run(tc.expectedValue, func(t *testing.T) {
42+
attr := da.Get(tc.decision)
43+
assert.Equal(t, attrKey, attr.Key, "Key should match")
44+
assert.Equal(t, tc.expectedValue, attr.Value.AsString(), "Value should match decision string")
45+
})
46+
}
47+
}
48+
49+
func TestDecisionAttributes_Get_UnknownDecisions(t *testing.T) {
50+
attrKey := attribute.Key("test.key")
51+
da := NewDecisionAttributes(attrKey)
52+
53+
testCases := []struct {
54+
name string
55+
decision samplingpolicy.Decision
56+
}{
57+
{"negative decision", samplingpolicy.Decision(-1)},
58+
{"decision 8", samplingpolicy.Decision(8)},
59+
{"decision 100", samplingpolicy.Decision(100)},
60+
{"decision 255", samplingpolicy.Decision(255)},
61+
}
62+
63+
for _, tc := range testCases {
64+
t.Run(tc.name, func(t *testing.T) {
65+
attr := da.Get(tc.decision)
66+
assert.Equal(t, attrKey, attr.Key, "Key should match")
67+
assert.Equal(t, "missing.preallocation", attr.Value.AsString(), "Should return unknown value")
68+
})
69+
}
70+
71+
// Verify unknownAttr is consistent across multiple calls
72+
attr1 := da.Get(samplingpolicy.Decision(-1))
73+
attr2 := da.Get(samplingpolicy.Decision(100))
74+
assert.Equal(t, attr1.Value.AsString(), attr2.Value.AsString(), "Unknown attributes should be consistent")
75+
}
76+
77+
func TestDecisionAttributes_StringConsistency(t *testing.T) {
78+
attrKey := attribute.Key("policy.decision")
79+
da := NewDecisionAttributes(attrKey)
80+
81+
// Verify each decision's attribute value matches its String() method
82+
allDecisions := []samplingpolicy.Decision{
83+
samplingpolicy.Unspecified,
84+
samplingpolicy.Pending,
85+
samplingpolicy.Sampled,
86+
samplingpolicy.NotSampled,
87+
samplingpolicy.Dropped,
88+
samplingpolicy.Error,
89+
samplingpolicy.InvertSampled, //nolint:staticcheck // Testing deprecated but still supported values
90+
samplingpolicy.InvertNotSampled, //nolint:staticcheck // Testing deprecated but still supported values
91+
}
92+
93+
for _, decision := range allDecisions {
94+
t.Run(decision.String(), func(t *testing.T) {
95+
attr := da.Get(decision)
96+
expectedString := decision.String()
97+
actualString := attr.Value.AsString()
98+
assert.Equal(t, expectedString, actualString,
99+
"Decision %d: attribute value should match String() method", decision)
100+
})
101+
}
102+
}
103+
104+
func TestDecisionAttributes_DifferentKeys(t *testing.T) {
105+
key1 := attribute.Key("decision.final")
106+
key2 := attribute.Key("policy.decision")
107+
108+
da1 := NewDecisionAttributes(key1)
109+
da2 := NewDecisionAttributes(key2)
110+
111+
// Test with the same decision type but different keys
112+
decision := samplingpolicy.Sampled
113+
114+
attr1 := da1.Get(decision)
115+
attr2 := da2.Get(decision)
116+
117+
// Keys should be different
118+
assert.NotEqual(t, attr1.Key, attr2.Key, "Keys should differ")
119+
assert.Equal(t, key1, attr1.Key, "First attribute should use first key")
120+
assert.Equal(t, key2, attr2.Key, "Second attribute should use second key")
121+
122+
// Values should be the same (both "sampled")
123+
assert.Equal(t, attr1.Value.AsString(), attr2.Value.AsString(), "Values should be the same")
124+
assert.Equal(t, "sampled", attr1.Value.AsString(), "Value should be 'sampled'")
125+
}
126+
127+
func TestDecisionAttributes_Concurrent(t *testing.T) {
128+
attrKey := attribute.Key("concurrent.test")
129+
da := NewDecisionAttributes(attrKey)
130+
131+
// Number of goroutines and iterations
132+
numGoroutines := 100
133+
iterations := 1000
134+
135+
var wg sync.WaitGroup
136+
wg.Add(numGoroutines)
137+
138+
// All valid decisions
139+
validDecisions := []samplingpolicy.Decision{
140+
samplingpolicy.Unspecified,
141+
samplingpolicy.Pending,
142+
samplingpolicy.Sampled,
143+
samplingpolicy.NotSampled,
144+
samplingpolicy.Dropped,
145+
samplingpolicy.Error,
146+
samplingpolicy.InvertSampled, //nolint:staticcheck // Testing deprecated but still supported values
147+
samplingpolicy.InvertNotSampled, //nolint:staticcheck // Testing deprecated but still supported values
148+
}
149+
150+
// Spawn multiple goroutines that concurrently call Get()
151+
for i := range numGoroutines {
152+
go func(_ int) {
153+
defer wg.Done()
154+
for j := range iterations {
155+
// Cycle through all valid decisions
156+
decision := validDecisions[j%len(validDecisions)]
157+
attr := da.Get(decision)
158+
159+
// Verify the returned attribute is valid
160+
assert.Equal(t, attrKey, attr.Key)
161+
assert.Equal(t, decision.String(), attr.Value.AsString())
162+
}
163+
}(i)
164+
}
165+
166+
wg.Wait()
167+
}
168+
169+
func TestDecisionAttributes_ZeroValueKey(t *testing.T) {
170+
// Test with empty key (unusual but valid)
171+
emptyKey := attribute.Key("")
172+
da := NewDecisionAttributes(emptyKey)
173+
174+
attr := da.Get(samplingpolicy.Sampled)
175+
assert.Equal(t, emptyKey, attr.Key, "Should work with empty key")
176+
assert.Equal(t, "sampled", attr.Value.AsString(), "Value should still be correct")
177+
}
178+
179+
func TestDecisionAttributes_ImmutabilityAfterConstruction(t *testing.T) {
180+
attrKey := attribute.Key("immutable.test")
181+
da := NewDecisionAttributes(attrKey)
182+
183+
// Get the same decision multiple times
184+
attr1 := da.Get(samplingpolicy.Sampled)
185+
attr2 := da.Get(samplingpolicy.Sampled)
186+
attr3 := da.Get(samplingpolicy.Sampled)
187+
188+
// All should return the same value
189+
assert.Equal(t, attr1.Value.AsString(), attr2.Value.AsString())
190+
assert.Equal(t, attr2.Value.AsString(), attr3.Value.AsString())
191+
assert.Equal(t, "sampled", attr1.Value.AsString())
192+
}

processor/tailsamplingprocessor/pkg/samplingpolicy/samplingpolicy.go

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,30 @@ const (
5151
InvertNotSampled
5252
)
5353

54+
// String returns a string representation of the Decision.
55+
func (d Decision) String() string {
56+
switch d {
57+
case Unspecified:
58+
return "unspecified"
59+
case Pending:
60+
return "pending"
61+
case Sampled:
62+
return "sampled"
63+
case NotSampled:
64+
return "not_sampled"
65+
case Dropped:
66+
return "dropped"
67+
case Error:
68+
return "error"
69+
case InvertSampled:
70+
return "invert_sampled"
71+
case InvertNotSampled:
72+
return "invert_not_sampled"
73+
default:
74+
return "unknown"
75+
}
76+
}
77+
5478
// Evaluator implements a tail-based sampling policy evaluator,
5579
// which makes a sampling decision for a given trace when requested.
5680
type Evaluator interface {

0 commit comments

Comments
 (0)