What is a good tail sampling configuration for long running celery tasks

[skippaydevs]

Hi,

I have a few celery tasks that run for a very long time (which is not the best design but it is what it is). The time they can run for is 1 hour. After that, they get killed and the SoftTimeLimitException is raised. I would like to sample traces for these tasks in 100 % of cases. For tasks that finish successfully, I want to trace only 10 % of these.

This is my configuration:

  tail_sampling/celery-very-long:
    decision_wait: 3606s
    num_traces: 40
    expected_new_traces_per_sec: 1
    decision_cache:
      sampled_cache_size: 8
      non_sampled_cache_size: 80
    sample_on_first_match: true
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: baseline_sampling
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

My understanding is this:

• decision_wait - I want to wait for the whole trace, which is 1 hour and a bit longer to make sure the tasks failed with soft time limit get to be sampled correctly
• num_traces - There are about 20 celery tasks per hour being processed, so just to be sure, this is twice the necessary size
• expected_new_traces_per_sec - that would be about 0.005, not sure I can set float, so 1
• decision_cache - 8 for sampled, 80 for non_sampled, probably a bit more than necessary

Does this configuration make sense? What is the recommended approach for sampling long running celery tasks?

[s3onghyun]

It can work, but two of the assumptions are risky at 1-hour decision_wait, and one is likely to silently drop traces. Worth walking through before you deploy it.

1. decision_wait: 3606s buffers every trace in memory for an hour, not just the celery ones. tail_sampling in the default trace-complete mode accumulates a trace's spans until the timer fires (decision_wait), then decides. That timer applies to all traces flowing through this processor, so with a 1-hour wait, every trace — short HTTP requests included — sits in memory for up to an hour before a decision is made. The docs call this out for trace-complete: "later decisions and higher in-memory/storage pressure." If this collector only sees the celery tasks, that's ~20 traces/hour and fine; if it also sees your normal request traffic, you're holding an hour of all traffic in RAM.

2. num_traces: 40 is "traces kept in memory", and it's shared by everything in that hour window — not just your 20 celery tasks. So it's not "2× the celery count"; it's a cap on all concurrently-buffered traces. Any trace beyond 40 in-flight evicts an older one from memory before its decision is made, and an evicted long-running trace is exactly the one you wanted at 100%. Given num_traces defaults to 50000, 40 is very low unless this pipeline is truly celery-only. I'd size it from peak concurrent traces across everything this processor sees, with headroom.

3. expected_new_traces_per_sec is just a sizing hint for pre-allocating structures — it doesn't rate-limit anything, so rounding 0.005 up to 1 is harmless. No worries there.

Recommended shape:

• Isolate the long tasks onto their own pipeline so the 1-hour wait only applies to them. Route celery traces (by service.name / a resource attribute) via the routing connector into a dedicated tail_sampling with the long decision_wait, and send everything else through a normal short-wait sampler. This keeps the hour-long buffering off your high-volume traffic — it's the single biggest win here.
• On the celery sampler, keep decision_wait a bit above the max task duration (your 3606s is right), but size num_traces to the real concurrent celery count with margin (hundreds, not 40 — memory per trace is small).
• Your error/latency-first + 10% probabilistic policy set is the right idea. Since SoftTimeLimitException surfaces as an error span, the status_code: ERROR policy will catch the killed tasks at 100% as you intend.

One caveat if memory is the real worry: sampling_strategy: span-ingest decides per incoming batch instead of buffering the whole trace, which cuts memory a lot — but it rejects stateful policies and finalizes non-terminal traces as "not sampled" on cleanup, so it changes semantics. For "keep all errors, 10% otherwise" over hour-long traces, trace-complete on an isolated pipeline is the safer fit; reach for span-ingest only if the in-memory pressure from (1) becomes the binding constraint.