Proposal: Last Seen Connector for detecting stale OTLP clients

[aswath-anand]

Problem

We're using the following architecture:

OTel Agents (VMs)
↓
OTLP
↓
OpenTelemetry Collector Gateway
↓
Prometheus Remote Write
↓
Prometheus
↓
Grafana

Each VM continuously exports metrics through the gateway.

One of our operational requirements is to detect when an agent stops sending telemetry and keep that condition visible until the agent starts sending data again.

Today we're alerting using metrics such as: time() - max by (host_name)timestamp(otelcol_process_uptime_seconds_total)
or other metrics emitted by the agent.

The issue is that when an agent stops sending telemetry, Prometheus eventually marks the time series as stale. Once the series disappears, Grafana resolves the alert because there is no longer a series to evaluate.

This makes it difficult to:

Keep incidents open until the agent actually recovers
Troubleshoot offline agents
Build dashboards showing the last time an agent was seen

Question

Is there a recommended OpenTelemetry Collector pattern for solving this problem?
Has this problem already been discussed or solved elsewhere?

I'm trying to understand whether there is already a recommended solution before building something custom.

Thanks!

[agrovr]

I would model this at the monitoring/inventory layer rather than as a Collector connector.

With push telemetry, Prometheus can only alert on label sets that still exist. Once the final sample becomes stale, neither Grafana nor an absent_over_time() query can dynamically recreate the identity of every missing host. You need a second, continuously present series that says which agents should exist, for example from your CMDB/cloud inventory/service discovery:

expected_otel_agent_info{host_id="vm-123", host_name="api-01"} 1

Then alert on expected agents that have produced no heartbeat in the lookback window:

max by (host_id, host_name) (expected_otel_agent_info)
unless on (host_id)
max by (host_id) (
  present_over_time(otelcol_process_uptime_seconds_total[10m])
)

Use a stable host.id-derived label as the join key; names are often mutable or reused. If Prometheus can scrape each agent directly, its target inventory already provides this identity and up == 0 is the simpler solution.

keep_firing_for can bridge a bounded gap, but it cannot keep the incident alive indefinitely. Likewise, absent_over_time() is useful when the target label is known in the rule, but it cannot enumerate arbitrary hosts after their series have vanished. Prometheus documents the range functions and unless set operator here:

• https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time
• https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators

A “last seen” connector would have to emit on a timer even when no telemetry arrives, persist identities across restarts, define expiry/removal semantics, and deduplicate state in an HA gateway deployment. That is effectively an inventory/state service, which is why keeping expected membership outside the telemetry stream is usually more reliable.

[dlanov]

The existing reply covers the Prometheus-layer approach well, but there are several native Collector-side options worth knowing about before building anything custom.

Option 1 transform processor to emit a heartbeat gauge

The simplest Collector-side solution is to use the transform processor to add a last_seen_timestamp metric on every batch that passes through. This keeps the series alive as long as the agent is sending:

  processors:
    transform/heartbeat:
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["last_seen_epoch_s"], Double(UnixNano(Now())) / 1e9)

This doesn't solve the stale series problem on its own because once the agent goes silent, Prometheus still stalens the series. But combined with a generous --storage.tsdb.retention.time and absent_over_time() on a known label set it handles bounded outage windows.

Option 2 count connector as a presence signal

The count connector (available in collector-contrib) can emit a running count metric keyed on any resource attribute — including host.name or host.id. As long as the agent sends at least one datapoint in the scrape window, the count series stays alive:

  connectors:
    count:
      metrics:
        agent.heartbeat.count:
          description: "Datapoints received per agent"
          conditions: []
          attributes:
            - key: host.name

  service:
    pipelines:
      metrics/in:
        receivers: [otlp]
        processors: [batch]
        exporters: [count]
      metrics/out:
        receivers: [count]
        exporters: [prometheusremotewrite]

This gives you a agent.heartbeat.count series per host that flatlines (and eventually stalens) when the agent goes silent — same stale problem, but a cleaner signal to alert on.

Option 3 The correct long-term solution (inventory-join in Prometheus)

The existing reply is right about this being the most reliable approach. The key insight is that the Collector is the wrong place to track presence because it only knows about agents that are currently talking to it. An inventory source (cloud provider metadata, Ansible inventory, Terraform state, a simple ConfigMap) is the authoritative list of
expected agents.

The recommended pattern:

  # Alert: expected agent has not been seen in 10 minutes
  max by (host_id, host_name) (expected_agents_info)
  unless on (host_id)
  max by (host_id) (
    present_over_time(otelcol_process_uptime_seconds_total[10m])
  )

Expose expected_agents_info via a Prometheus file-based service discovery file or a custom exporter that reads your inventory. This fires continuously even after the series stalens because the expected_agents_info series is always present.

Why a "Last Seen Connector" is complex to build correctly

If you do go the custom connector route, the main challenges are:

• State persistence across restarts: the connector needs durable storage to remember agents it last saw before a crash
• HA deduplication: in a gateway deployment with multiple collector replicas, each replica sees a partition of agents; a single "last seen" view requires shared state
• Expiry semantics: when do you stop emitting a "last seen" metric for an agent that has been gone for weeks?

These are the reasons the inventory-join approach wins operationally. The connector would essentially be reimplementing a lightweight inventory service inside the Collector.