k8seventsreceiver milliseconds precision #46995
Replies: 1 comment
|
Looking at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8seventsreceiver/k8s_event_to_logdata.go, it never seems to set ObservedTimestamp, only sets Timestamp from the Kubernetes event fields. So ObservedTimestamp leaves the receiver as zero. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/opensearchexporter/encoder.go then fills it unconditionally with time.Now() at export time, no matter what. So what ends up in observedTimestamp is when the exporter serialized the record, not when the collector actually saw the event come off the Watch stream. That would explain the re-list gap since the Kubernetes timestamp stays as-is but the export happens an hour later, and that's what gets stamped. Might be worth raising a bug if i'm reading this all correct. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi, we are trying to use k8seventsreceiver with the opensearchexporter (ss4o mode) and noticed that Kubernetes events only provide second-precision timestamps.
In metricbeat, @timestamp was set to time.Now() at the moment the Watch callback processed each event. This gave every event a unique millisecond-precision timestamp that preserved the exact Watch stream delivery order — useful for incident investigation when multiple events happen within the same second.
We were hoping observedTimestamp could serve the same purpose in OTel, since it has nanosecond precision. However, we observed cases where observedTimestamp diverges significantly from @timestamp. For example:
@timestamp: 2026-03-18T10:10:00Z
observedTimestamp: 2026-03-18T11:12:17.356340573Z
This is a ~1 hour gap, suggesting the event was re-delivered during a re-list and got a new observedTimestamp on re-ingestion. It is expected?
All reactions