[exporter/prometheus] Removing job and instance_id labels

[kaushal02]

The metrics exported by prometheus have the extra labels job and instance_id. I tried the following processors which are unable to get rid of them:

  resource/remove_labels:
    attributes:
      - key: job
        action: delete

  transform/metrics:
    metric_statements: 
      - context: resource
        statements:
          - delete_key(attributes, "job")
          - delete_key(attributes, "instance_id")
    
  attributes/metrics:
    actions:
      - key: job
        action: delete

I am using otelcol-contrib version 0.142.0.

Any idea why I am unable to remove these labels?

[s3onghyun]

job and instance are not attributes, which is why deleting them does nothing — there is no such key to delete. They are synthesised at export time from resource attributes:

• service.name (+ service.namespace when present) → job
• service.instance.id → instance

You can see it in the translator (pkg/translator/prometheusremotewrite/helper.go):

// Map service.name + service.namespace to job
// Map service.instance.id to instance

So delete_key(attributes, "job") looks for a key that was never there, and the exporter then derives job from service.name anyway. To change or drop those labels you have to act on the source attributes instead:

processors:
  transform/drop_service_identity:
    error_mode: ignore
    metric_statements:
      - context: resource
        statements:
          - delete_key(attributes, "service.name")
          - delete_key(attributes, "service.namespace")
          - delete_key(attributes, "service.instance.id")

Two things to be aware of before you do that:

1. This is the Prometheus data model's identity pair. job/instance are what Prometheus uses to identify a target, and target_info is joined on them (... * on (job, instance) group_left(...) target_info). Removing them makes series from different sources collide and breaks that join. Usually the real goal is to change what lands in them — set service.name/service.instance.id to what you want — rather than to remove them.
2. If your actual problem is the duplicate service_name / service_instance_id labels appearing alongside job/instance because you enabled resource_to_telemetry_conversion, there is a dedicated switch for exactly that:

exporters:
  prometheusremotewrite:
    resource_to_telemetry_conversion:
      enabled: true
      exclude_service_attributes: true   # skip service.* — they are already job/instance

Docs describe it as: "the service.name, service.instance.id and service.namespace resource attributes, which are already converted to job and instance labels respectively, will be excluded from the final metrics."

If you can say which of the two you are after — removing the identity labels entirely, or removing the duplicated service_* copies — I can be more specific.