Production advice needed: single OpenTelemetry Collector for traces + logs, or keep Filebeat for logs?

[am36166]

Component(s)

No response

Describe the issue you're reporting

I’d like some production advice on my current OpenTelemetry Collector setup.
Right now, I’m evaluating whether it is a good idea to use a single OpenTelemetry Collector deployment for both:
traces → forwarded to Jaeger
logs → forwarded to Elasticsearch
Or whether it is better in production to keep:
OpenTelemetry Collector for traces
Filebeat for logs
My environment already has Elasticsearch and Filebeat in place, and Filebeat is currently handling logs.
This is the collector manifest I am testing:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: simplest
  namespace: observability
spec:
  image: otel/opentelemetry-collector-contrib@sha256:b14234c4bc1b7364629af272e564913bb57bdc9736d45b8b6db5ab3417dc75f9
  mode: deployment
  replicas: 2
  env:
    - name: ES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: elasticsearch-user
          key: password

  config:
    extensions:
      basicauth/server:
        htpasswd:
          file: /etc/otel/htpasswd

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            auth:
              authenticator: basicauth/server
          http:
            endpoint: 0.0.0.0:4318
            auth:
              authenticator: basicauth/server

    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15

      attributes/remove-db-info:
        actions:
          - key: db.connection_string
            action: delete
          - key: db.user
            action: delete

      batch:
        send_batch_size: 1024
        timeout: 5s

    exporters:
      otlp/jaeger:
        endpoint: simple-prod-collector.observability:4317
        tls:
          insecure: true

      elasticsearch/logs:
        endpoints: ["https://elasticsearch-master-es-http.elastic-system:9200"]
        mapping:
          mode: "otel"
        user: "elastic"
        password: "${ES_PASSWORD}"
        tls:
          insecure: false
          ca_file: /etc/elasticsearch/certs/ca.crt
        retry:
          enabled: true
          initial_interval: 1
        sending_queue:
          enabled: true
          num_consumers: 10 
          queue_size: 5000

    service:
      extensions: [basicauth/server]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, attributes/remove-db-info, batch]
          exporters: [otlp/jaeger]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [elasticsearch/logs]

  volumeMounts:
    - name: basicauth
      mountPath: /etc/otel/htpasswd
      subPath: htpasswd
    - name: elasticsearch-certs
      mountPath: /etc/elasticsearch/certs
      readOnly: true

  volumes:
    - name: basicauth
      secret:
        secretName: otel-basicauth
    - name: elasticsearch-certs
      secret:
        secretName: elasticsearch-master-es-http-certs-public
        defaultMode: 0444
        items:
          - key: ca.crt
            path: ca.crt

In real production environments, is it common to keep traces and logs in the same collector deployment,
or do most teams separate them?
For a setup that already uses Elasticsearch + Filebeat, would you keep Filebeat for logs and use OTel only for traces?
Is using one collector for both signals a reasonable production pattern, or does it usually create operational problems under load?
Are there any obvious issues in this manifest from a production perspective?
Thank you in advance

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[abdellahfd270-max]

1- Since you already have OTLP receivers configured, use them to receive logs directly from your applications. This provides immediate trace-log correlation without extra work.
2- If you have legacy systems writing to local files that Filebeat is already scraping reliably, there is no urgent need to replace it. Filebeat's log-rotation and back-pressure mechanisms are exceptionally mature.

[s3onghyun]

Both work technically — a single Collector can run a traces pipeline (→ Jaeger) and a logs pipeline (→ Elasticsearch) side by side. So the question is really an operational one, and here's how I'd decide it.

Keep Filebeat for logs if: it's already working and you have no concrete pain with it. "Migrate working log infra" is a cost with little upside unless you're consolidating on OTLP on purpose. Smallest change wins.

Move logs onto the Collector if: you specifically want one agent / one config surface / OTLP end-to-end, or you want resource attributes and processors shared between signals. That's a real benefit — but it's a consolidation decision, not a "this is better" one.

The one thing I'd push back on regardless of which way you go: don't run both signals in a single Collector process if log volume is significant. Logs are usually far higher volume than traces, and in one process they share the same memory_limiter and the same blast radius — a log spike or a flaky ES exporter can back-pressure and drop your traces. If you do consolidate, prefer separate Collector deployments per signal (or at least be ready to split them), so traces stay healthy when logs misbehave.

Also, a config nit in your manifest:

memory_limiter:
  limit_percentage: 75
  spike_limit_percentage: 1   # <- too low

spike_limit_percentage is the headroom the limiter reserves above the soft limit; 1% is effectively no headroom, so the limiter will trip hard and abruptly under bursts. The usual guidance is ~20% of the limit — i.e. limit_percentage: 75 with spike_limit_percentage: 15-ish. Also make sure the container's memory request/limit is set in line with these percentages, otherwise the percentages are measured against a number you didn't expect.

Net: if Filebeat works, keeping it is the lower-risk choice; consolidate onto the Collector only if you actually want OTLP-everywhere — and if you do, split traces and logs into separate Collector deployments rather than one process.