Skip to content

redis: report Redis command and connection metrics - #1301

Merged
psbrar99 merged 2 commits into
envoyproxy:mainfrom
nacx:pr/j-redis-pool-metrics
Nov 9, 2026
Merged

psbrar99 merged 2 commits into
envoyproxy:mainfrom
nacx:pr/j-redis-pool-metrics

Conversation

@nacx

@nacx nacx commented Nov 4, 2026

Copy link
Copy Markdown
Member

When Redis stops answering, the only Redis metric, cx_active, keeps reporting healthy-looking connections while every command on them is stuck, and nothing reports commands failing, timing out or waiting.

Per Redis client (redis_pool / redis_per_second_pool), add:

  • cmd_inflight (gauge), cmd_total, cmd_success, cmd_error, cmd_io_timeout, cmd_deadline_exceeded, cmd_rejected and a cmd_time timer for the commands (pipelines) issued by rate limit calls;
  • cx_connect_error, and cx_local_close_error for connections closed after an error, which are also logged at warning level, rate-limited to once every 10s.

Adds utils.LogLimiter for the rate-limited log.

@nacx
nacx force-pushed the pr/j-redis-pool-metrics branch from 1cb0295 to c1da9b3 Compare November 9, 2026 15:03
@psbrar99

psbrar99 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

should we switch cmd_time to NewMilliTimer and add a Prometheus mapper entry. NewTimer records microseconds, but the sinks expect milliseconds, so a 22 ms command shows up as 22235|ms in statsd. With NewMilliTimer("cmd_time") it's 21|ms (tested). For Prometheus, add a mapping such as ratelimit_redis_pool_cmd_time_seconds with timer_type: histogram and scale: 0.001, placed after the response_time entry. PROMETHEUS_RESPONSE_TIME_AS_MILLISECONDS removes only the first scale: 0.001 in the file, so it would otherwise hit this one?

@nacx

nacx commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Fixed

nacx and others added 2 commits November 9, 2026 20:25
When Redis stopped answering, the only Redis metric, cx_active, kept
reporting healthy-looking connections while every command on them was
stuck, and nothing reported commands failing, timing out or waiting.

Per Redis client (redis_pool / redis_per_second_pool), add:
- cmd_inflight (gauge), cmd_total, cmd_success, cmd_error,
  cmd_io_timeout, cmd_deadline_exceeded, cmd_rejected and a cmd_time
  timer for the commands (pipelines) issued by rate limit calls;
- cx_connect_error, and cx_local_close_error for connections closed
  after an error, which are also logged at warning level, rate-limited
  to once every 10s.

Adds utils.LogLimiter for the rate-limited log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
Stats sinks read timer values as milliseconds, but cmd_time was a
gostats NewTimer timespan, which records whole microseconds, so every
sample was 1000 times too large (a 22 ms command was sent as 22235|ms).
NewMilliTimer would fix the unit but record whole milliseconds, which
reads most Redis commands, well under a millisecond, as 0. Record
fractional milliseconds instead.

Add a Prometheus mapping that exports cmd_time from both Redis clients
as ratelimit_redis_cmd_time_seconds with a pool label. Scope
PROMETHEUS_RESPONSE_TIME_AS_MILLISECONDS to the response_time mapping:
it removed the first "scale: 0.001" in the mapper file, so its effect
depended on the order of the mappings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
@nacx
nacx force-pushed the pr/j-redis-pool-metrics branch from 5779103 to 8dc9d53 Compare November 9, 2026 19:28
@psbrar99
psbrar99 merged commit 085b18c into envoyproxy:main Nov 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants