Repository navigation
redis: report Redis command and connection metrics - #1301
Merged
Merged
Conversation
nacx
force-pushed
the
pr/j-redis-pool-metrics
branch
from
November 9, 2026 15:03
1cb0295 to
c1da9b3
Compare
Contributor
|
should we switch cmd_time to NewMilliTimer and add a Prometheus mapper entry. NewTimer records microseconds, but the sinks expect milliseconds, so a 22 ms command shows up as 22235|ms in statsd. With NewMilliTimer("cmd_time") it's 21|ms (tested). For Prometheus, add a mapping such as ratelimit_redis_pool_cmd_time_seconds with timer_type: histogram and scale: 0.001, placed after the response_time entry. PROMETHEUS_RESPONSE_TIME_AS_MILLISECONDS removes only the first scale: 0.001 in the file, so it would otherwise hit this one? |
Member
Author
|
Fixed |
When Redis stopped answering, the only Redis metric, cx_active, kept reporting healthy-looking connections while every command on them was stuck, and nothing reported commands failing, timing out or waiting. Per Redis client (redis_pool / redis_per_second_pool), add: - cmd_inflight (gauge), cmd_total, cmd_success, cmd_error, cmd_io_timeout, cmd_deadline_exceeded, cmd_rejected and a cmd_time timer for the commands (pipelines) issued by rate limit calls; - cx_connect_error, and cx_local_close_error for connections closed after an error, which are also logged at warning level, rate-limited to once every 10s. Adds utils.LogLimiter for the rate-limited log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
Stats sinks read timer values as milliseconds, but cmd_time was a gostats NewTimer timespan, which records whole microseconds, so every sample was 1000 times too large (a 22 ms command was sent as 22235|ms). NewMilliTimer would fix the unit but record whole milliseconds, which reads most Redis commands, well under a millisecond, as 0. Record fractional milliseconds instead. Add a Prometheus mapping that exports cmd_time from both Redis clients as ratelimit_redis_cmd_time_seconds with a pool label. Scope PROMETHEUS_RESPONSE_TIME_AS_MILLISECONDS to the response_time mapping: it removed the first "scale: 0.001" in the mapper file, so its effect depended on the order of the mappings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ignasi Barrera <ignasi@tetrate.io>
nacx
force-pushed
the
pr/j-redis-pool-metrics
branch
from
November 9, 2026 19:28
5779103 to
8dc9d53
Compare
psbrar99
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When Redis stops answering, the only Redis metric, cx_active, keeps reporting healthy-looking connections while every command on them is stuck, and nothing reports commands failing, timing out or waiting.
Per Redis client (redis_pool / redis_per_second_pool), add:
Adds utils.LogLimiter for the rate-limited log.