Prevent response degradation! A summary of the basics and important metrics for monitoring Redis/Valkey.
This is Kobayashi (@berlinbytes) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article explains the essential monitoring concepts for stable operation of Redis/Valkey, and introduces specific metrics and practical steps to prevent response degradation.
Introduction
At our company, Redis and its open-source fork, Valkey, are used as extremely fast in-memory data stores for a wide range of applications, including caching, session storage, real-time analytics, and primary databases.
Its defining characteristic is its incredibly low latency, achieved by directly manipulating data in memory.
However, due to its high speed, even a slight decrease in performance can affect the overall system responsiveness and potentially detract from the user experience.
This article summarizes essential monitoring concepts for stable operation of Redis and Valkey, and introduces specific metrics and practical steps to prevent response degradation.
Five key layers that make up monitoring
To build an effective monitoring system, it is helpful to consider the system in terms of multiple layers.
Here, we will explain the five essential monitoring layers.
1. Performance (Latency & Throughput)
Performance is the metric that most directly impacts the user experience.
It is important to measure both the round-trip latency from the application's perspective and the command processing delay within the server.
By monitoring both the number of commands processed per second and the cache hit rate, we gain a comprehensive understanding of performance.
Since Redis/Valkey typically responds in sub-milliseconds, even a slight increase in latency is a sign of a significant change.
Furthermore, since Redis/Valkey is fundamentally single-threaded, CPU usage is also an important indicator of server performance.
LATENCY HISTORY2. Memory health
For Redis/Valkey, an in-memory database, memory is the most important resource.
evicted_keysA fragmentation rate consistently exceeding 1.5 is a sign that fragmentation is too advanced.
Conversely, if the value remains below 1.0 for an extended period, it may indicate that the OS is performing swapping operations, and caution is advised.
3. Replication and Clustering
When implementing replication or cluster configurations to increase availability, monitoring their health is also essential.
It monitors data synchronization delays between the primary and replica (replication lag), synchronization position differences (offset), and connection status (link status).
Managed services such as AWS ElastiCache and Google Cloud Memorystore often provide these metrics as standard.
4. The impact of persistence and forking
When using snapshots (RDB) or append-only files (AOF) for data persistence, background saving processes can impact performance.
fork()Recording flags indicating the execution status of background saving processes and the time taken for those processes can help in troubleshooting performance issues.
5. Connect to the client
Client connection status is also an important area to monitor.
It monitors the current number of connected clients, the number of clients waiting due to blocking commands, and the number of connections being rejected.
If connection rejections occur, it may be because the maximum number of clients has been reached. You will need to review your resources and check the connection pooling settings on the application side.
Key metrics and alerts that should be monitored with top priority.
This section introduces core metrics that are particularly important when starting monitoring, as well as examples of alert settings for detecting anomalies.
List of core metrics
- memory
maxmemorymem_fragmentation_ratioevicted_keysactive_defrag_*
- CPU usage
- CPU usage of the main thread
- Caution when under heavy load
- activity
connected_clients- Replication lag
- Latency
- Command processing delay. This is evaluated in conjunction with the application's response time.
- Cash efficiency
- Cache hit rate. This indicates the efficiency when using it as a cache.
- network
- Round-trip time (RTT) between the client and the instance.
Practical alert setting examples
LATENCY
evicted_keys
- Memory shortage: This occurs when memory usage exceeds 80% and the fragmentation rate remains high.
- Replication delay: This occurs when the replication lag persists beyond a predetermined acceptable limit (e.g., 1-3 seconds).
blocked_clients
How to create an effective monitoring dashboard
Rather than simply collecting metrics, creating dashboards that allow you to understand the situation at a glance will significantly improve operational efficiency.
The following is an example of a dashboard configuration.
PING
SET
evicted_keys
- Throughput Panel: Displays the number of commands processed per second (OPS), network transmission and reception volume, and the amount of data transferred via replication, allowing you to understand load trends.
- Client Panel: Monitors trends such as the number of connected clients, blocked clients, rejected connections, and authentication failures.
- Cache Efficiency Panel: Displays the cache hit rate in a separate, large panel, allowing for early detection of sudden drops.
Action plan for implementing monitoring
Finally, I will introduce a concrete action plan for implementing monitoring.
Establishing a baseline
First, we'll understand how the system behaves under normal circumstances.
Over a period of one to two weeks, key metrics such as latency, memory usage, OPS, and number of connections are recorded to establish baseline trends for different times of day and days of the week.
Initial alert settings
Based on a baseline, set alerts with realistic thresholds.
For example, you can set the latency to start at "baseline + 5ms," the memory usage to start at values like "80%" or "90%," and configure it to notify you immediately if a connection is refused or a client is blocked.
Establishing an analysis cycle
We will establish a system for investigating the cause of alerts when they occur.
LATENCY DOCTORContinuous threshold adjustment
As we continue to operate the service, I expect that the normal ranges for metrics such as cache hit rate and fragmentation rate will change as the service grows.
Based on that data, it is necessary to continuously review and improve the alert thresholds to reduce false positives and detect anomalies earlier.
In conclusion
Building a proactive monitoring system through the steps outlined will lead to stable operation of services utilizing Redis/Valkey and the provision of an excellent user experience.
Finally, here are some links to best practices related to various cloud services and their official configurations and operations.
SRG is looking for new team members.
If you are interested, please contact us here.