About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Preventing problems with MongoDB Atlas and Cloud Manager! A summary of troubleshooting examples and preventative measures.

Preventing problems with MongoDB Atlas and Cloud Manager! A summary of troubleshooting examples and preventative measures.

This is Kobayashi (@berlinbytes) from the Service Reliability Group (SRG) of the Media Division.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article will introduce potential problems and preventative measures based on our experience using MongoDB, including case studies.
 

Introduction

Our company has been using MongoDB since the early stages of developing social games.
Its history began with operation in physical enclosures, then evolved into more highly concentrated hosts, and finally into servers with high-speed storage such as SSDs and NVMe.
Currently, its use is predominantly in private clouds, virtual instances on public clouds, and managed services.
MongoDB is a NoSQL database that enables flexible data management using JSON-like documents.
The introduction of management tools such as MongoDB Atlas and Cloud Manager has changed the key points to focus on during operation.

Learning from past troubles: the risks that still lurk today

Most of the problems we've encountered so far are still possible today. Aside from issues specific to particular versions, we'll explain the main problems here, categorized into three groups.

Version-related issues

In some cases, support was discontinued due to outdated versions of the MongoDB core or the MongoDB Agent, which is essential for operation with Cloud Manager.
The MongoDB Agent is a crucial component that provides features such as monitoring, backup, and automation in a single binary.
Once support ends, tasks such as manually performing upgrades by directly connecting to the instance via SSH will become necessary, leading to an increased operational burden.

Backup problems

Version issues can also affect backups.
Backups may not be possible for versions of MongoDB that have reached end-of-life (EOL).
Additionally, there are cases where manual errors during the process necessitate a backup resynchronization (resync).

Collection or index configuration errors

Database design flaws can have a serious impact on performance.
In particular, the following configuration errors can cause abnormally high loads on the entire cluster or on specific instances.
  • Concentration of access to non-sharded collections
    • Sharding is a horizontal partitioning technique that distributes data across multiple servers.
  • Executing inefficient queries without using indexes
    • A query is a request or command made to a database.
    • If indexes are not properly configured, data retrieval will take a long time, resulting in slow queries.
  • Design that causes data bias
    • If sharding is performed using master data with a small amount of data as the key, or if the data is composed of data with low cardinality (number of possible values), the data will be concentrated on specific shards (distributed servers), leading to a concentration of load.
These problems can be summarized into the following three points:
  • Failures and their signs are often missed due to inadequate monitoring and alerting.
  • Delays in keeping up with agent and binary versions can lead to loss of support or inability to use new features.
  • High load and slow queries can occur due to misconfigurations of collections and indexes.

Specific preventative measures for stable operation

To prevent the aforementioned problems from occurring, preventative measures are crucial.

First, keep up with the version.

MongoDB currently releases major versions approximately once a year.
While immediate updates aren't always necessary, it's recommended to periodically assess whether version updates are required, taking End of Life (EOL) policies into consideration.
In particular, for patch releases that include security fixes and bug fixes, it is desirable to adopt an upgrade policy to the latest version.

Establish appropriate alerts.

Establishing an alert system is crucial for early detection of problems.
Here, we'll introduce some typical alerts that should be configured in both MongoDB Atlas and Cloud Manager.

In the case of MongoDB Atlas

From the Atlas UI, set up an alert with the following conditions:
  • Backup problems
    • Snapshot failed
    • Snapshot schedule fell behind
  • Collection or index configuration errors
    • Query Targeting: Scanned Objects / Returned
    • Host has index suggestions

In the case of MongoDB Cloud Manager

From the Cloud Manager UI, set up an alert with the following conditions:
  • Version-dependent
    • Monitoring does not have the latest version
    • Host does not have the latest version
  • Backup problems
    • Backup does not have the latest version
    • Backup is down
    • Backup requires a resync
    • Backup oplog is behind
  • Collection or index structure errors
    • Host has index suggestions

List of alerts to configure in MongoDB Atlas

In addition to the alerts mentioned above, we recommend the following settings as best practice.
  • Atlas Auto Scaling
    • Compute auto-scaling initiated for base tier
    • Compute auto-scaling initiated for analytics tier
    • Disk auto-scaling initiated
  • Backup
    • Snapshot failed
    • Snapshot schedule fell behind
  • Billing
    • Credit card is about to expire
  • Maintenance Window
    • Maintenance is scheduled
    • Maintenance started
  • Host
    • Host has index suggestions
    • System: CPU (User) % above 95
    • Query Targeting: Scanned Objects / Returned above 1000
    • Disk space % used on Data Partition above 90
    • Connections % of configured limit above 80
  • Limit
    • An overall request rate limit has been hit
  • Replica Set
    • Replica set has no primary
    • Replication Oplog Window is below 1 hours

List of alerts to configure in MongoDB Cloud Manager

In a Cloud Manager environment, the following settings are recommended:
  • Agent
    • Monitoring is down
    • Backup is down
    • Monitoring does not have the latest version
    • Backup does not have the latest version
  • Backup
    • Backup oplog is behind
    • Backup requires a resync
  • Billing
    • Credit card is about to expire
  • Host
    • Host is down
    • Host is exposed to the public Internet
    • Host is recovering
    • Host does not have the latest version
  • Replica Set
    • Number of healthy members is...

Metrics that should be collected using external tools such as Datadog

In addition to the standard features of Atlas and Cloud Manager, leveraging external monitoring tools such as Datadog and Percona Monitoring and Management (PMM) enables more flexible alert configuration and earlier detection of failures.
Below are some key metrics that you should collect.

Hosts

  • CPUProcess CPU usage
  • Memory: Memory usage
  • Network I/O UsageNetwork usage
  • Disk Usage / I/O UsageDisk usage and I/O volume
  • Page FaultsPage fault occurrence status

DB unit

  • Read Request (query, getmore)Increase or decrease in read requests
  • Write Request (insert, update, delete)Increase or decrease in write requests
  • Assertions: Number of assertion errors

Replication (ReplicaSet) unit

  • ConnectionsNumber of connections
  • oplog size / oplog window size: Operation log size and retention period
  • Replication LagReplication latency
  • Read / Write tickets: Read/Write waiting status
  • Lock (Read / Write)Lock occurrence status

In conclusion

To ensure stable operation of MongoDB, keeping up with versions, maintaining healthy backups, and configuring appropriate alerts are extremely important.
In particular, version control and monitoring can prevent many problems from occurring.
Furthermore, by combining external tools such as Datadog, more detailed metrics-based monitoring and rapid problem resolution become possible, leading to increased operational efficiency and sophistication.
I also hope to introduce other useful third-party tools such as Datadog Database Monitoring for MongoDB and Percona Monitoring and Management.
 
If you are interested in SRG, please contact us here.