Things to keep in mind to improve MongoDB availability: Configuration
Kobayashi from the Service Reliability Group (SRG) of the Media Management Division (@berlinbytes)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
IntroductionOur MongoDB cluster configuration patternsStandaloneReplica setShardingFor high availabilityManagement ToolsMongoDB Cloud ManagerMongoDB AtlassummaryIn conclusion
Introduction
MongoDBAs you may know, it is a document-oriented distributed database that falls into the category of NoSQL databases.
At our companyNode.jsDue to its high compatibility with [the system/technology], it has been used for a long time, primarily in the gaming industry.
Around 2011, MongoDB began to be used partially, starting with version 1.X, and by the time version 2 was released, it was being used on a large scale, mainly in the social game industry.
Currently, due to its schema-less design and scalability, it is used not only in the gaming sector but also as a database for handling master data in media services.
When searching the web for information about our company's use of MongoDB, we find relatively old information, but there isn't much recent information available.
- Examples from Tapple
- Examples from Pig Party
Therefore, in this article, I would like to briefly introduce the current situation.
Our MongoDB clusterConfiguration patterns
Standalone
It consists of a single unit, without the need for replica sets or sharding.
It is mainly used for testing environments with small datasets, for investigating the deployment location of temporary data and logs, and for personal development environments.
Is it managed entirely manually?MongoDB Cloud ManagerIt operates under the management of [company name].
Replica set
While horizontal dispersion through sharding isn't strictly necessary, it's a system where the inability to use it would significantly disrupt business operations.Replica setThis has been adopted.
This is used for daily testing environments and some system functions.
Sharding
ShardingImplementing this solution enables you to handle applications requiring high throughput with large datasets, and allows for horizontal distribution of both read and write traffic across the entire cluster. The ease of scaling as needed is another advantage.
Furthermore, scalability can be improved by configuring each shard as an independent replica set.
On the other hand, an increase in costs is unavoidable because the number of components to be managed increases.
These are also managed under MongoDB Cloud Manager or run on MongoDB Atlas.
For high availability
At our company, most production environments utilize replica sets or sharding configurations.
In the past, we operated large-scale systems with over 30 shards in data centers, but with improved equipment performance and the extremely high performance of instances, we no longer operate physical servers.
Our platform is built on our own proprietary private cloud, "Cycloud," as well as on public clouds such as AWS and Google Cloud.
As can be seen from the currently used configuration patterns, since MongoDB is the database at the core of the service, the use of replica sets is necessary to achieve a high level of availability. Sharding is also utilized for services that require high read and write speeds. As the cluster grows larger and the amount of data to be managed increases, procedures such as scaling and version upgrades become more complex, so management tools are used to reduce the operational burden.
Management Tools
MongoDB Cloud Manager
MongoDB Cloud Manager automates infrastructure management to a near-complete degree, allowing you to perform upgrades, monitoring, backups, and more from a web-based UI.
MongoDB Atlas
MongoDB Atlas is a DBaaS (Database-as-a-Service) managed service from MongoDB, hosted on various public clouds.
Compared to MongoDB Cloud Manager, it offers automatic scaling based on workload, enabling performance and cost optimization. Additionally, tools such as index suggestions and performance advisors are available, further optimizing operational management costs.
As a result, opportunities to use official management tools are increasing. This is because using automated deployment management services reduces management costs.
Also, since this would just sound like an introductory article about MongoDB, let me add the following:
There are also examples of services that were using MongoDB migrating to other databases.
In recent years, databases with MongoDB-compatible APIs have also emerged, such as the following:
- Amazon DocumentDB
- Azure Cosmos DB for MongoDB
- Oracle Database API for MongoDB
I would like to write an article about these introductions and migration examples at some point.
summary
In summary, the MongoDB configuration is
- Use of replica set
- Introduction of sharding
This is important for ensuring availability and enables improved fault tolerance and load balancing.
Furthermore, the introduction of management tools contributes to reducing operational burden and operational errors.
And, while it depends on the specific application, the introduction of DBaaS is also highly effective.
Migrating to a database with compatible APIs can also be effective depending on the requirements and is worth considering.
In conclusion
This time, we introduced the MongoDB configuration used within our company from the perspective of availability.
Also, the following points were not mentioned this time:
- Considerations regarding memory size
- Determining instance specifications
- Query optimization
- Cost optimization
- About the index
- Introducing a cache layer
- About monitoring and surveillance
I plan to continue posting about various topics like these in the future.
SRG is looking for new team members.
If you are interested, please contact us here.