OPTIONS

Sharded Cluster High Availability

A production cluster has no single point of failure. This section introduces the availability concerns for MongoDB deployments in general and highlights potential failure scenarios and available resolutions.

Application Servers or mongos Instances Become Unavailable

If each application server has its own mongos instance, other application servers can continue access the database. Furthermore, mongos instances do not maintain persistent state, and they can restart and become unavailable without losing any state or data. When a mongos instance starts, it retrieves a copy of the config database and can begin routing queries.

A Single mongod Becomes Unavailable in a Shard

Replica sets provide high availability for shards. If the unavailable mongod is a primary, then the replica set will elect a new primary. If the unavailable mongod is a secondary, and it disconnects the primary and secondary will continue to hold all data. In a three member replica set, even if a single member of the set experiences catastrophic failure, two other members have full copies of the data. [1]

Always investigate availability interruptions and failures. If a system is unrecoverable, replace it and create a new member of the replica set as soon as possible to replace the lost redundancy.

All Members of a Replica Set Become Unavailable

If all members of a replica set within a shard are unavailable, all data held in that shard is unavailable. However, the data on all other shards will remain available, and it’s possible to read and write data to the other shards. However, your application must be able to deal with partial results, and you should investigate the cause of the interruption and attempt to recover the shard as soon as possible.

One or Two Config Databases Become Unavailable

Three distinct mongod instances provide the config database using a special two-phase commits to maintain consistent state between these mongod instances. Cluster operation will continue as normal but chunk migration and the cluster can create no new chunk splits. Replace the config server as soon as possible. If all config databases become unavailable, the cluster can become inoperable.

Note

All config servers must be running and available when you first initiate a sharded cluster.

[1]If an unavailable secondary becomes available while it still has current oplog entries, it can catch up to the latest state of the set using the normal replication process, otherwise it must perform an initial sync.

Renaming Config Servers and Cluster Availability

If the name or address that a sharded cluster uses to connect to a config server changes, you must restart every mongod and mongos instance in the sharded cluster. Avoid downtime by using CNAMEs to identify config servers within the MongoDB deployment.

To avoid downtime when renaming config servers, use DNS names unrelated to physical or virtual hostnames to refer to your config servers.

Generally, refer to each config server using the DNS alias (e.g. a CNAME record). When specifying the config server connection string to mongos, use these names. These records make it possible to change the IP address or rename config servers without changing the connection string and without having to restart the entire cluster.

Shard Keys and Cluster Availability

The most important consideration when choosing a shard key are:

  • to ensure that MongoDB will be able to distribute data evenly among shards, and
  • to scale writes across the cluster, and
  • to ensure that mongos can isolate most queries to a specific mongod.

Furthermore:

  • Each shard should be a replica set, if a specific mongod instance fails, the replica set members will elect another to be primary and continue operation. However, if an entire shard is unreachable or fails for some reason, that data will be unavailable.
  • If the shard key allows the mongos to isolate most operations to a single shard, then the failure of a single shard will only render some data unavailable.
  • If your shard key distributes data required for every operation throughout the cluster, then the failure of the entire shard will render the entire cluster unavailable.

In essence, this concern for reliability simply underscores the importance of choosing a shard key that isolates query operations to a single shard.