OPTIONS

Migrate a Sharded Cluster to Different Hardware

This procedure moves the components of the sharded cluster to a new hardware system without downtime for reads and writes.

Important

While the migration is in progress, do not attempt to change to the cluster metadata. Do not use any operation that modifies the cluster metadata in any way. For example, do not create or drop databases, create or drop collections, or use any sharding commands.

If your cluster includes a shard backed by a standalone mongod instance, consider converting the standalone to a replica set to simplify migration and to let you keep the cluster online during future maintenance. Migrating a shard as standalone is a multi-step process that may require downtime.

To migrate a cluster to new hardware, perform the following tasks.

Disable the Balancer

Disable the balancer to stop chunk migration and do not perform any metadata write operations until the process finishes. If a migration is in progress, the balancer will complete the in-progress migration before stopping.

To disable the balancer, connect to one of the cluster’s mongos instances and issue the following method:

sh.stopBalancer()

To check the balancer state, issue the sh.getBalancerState() method.

For more information, see Disable the Balancer.

Migrate Each Config Server Separately

Migrate each config server by starting with the last config server listed in the configDB string. Proceed in reverse order of the configDB string. Migrate and restart a config server before proceeding to the next. Do not rename a config server during this process.

Note

If the name or address that a sharded cluster uses to connect to a config server changes, you must restart every mongod and mongos instance in the sharded cluster. Avoid downtime by using CNAMEs to identify config servers within the MongoDB deployment.

See Migrate Config Servers with Different Hostnames for more information.

Important

Start with the last config server listed in configDB.

  1. Shut down the config server.

    This renders all config data for the sharded cluster “read only.”

  2. Change the DNS entry that points to the system that provided the old config server, so that the same hostname points to the new system. How you do this depends on how you organize your DNS and hostname resolution services.

  3. Copy the contents of dbPath from the old config server to the new config server.

    For example, to copy the contents of dbPath to a machine named mongodb.config2.example.net, you might issue a command similar to the following:

    rsync -az /data/configdb/ mongodb.config2.example.net:/data/configdb
    
  4. Start the config server instance on the new system. The default invocation is:

    mongod --configsvr
    

Restart the mongos Instances

If the configDB string will change as part of the migration, you must shut down all mongos instances before changing the configDB string. This avoids errors in the sharded cluster over configDB string conflicts.

If the configDB string will remain the same, you can migrate the mongos instances sequentially or all at once.

  1. Shut down the mongos instances using the shutdown command. If the configDB string is changing, shut down all mongos instances.

  2. If the hostname has changed for any of the config servers, update the configDB string for each mongos instance. The mongos instances must all use the same configDB string. The strings must list identical host names in identical order.

    Tip

    To avoid downtime, give each config server a logical DNS name (unrelated to the server’s physical or virtual hostname). Without logical DNS names, moving or renaming a config server requires shutting down every mongod and mongos instance in the sharded cluster.

  3. Restart the mongos instances being sure to use the updated configDB string if hostnames have changed.

For more information, see Start the mongos Instances.

Migrate the Shards

Migrate the shards one at a time. For each shard, follow the appropriate procedure in this section.

Migrate a Replica Set Shard

To migrate a sharded cluster, migrate each member separately. First migrate the non-primary members, and then migrate the primary last.

If the replica set has two voting members, add an arbiter to the replica set to ensure the set keeps a majority of its votes available during the migration. You can remove the arbiter after completing the migration.

Migrate a Member of a Replica Set Shard

  1. Shut down the mongod process. To ensure a clean shutdown, use the shutdown command.

  2. Move the data directory (i.e., the dbPath) to the new machine.

  3. Restart the mongod process at the new location.

  4. Connect to the replica set’s current primary.

  5. If the hostname of the member has changed, use rs.reconfig() to update the replica set configuration document with the new hostname.

    For example, the following sequence of commands updates the hostname for the instance at position 2 in the members array:

    cfg = rs.conf()
    cfg.members[2].host = "pocatello.example.net:27017"
    rs.reconfig(cfg)
    

    For more information on updating the configuration document, see Examples.

  6. To confirm the new configuration, issue rs.conf().

  7. Wait for the member to recover. To check the member’s state, issue rs.status().

Migrate the Primary in a Replica Set Shard

While migrating the replica set’s primary, the set must elect a new primary. This failover process which renders the replica set unavailable to perform reads or accept writes for the duration of the election, which typically completes quickly. If possible, plan the migration during a maintenance window.

  1. Step down the primary to allow the normal failover process. To step down the primary, connect to the primary and issue the either the replSetStepDown command or the rs.stepDown() method. The following example shows the rs.stepDown() method:

    rs.stepDown()
    
  2. Once the primary has stepped down and another member has become PRIMARY state. To migrate the stepped-down primary, follow the Migrate a Member of a Replica Set Shard procedure

    You can check the output of rs.status() to confirm the change in status.

Migrate a Standalone Shard

The ideal procedure for migrating a standalone shard is to convert the standalone to a replica set and then use the procedure for migrating a replica set shard. In production clusters, all shards should be replica sets, which provides continued availability during maintenance windows.

Migrating a shard as standalone is a multi-step process during which part of the shard may be unavailable. If the shard is the primary shard for a database,the process includes the movePrimary command. While the movePrimary runs, you should stop modifying data in that database. To migrate the standalone shard, use the Remove Shards from an Existing Sharded Cluster procedure.

Re-Enable the Balancer

To complete the migration, re-enable the balancer to resume chunk migrations.

Connect to one of the cluster’s mongos instances and pass true to the sh.setBalancerState() method:

sh.setBalancerState(true)

To check the balancer state, issue the sh.getBalancerState() method.

For more information, see Enable the Balancer.