In the general case, the upgrade from MongoDB 2.2 to 2.4 is a binary-compatible “drop-in” upgrade: shut down the mongod instances and replace them with mongod instances running 2.4. However, before you attempt any upgrade please familiarize yourself with the content of this document, particularly the procedure for upgrading sharded clusters and the considerations for reverting to 2.2 after running 2.4.
Content
When upgrading, consider the following:
For all deployments using authentication, upgrade the drivers (i.e. client libraries), before upgrading the mongod instance or instances.
To upgrade to 2.4 sharded clusters must upgrade following the meta-data upgrade procedure.
If you’re using 2.2.0 and running with auth enabled, you will need to upgrade first to 2.2.1 and then upgrade to 2.4. See Rolling Upgrade Limitation for 2.2.0 Deployments Running with auth Enabled.
If you have system.users documents (i.e. for auth) that you created before 2.4 you must ensure that there are no duplicate values for the user field in the system.users collection in any database. If you do have documents with duplicate user fields, you must remove them before upgrading.
See Compatibility Change: User Uniqueness Enforced for more information.
You can upgrade to 2.4 by performing a “rolling” upgrade of the set by upgrading the members individually while the other members are available to minimize downtime. Use the following procedure:
Upgrade the secondary members of the set one at a time by shutting down the mongod and replacing the 2.2 binary with the 2.4 binary. After upgrading a mongod instance, wait for the member to recover to SECONDARY state before upgrading the next instance. To check the member’s state, issue rs.status() in the mongo shell.
Use the mongo shell method rs.stepDown() to step down the primary to allow the normal failover procedure. rs.stepDown() expedites the failover procedure and is preferable to shutting down the primary directly.
Once the primary has stepped down and another member has assumed PRIMARY state, as observed in the output of rs.status(), shut down the previous primary and replace mongod binary with the 2.4 binary and start the new process.
Note
Replica set failover is not instant but will render the set unavailable to read or accept writes until the failover process completes. Typically this takes 10 seconds or more. You may wish to plan the upgrade during a predefined maintenance window.
Important
Only upgrade sharded clusters to 2.4 if all members of the cluster are currently running instances of 2.2. The only supported upgrade path for sharded clusters running 2.0 is via 2.2.
Upgrading a sharded cluster from MongoDB version 2.2 to 2.4 (or 2.3) requires that you run a 2.4 mongos with the --upgrade option, described in this procedure. The upgrade process does not require downtime.
The upgrade to MongoDB 2.4 adds epochs to the meta-data for all collections and chunks in the existing cluster. MongoDB 2.2 processes are capable of handling epochs, even though 2.2 did not require them.
This procedure applies only to upgrades from version 2.2. Earlier versions of MongoDB do not correctly handle epochs.
Warning
Note
The upgraded config database will require more storage space than before, to make backup and working copies of the config.chunks and config.collections collections. As always, if storage requirements increase, the mongod might need to pre-allocate additional data files. See What tools can I use to investigate storage use in MongoDB? for more information.
Changes to the meta-data format for sharded clusters, stored in the config database, require a special meta-data upgrade procedure when moving to 2.4.
Do not perform operations that modify meta-data while performing this procedure. See Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4 for examples of prohibited operations.
Before you start the upgrade, ensure that the amount of free space on the filesystem for the config database is 4 to 5 times the amount of space currently used by the config database data files.
Turn off the balancer in the sharded cluster, as described in Disable the Balancer.
Optional
For additional security during the upgrade, you can make a backup of the config database using mongodump or other backup tools.
Ensure there are no version 2.0 mongod or mongos processes still active in the sharded cluster. The automated upgrade process checks for 2.0 processes, but network availability can prevent a definitive check. Wait 5 minutes after stopping or upgrading version 2.0 mongos processes to confirm that none are still active.
Start a single 2.4 mongos process with configdb pointing to the sharded cluster’s config servers and with the --upgrade option. The upgrade process happens before the process becomes a daemon (i.e. before --fork.)
You can upgrade an existing mongos instance to 2.4 or you can start a new mongos instance that can reach all config servers if you need to avoid reconfiguring a production mongos.
Start the mongos with a command that resembles the following:
mongos --configdb <config server> --upgrade
Without the --upgrade option 2.4 mongos processes will fail to start until the upgrade process is complete.
The upgrade will prevent any chunk moves or splits from occurring during the upgrade process. If there are very many sharded collections or there are stale locks held by other failed processes, acquiring the locks for all collections can take seconds or minutes. See the log for progress updates.
When the mongos process starts successfully, the upgrade is complete. If the mongos process fails to start, check the log for more information.
If the mongos terminates or loses its connection to the config servers during the upgrade, you may always safely retry the upgrade.
However, if the upgrade failed during the short critical section, the mongos will exit and report that the upgrade will require manual intervention. To continue the upgrade process, you must follow the Resync after an Interruption of the Critical Section procedure.
Optional
If the mongos logs show the upgrade waiting for the upgrade lock, a previous upgrade process may still be active or may have ended abnormally. After 15 minutes of no remote activity mongos will force the upgrade lock. If you can verify that there are no running upgrade processes, you may connect to a 2.2 mongos process and force the lock manually:
mongo <mongos.example.net>
db.getMongo().getCollection("config.locks").findOne({ _id : "configUpgrade" })
If the process specified in the process field of this document is verifiably offline, run the following operation to force the lock.
db.getMongo().getCollection("config.locks").update({ _id : "configUpgrade" }, { $set : { state : 0 } })
It is always more safe to wait for the mongos to verify that the lock is inactive, if you have any doubts about the activity of another upgrade operation. In addition to the configUpgrade, the mongos may need to wait for specific collection locks. Do not force the specific collection locks.
Upgrade and restart other mongos processes in the sharded cluster, without the --upgrade option.
See Complete Sharded Cluster Upgrade for more information.
Re-enable the balancer. You can now perform operations that modify cluster meta-data.
Once you have upgraded, do not introduce version 2.0 MongoDB processes into the sharded cluster. This can reintroduce old meta-data formats into the config servers. The meta-data change made by this upgrade process will help prevent errors caused by cross-version incompatibilities in future versions of MongoDB.
During the short critical section of the upgrade that applies changes to the meta-data, it is unlikely but possible that a network interruption can prevent all three config servers from verifying or modifying data. If this occurs, the config servers must be re-synced, and there may be problems starting new mongos processes. The sharded cluster will remain accessible, but avoid all cluster meta-data changes until you resync the config servers. Operations that change meta-data include: adding shards, dropping databases, and dropping collections.
Note
Only perform the following procedure if something (e.g. network, power, etc.) interrupts the upgrade process during the short critical section of the upgrade. Remember, you may always safely attempt the meta data upgrade procedure.
To resync the config servers:
Turn off the balancer in the sharded cluster and stop all meta-data operations. If you are in the middle of an upgrade process (Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4), you have already disabled the balancer.
Shut down two of the three config servers, preferably the last two listed in the configdb string. For example, if your configdb string is configA:27019,configB:27019,configC:27019, shut down configB and configC. Shutting down the last two config servers ensures that most mongos instances will have uninterrupted access to cluster meta-data.
mongodump the data files of the active config server (configA).
Move the data files of the deactivated config servers (configB and configC) to a backup location.
Create new, empty data directories.
Restart the disabled config servers with --dbpath pointing to the now-empty data directory and --port pointing to an alternate port (e.g. 27020).
Use mongorestore to repopulate the data files on the disabled documents from the active config server (configA) to the restarted config servers on the new port (configB:27020,configC:27020). These config servers are now re-synced.
Restart the restored config servers on the old port, resetting the port back to the old settings (configB:27019 and configC:27019).
In some cases connection pooling may cause spurious failures, as the mongos disables old connections only after attempted use. 2.4 fixes this problem, but to avoid this issue in version 2.2, you can restart all mongos instances (one-by-one, to avoid downtime) and use the rs.stepDown() method before restarting each of the shard replica set primaries.
The sharded cluster is now fully resynced; however before you attempt the upgrade process again, you must manually reset the upgrade state using a version 2.2 mongos. Begin by connecting to the 2.2 mongos with the mongo shell:
mongo <mongos.example.net>
Then, use the following operation to reset the upgrade process:
db.getMongo().getCollection("config.version").update({ _id : 1 }, { $unset : { upgradeState : 1 } })
Finally retry the upgrade process, as in Upgrade a Sharded Cluster from MongoDB 2.2 to MongoDB 2.4.
After you have successfully completed the meta-data upgrade process described in Meta-Data Upgrade Procedure, and the 2.4 mongos instance starts, you can upgrade the other processes in your MongoDB deployment.
While the balancer is still disabled, upgrade the components of your sharded cluster in the following order:
When this process is complete, you can now re-enable the balancer.
MongoDB cannot support deployments that mix 2.2.0 and 2.4.0, or greater, components. MongoDB version 2.2.1 and later processes can exist in mixed deployments with 2.4-series processes. Therefore you cannot perform a rolling upgrade from MongoDB 2.2.0 to MongoDB 2.4.0. To upgrade a cluster with 2.2.0 components, use one of the following procedures.
If you used a mongod from the 2.3 or 2.4-rc (release candidate) series, you can safely transition these databases to 2.4.0 or later; however, if you created 2dsphere or text indexes using a mongod before v2.4-rc2, you will need to rebuild these indexes. For example:
db.records.dropIndex( { loc: "2dsphere" } )
db.records.dropIndex( "records_text" )
db.records.ensureIndex( { loc: "2dsphere" } )
db.records.ensureIndex( { records: "text" } )
For some cases the on-disk format of data files used by 2.4 and 2.2 mongod is compatible, and you can upgrade and downgrade if needed. However, several new features in 2.4 are incompatible with previous versions:
Note
In sharded clusters, once you have completed the meta-data upgrade procedure, you cannot use 2.0 mongod or mongos instances in the same cluster.
If you complete the meta-data upgrade, you can have a mixed cluster that has both 2.2 and 2.4 mongod and mongos instances, if needed. However, do not create 2dsphere or text indexes in a cluster that has 2.2 components.
If you upgrade to MongoDB 2.4, and then need to run MongoDB 2.2 with the same data files, consider the following limitations.
Except as described below, moving between 2.2 and 2.4 is a drop-in replacement:
stop the existing mongod, using the --shutdown option as follows:
mongod --dbpath /var/mongod/data --shutdown
Replace /var/mongod/data with your MongoDB dbpath.
start the new mongod processes with the same dbpath setting, for example:
mongod --dbpath /var/mongod/data
Replace /var/mongod/data with your MongoDB dbpath.
If you have created 2dsphere or text indexes while running a 2.4 mongod instance, you can downgrade at any time, by starting the 2.2 mongod with the --upgrade option as follows:
mongod --dbpath /var/mongod/data/ --upgrade
Then, you will need to drop any existing 2dsphere or text indexes using db.collection.dropIndex(), for example:
db.records.dropIndex( { loc: "2dsphere" } )
db.records.dropIndex( "records_text" )
Warning
--upgrade will run repairDatabase on any database where you have created a 2dsphere or text index, which will rebuild all indexes.
If you do not use --upgrade, when you attempt to start a 2.2 mongod and you have created a 2dsphere or text index, mongod will return the following message:
'need to upgrade database index_plugin_upgrade with pdfile version 4.6, new version: 4.5 Not upgrading, exiting'
While running 2.4, to check the data file version of a MongoDB database, use the following operation in the shell:
db.getSiblingDB('<databasename>').stats().dataFileVersion
The major data file version for both 2.2 and 2.4 is 4, the minor data file version for 2.2 is 5 and the minor data file version for 2.4 is 6 if you have created a 2dsphere or text.