Skip to main content

Upgrade

Keeping MooseFS up to date ensures improved performance, enhanced security, and access to new features. This chapter provides general guidelines for upgrading MooseFS, covering best practices, potential challenges, and steps to minimize downtime.

Whether performing a minor update or a major version upgrade, careful planning is crucial to maintaining data integrity and system stability. Follow the recommendations in this chapter to ensure a smooth and successful upgrade process.

Packages' names​

For the sake of brevity, we are using package names corresponding to the Pro version of MooseFS, whenever the upgrade instructions for Pro and CE versions are identical, which is most of the time. If you are using MooseFS CE, simply remove -pro from packages names when following instructions.

Before any kind of upgrade​

To safely and securely perform an upgrade, we recommend that you first prepare your MooseFS instance:

Read the NEWS file​

Read the NEWS file attached to MooseFS distribution and check if there are any new features that will require some extra action (e.g. new configuration options that you want to change from their default settings)

Check your chunk matrix​

Check your chunk matrix via GUI or CLI and make sure there are no endangered or undergoal chunks. If any exists, wait for the system to replicate them. If there are hardware issues (like faulty hard drives) that are causing your system to have undergoal chunks, try to resolve these issues before you proceed to upgrade your instance

Plan the timing​

Plan the timing of your upgrade to a time when you can either disconnect all your client processes and perform the upgrade on the system without I/O or when the I/O is at its lowest (e.g. during the night). While MooseFS can be successfully and safely upgraded during normal operating time, module’s restarts will cause waiting times for I/O. The client (mount) process itself will handle the delays, but external processes using files stored on MooseFS may show timeouts or errors, depending on their architecture. This step is just a precaution, if you are sure that all processes using files stored on MooseFS have reasonably long timeouts for I/O waits, you don’t need to adjust your upgrade time so strictly.

Save metadata​

Just before you start your upgrade, if you already have MooseFS 4, use the mfssupervisor tool to force Master(s) metadata save to disk. if you are using the Pro version, use the ”all servers with CRC check” mode and verify via GUI or CLI that the CRC of metadata matches on all Masters.

Make a backup of your Master's data before any Master upgrade​

When, during any of the upgrade procedures, you will be instructed to ”upgrade your Master module”, just before you start performing that step, make a backup of your Master’s metadata (for the Pro version it’s enough to do this just before the first Follower Master Server upgrade).

How to find metadata and changelog files?​

To find your metadata files and changelog files, do the following:

  1. Check in your Master’s config file what is the location of the Master’s working directory, defined by the variable DATA_PATH.
  2. Go to that directory and locate a file named metadata.mfs.back and all files named changelog.*.mfs. These files are needed to restore full metadata information.
  3. Copy them to a secure location for the duration of the upgrade.
warning

If you are stopping and starting, rather then restarting, your Master modules during the upgrade, then, after stopping the Master process, in the same location you will find a file named metadata.mfs - make a backup of this file instead of the other ones.

Upgrading from a previous major version of MooseFS to MooseFS 4​

MooseFS is designed to be backward compatible as much as possible. However, it’s impossible to assure that there is 100% compatibility between all possible combinations of versions.

Prior to upgrading to MooseFS 4, ensure that your current version is at least MooseFS 2.1.x. If you are using an older version, first upgrade to the latest MooseFS 2.1.x release, then proceed to MooseFS 3.x, and finally to MooseFS 4.x.

It is possible to switch (both ways) between PRO and CE version during upgrade. Note following guidelines when performing such upgrade:

Upgrade CE → CE or Pro → Pro​

Upgrade incrementally through major versions. For example, upgrade from 2.1.7 to 2.1.12, then to 3.0.118, and finally to the latest MooseFS 4 version.

Upgrade CE → Pro​

  1. Upgrade to the Latest CE Version First

    Whenever possible, upgrade your instance to the latest available CE version before switching to Pro. If upgrading to the latest CE version is not feasible, contact MooseFS Pro Support for personalized guidance.

  2. Switching and Upgrading Simultaneously

    A version upgrade combined with a switch from CE to Pro is possible but not recommended. This approach introduces additional complexity and risks. If this is your only option, always consult MooseFS Pro Support for tailored guidance and instructions, as the procedure can vary based on your current version and specific setup.

  3. Special Upgrade Order for CE → Pro

    Unlike other upgrade scenarios, the package upgrade order for CE to Pro differs:

    • First, upgrade all chunkservers to the Pro version.
    • Next, upgrade the master(s) to the Pro version.
    • Finally, upgrade and remount the client mounts.

Switching Pro → CE​

Switching is only possible between identical versions (e.g., 4.57.1 Pro to 4.57.1 CE). Follow the upgrade order for minor version upgrades and remove any obsolete packages afterward.

Upgrading between minor versions of MooseFS 4​

MooseFS 4 version numbers consist of 3 parts: major number (4), minor number, and patch level (e.g. MooseFS version 4.55.5 has the major number of 4, minor number of 55, and patch level of 5). When performing an upgrade, you need to follow these rules:

  1. Modules with the same minor version and different patch levels can all cooperate freely.
  2. Modules with different minor version may not have compatible protocols (i.e. different communication packets or different metadata format) so a MooseFS instance using different minor versions needs to always follow these rules:
    • The metalogger module must have the same or higher version as the Master module
    • The Master module must have the same or higher version as chunk server modules. In Pro version Follower Master server must have the same or higher version as Leader Master server.
    • The Master and chunk server modules must have the same or higher version as the client (mount) modules.
  3. It is strongly recommended that the GUI module has the the same or higher version as the Master server(s); GUI in lower version will work with the Master, but may not show all informations needed

In the light of the above rules, the recommended order of upgrade is as follows (refer to Installing MooseFS 4 for instructions on how to install the appropriate modules on your particular OS):

1. Upgrade GUI, CLI and netdump​

First, upgrade the moosefs-pro-gui, moosefs-pro-cli and moosefs-pro-netdump (if you are using it) packages, restart necessary services for the new GUI to be loaded.

2. Upgrade metalogger​

Next, upgrade your moosefs-pro-metalogger package (if you are using it; the use of this package is recommended for both the CE and the Pro version) and restart your metalogger process

3. Upgrade Masters​

Depending on the version you are using:

  • For CE version, upgrade your Master moosefs-master package and restart your Master process. Wait for all the chunk servers to connect and finish the chunk registration process, this information is displayed on the INFO tab of GUI and is logged to system logs.

  • For Pro version, upgrade first all your follower Master modules, one by one, by upgrading the moosefs-pro-master package and restarting the Master process. After upgrading and restarting each module wait for it to connect to Leader Master server and become a functioning follower again (the state shown on the INFO tab of GUI should be FOLLOWER and not FOLLOWER (DESYNC) or anything else). Then upgrade your Leader Master module. Restart of the Master leader module will cause one of the followers to take over the leader role and the restarted ex-leader will now be a follower. If you want this particular machine to host the leader process again, use the mfssupervisor tool to switch the leader to this machine. Wait for the switching process to complete before you proceed to the next step.

4. Upgrade Chunkservers​

Upgrade your Chunkserver modules, one by one,by upgrading the moosefs-pro-chunkserver packet and restarting the chunk server process. Wait for the chunk server to connect to the Leader Master server and register all of its chunks before you proceed to the next one. Observe the CGI tab INFO for information about finished chunk registration

5. Upgrade clients​

Upgrade your clients by upgrading the moosefs-pro-client packet and remounting. This can be done in parallel.