Skip to main content

Upgrade

Keeping MooseFS up to date ensures improved performance, enhanced security, and access to new features. This chapter provides general guidelines for upgrading MooseFS, covering best practices, potential challenges, and steps to minimize downtime.

Whether performing a minor update or a major version upgrade, careful planning is crucial to maintaining data integrity and system stability. Follow the recommendations in this chapter to ensure a smooth and successful upgrade process.

Packages' names​

For the sake of brevity, we are using package names corresponding to the Pro version of MooseFS, whenever the upgrade instructions for Pro and CE versions are identical, which is most of the time. If you are using MooseFS CE, simply remove -pro from packages names when following instructions.

Before any kind of upgrade​

To safely and securely perform an upgrade, we recommend that you first prepare your MooseFS instance:

Read the NEWS file​

Read the NEWS file attached to MooseFS distribution and check if there are any new features that will require some extra action (e.g. new configuration options that you want to change from their default settings)

Check your chunk matrix​

Check your chunk matrix via GUI or CLI and make sure there are no endangered or undergoal chunks. If any exists, wait for the system to replicate them. If there are hardware issues (like faulty hard drives) that are causing your system to have undergoal chunks, try to resolve these issues before you proceed to upgrade your instance

Plan the timing​

Plan the timing of your upgrade to a time when you can either disconnect all your client processes and perform the upgrade on the system without I/O or when the I/O is at its lowest (e.g. during the night). While MooseFS can be successfully and safely upgraded during normal operating time, module’s restarts will cause waiting times for I/O. The client (mount) process itself will handle the delays, but external processes using files stored on MooseFS may show timeouts or errors, depending on their architecture. This step is just a precaution, if you are sure that all processes using files stored on MooseFS have reasonably long timeouts for I/O waits, you don’t need to adjust your upgrade time so strictly.

Save metadata​

Just before you start your upgrade, if you already have MooseFS 4, use the mfssupervisor tool to force Master(s) metadata save to disk. if you are using the Pro version, use the ”all servers with CRC check” mode and verify via GUI or CLI that the CRC of metadata matches on all Masters.

Make a backup of your Master's data before any Master upgrade​

When, during any of the upgrade procedures, you will be instructed to ”upgrade your Master module”, just before you start performing that step, make a backup of your Master’s metadata (for the Pro version it’s enough to do this just before the first Follower Master Server upgrade).

How to find metadata and changelog files?​

To find your metadata files and changelog files, do the following:

  1. Check in your Master’s config file what is the location of the Master’s working directory, defined by the variable DATA_PATH.
  2. Go to that directory and locate a file named metadata.mfs.back and all files named changelog.*.mfs. These files are needed to restore full metadata information.
  3. Copy them to a secure location for the duration of the upgrade.
warning

If you are stopping and starting, rather then restarting, your Master modules during the upgrade, then, after stopping the Master process, in the same location you will find a file named metadata.mfs - make a backup of this file instead of the other ones.

In practice, after gracefully stopping a Master or Metalogger module, back up the following files from its working directory: metadata.mfs, changelog.0.mfs, stats.mfs and metadata.crc.

Upgrading from a previous major version of MooseFS to MooseFS 4​

MooseFS is designed to be backward compatible as much as possible. However, it’s impossible to assure that there is 100% compatibility between all possible combinations of versions.

Prior to upgrading to MooseFS 4, ensure that your current version is at least MooseFS 2.1.x. If you are using an older version, first upgrade to the latest MooseFS 2.1.x release, then proceed to MooseFS 3.x, and finally to MooseFS 4.x.

It is possible to switch (both ways) between PRO and CE version during upgrade. Note following guidelines when performing such upgrade:

Upgrade CE → CE or Pro → Pro​

Upgrade incrementally through major versions. For example, upgrade from 2.1.7 to 2.1.12, then to 3.0.118, and finally to the latest MooseFS 4 version.

Upgrade CE → Pro​

  1. Upgrade to the Latest CE Version First

    Whenever possible, upgrade your instance to the latest available CE version before switching to Pro. If upgrading to the latest CE version is not feasible, contact MooseFS Pro Support for personalized guidance.

  2. Switching and Upgrading Simultaneously

    A version upgrade combined with a switch from CE to Pro is possible but not recommended. This approach introduces additional complexity and risks. If this is your only option, always consult MooseFS Pro Support for tailored guidance and instructions, as the procedure can vary based on your current version and specific setup.

  3. Special Upgrade Order for CE → Pro

    Unlike other upgrade scenarios, the package upgrade order for CE to Pro differs, because the Chunkservers need to be reinstalled before the Master:

    • First, replace the moosefs-gui, moosefs-cli and moosefs-netdump (if you are using it) packages with moosefs-pro-gui, moosefs-pro-cli and moosefs-pro-netdump respectively.
    • Next, stop your metalogger service and replace the moosefs-metalogger package with moosefs-pro-metalogger. No configuration changes are needed; start the metalogger service again.
    • One by one, stop each Chunkserver, uninstall moosefs-chunkserver and replace it with moosefs-pro-chunkserver, then start the Chunkserver again.
    • Finally, replace the CE Master with the Pro Master:
      1. Obtain a MooseFS Pro license and copy the license file to the machine hosting the Master.
      2. Edit mfsmaster.cfg and add the LICENCE_FILENAME option, pointing to the location of the license file.
      3. Stop the Master process, uninstall moosefs-master, replace it with moosefs-pro-master, and start the Master again.
    • Finish by replacing the moosefs-client package with moosefs-pro-client and remounting; this can be done last, since Pro and CE client packages are compatible with each other.

    At this point your instance is a MooseFS Pro instance running with a single Master in the Leader role.

  4. Adding Additional Master Followers

    To take advantage of the Pro version's multi-master capability, add one or more Follower Master servers:

    • Update the DNS entry for your mfsmaster domain (or whichever name you use) so that it resolves to the IP addresses of all machines that will host Master servers, in addition to the existing Leader. You can add all the Follower IPs at once, or add them one at a time before installing each corresponding Follower.
    • Wait for the DNS change to propagate to the GUI, the current Leader, the metalogger, and all Chunkservers. In the GUI, the Metadata Servers (Masters) table on the INFO tab should show the additional entries as DEAD until each one is installed — this is expected. Confirm the domain resolves to all expected IPs from every server before proceeding.
    • For each additional Master:
      1. Install the moosefs-pro-master package on the machine.
      2. Copy the license file and set the LICENCE_FILENAME and MASTER_HOST options in mfsmaster.cfg.
      3. Start the Master process with the -e switch, which makes it download metadata from the current Leader. The GUI shows the server's state as BUSY while downloading; wait until it changes to FOLLOWER before installing the next Master.
    note

    Review the Master Servers configuration parameters and pay particular attention to METADATA_DOWNLOAD_FREQ, METADATA_CRCCHECK_FREQ and METADATA_DOWNLOAD_LIMIT, which govern how Leader and Follower Master servers save metadata to disk.

Switching Pro → CE​

Switching is only possible between identical versions (e.g., 4.57.1 Pro to 4.57.1 CE). Follow this order:

  1. Shut down all surplus Master servers, leaving only one running (this will become the sole CE Master).
  2. Update the DNS entry for your mfsmaster domain so it resolves only to the IP of the remaining machine, instead of the multiple IPs used in the Pro setup. Wait for the change to propagate.
  3. Stop the remaining Master process, uninstall moosefs-pro-master, replace it with moosefs-master, and start it again. Wait for all Chunkservers to reconnect and finish the chunk registration process.
  4. One by one, stop each Chunkserver, uninstall moosefs-pro-chunkserver, replace it with moosefs-chunkserver, and start it again. Wait for each Chunkserver to finish registering with the Master before moving to the next one.
  5. Replace the moosefs-pro-gui, moosefs-pro-cli and moosefs-pro-netdump (if you are using it) packages with moosefs-gui, moosefs-cli and moosefs-netdump respectively.
  6. Replace moosefs-pro-client with moosefs-client and remount; this can be done last, as Pro and CE client packages are compatible with each other.
warning

Do not delay the client package switch indefinitely: it must be completed before your next version upgrade, because a Pro client cannot be upgraded directly to a newer CE client.

Storage classes are converted automatically to fit within CE version capabilities, and any Pro-only settings left in your configuration files are simply ignored by CE processes.

Upgrading between minor versions of MooseFS 4​

MooseFS 4 version numbers consist of 3 parts: major number (4), minor number, and patch level (e.g. MooseFS version 4.55.5 has the major number of 4, minor number of 55, and patch level of 5). When performing an upgrade, you need to follow these rules:

  1. Modules with the same minor version and different patch levels can all cooperate freely.
  2. Modules with different minor version may not have compatible protocols (i.e. different communication packets or different metadata format) so a MooseFS instance using different minor versions needs to always follow these rules:
    • The metalogger module must have the same or higher version as the Master module
    • The Master module must have the same or higher version as chunk server modules. In Pro version Follower Master server must have the same or higher version as Leader Master server.
    • The Master and chunk server modules must have the same or higher version as the client (mount) modules.
  3. It is strongly recommended that the GUI module has the same or higher version as the Master server(s); GUI in a lower version will work with the Master, but may not show all information needed.

In the light of the above rules, the recommended order of upgrade is as follows (refer to Installing MooseFS 4 for instructions on how to install the appropriate modules on your particular OS):

  1. Upgrade all monitoring tools (GUI, CLI, netdump, Supervisor)
  2. Upgrade all Master Followers (Pro only)
  3. Upgrade all Metaloggers (if any)
  4. Upgrade the Master (Leader Master Server in Pro)
  5. Upgrade the Chunkservers, one by one
  6. Upgrade the Clients and remount them
note

Whenever a step below instructs you to stop a service before upgrading its package, first compare the newly delivered *.cfg.sample file with your running *.cfg file (e.g. with vim -d) and merge any new options into your running configuration. Do not remove any files from the module's working directory (DATA_PATH, default /var/lib/mfs) — this is where metadata and changelog backups described in Before any kind of upgrade are taken from.

1. Upgrade GUI, CLI and netdump​

First, upgrade the moosefs-pro-gui, moosefs-pro-cli and moosefs-pro-netdump (if you are using it) packages, then restart the necessary services for the new GUI to be loaded. Confirm the GUI is running the new version — it is shown in the top-right corner of the webpage. These tools are usually installed on all Master Servers.

2. Upgrade metalogger​

Next, upgrade your moosefs-pro-metalogger package (if you are using it; the use of this package is recommended for both the CE and the Pro version) and restart your metalogger process.

3. Upgrade Masters​

Depending on the version you are using:

  • For the CE version, upgrade your Master moosefs-master package and restart your Master process. Wait for all the Chunkservers to connect and finish the chunk registration process; this information is displayed on the INFO tab of the GUI and is logged to system logs.

  • For the Pro version:

    1. Upgrade all your Follower Master modules, one by one: stop the Follower service gracefully, upgrade the moosefs-pro-master package, and restart the Master process. After restarting each module, wait for it to connect to the Leader Master server and become a functioning Follower again — the state shown on the INFO tab of the GUI should be FOLLOWER, not FOLLOWER (DESYNC), FOLLOWER (DELAYED), or anything else.
    2. Once every Follower is upgraded and synced, upgrade the Leader Master module. Stop the Leader Master Server service gracefully — do not use mfssupervisor to force a role switch beforehand, one of the Followers will automatically take over the Leader role as soon as the current Leader stops, as long as the cluster is configured correctly. Upgrade the package on the now-stopped ex-leader and restart it; it will rejoin the cluster as a Follower with the new version. Wait for it to become a synced Follower.
    3. If you want this particular machine to host the Leader process again, use the mfssupervisor tool to switch the Leader to this machine, and wait for the switch to complete before proceeding to the next step.

4. Upgrade Chunkservers​

Upgrade your Chunkserver modules one by one:

  1. Stop the Chunkserver service gracefully. It should appear as disconnected in the GUI's "Servers" tab, in Automatic Temporary Maintenance Mode (ATMM).
  2. Upgrade the moosefs-pro-chunkserver package and restart the Chunkserver process.
  3. Wait for the Chunkserver to connect to the Leader Master server and register all of its chunks — the "Servers" tab should show ATMM cleared automatically, and this is also logged to system logs.
  4. Observe the INFO tab's "All Chunks state matrix" and wait for all chunks to reach a stable state. If any undergoal or endangered chunks remain after the Chunkserver rejoins the cluster, wait for them to finish replicating before proceeding to the next Chunkserver.

5. Upgrade clients​

For each Client:

  1. Stop all processes and CWDs (current working directories) using the mountpoint.
  2. Unmount the mountpoint, upgrade the moosefs-pro-client package, then mount it again.
  3. Restart the processes that use the mountpoint.

This can be done in parallel across Clients.