Skip to main content

Master Server storage shortage

MooseFS master server storage (disk) usage is proportional to RAM usage and number of I/O operations performed on the cluster. The Master stores the following files:

  • metadata backups (files metadata.mfs.back.*) - big binary files, each is roughly the size of RAM occupied by master
  • changelogs (files changelog.*.mfs) - text files, depending on how busy your instance is, they may be small (300-400 KiB) to large (300-400 MiB)
  • statistics file for GUI charts - constant size, less than 10MiB
  • other files of negligible sizes: lock files, info file (if generated by info command) etc.

The size of metadata files and changelog files may change over time (especially the changelog files are prone to fluctuating). It's recommended to actively monitor the Master's storage space and to increase it whenever the utilisation goes over 80%.

The number of changelog files and backup metadata files is set by BACK_LOGS and BACK_META_KEEP_PREVIOUS variables respectively, set in mfsmaster.cfg. The first tells the master, how many changelogs it should keep, the second, how many previous versions of metadata it should keep (so the number of metadata files is actually +1, as the most current metadata file is always kept in addition to backups). The default values are, respecitvely: 50 and 1, so 50 changelogs and 2 metadata file (1 latest, 1 previous) will be kept. If you change (increase) these values, remember to ensure that there is enough storage in the data directory for all the files that will be created. Do not lower the values just to save space - the metadata backups are crucial in case of any failure!

When the Master runs out of storage space and cannot save metadata, it will enter "emergency mode", in which it will repeatedly try to save the metadata file in:

  • the configured data directory (as set by DATA_PATH variable in mfsmaster.cfg; default is /var/lib/mfs on most operating systems)
  • the home directory of the user running the master process
  • in one of the following, hard coded emergency locations:
    • "/metadata.mfs.emergency",
    • "/tmp/metadata.mfs.emergency",
    • "/var/metadata.mfs.emergency",
    • "/usr/metadata.mfs.emergency",
    • "/usr/share/metadata.mfs.emergency",
    • "/usr/local/metadata.mfs.emergency",
    • "/usr/local/var/metadata.mfs.emergency",
    • "/usr/local/share/metadata.mfs.emergency"

The master process exits when it sucessfully writes the metadata file in one of the above locations, if it does not succeed, it will try again and again. After each try, an error message will be sent to syslog: can't store metadata - try to make more space on your hdd or change privileges - retrying after 10 seconds.

When such situation happens, you can add storage space to one of the above mentioned locations (free some space on the disk, add another disk to a ZFS pool or a RAID matrix, mount a new empty drive as \tmp etc.). As soon as the master sucessfully writes the metadata in one of the locations, it will not make another attempt, but just exit. You have to then check, in which location the file was written, copy or move it to the configured data directory and change its name to metadata.mfs (if a file with that name already existed there, it will probably be corrupted, most likely too short, but you can back up it somewhere just in case).