Skip to main content

Master Server RAM shortage

MooseFS master server RAM usage is proportional to the number of objects in the filesystem (files + directories + special files), but can be increased by certain factors:

  • very long paths (long directory names, long file names)
  • a lot of extended attributes stored in filesystem objects

Moreover, the master process reserves RAM in blocks, and although it reuses empty blocks, it does not free them very often, if unused. So once memory is allocated for the Master, it mostly stays with the master.

If you want to force the master to free all of unused memory, you have to restart the process. In the Pro version this can be done without disturbing the system's work: you first restart the follower(s), then the leader (one of the followers will take over as leader in this case).

If there is not enough physical RAM, then one of 3 things might happen:

  • the Master will start using swap, which strongly impacts its efficiency
  • the out of memory killer will kill the Master process (Master tries to disable OOM killer on startup, but it might not succeed)
  • the Master will abort when it cannot allocate more RAM

In any case, it's important to know that the RAM usage must either be decreased, or the machine that hosts the Master must be upgraded with more RAM, for your Master to work efficiently.

If the Master aborts or is killed by OOM killer, it will not save it's metadata to disk. In this case you have to start it again with -a option, to instruct it to re-build metadata, using the previous (backup) metadata file and changelog files. You have to make sure that the RAM shortage will not repeat itself as soon as you start the process, so either add more RAM or enable swap (or add more swap space).

If you enabled swap, you need to decrease your Master's RAM usage to a level when it will no longer need swap, otherwise the performance of your Master will suffer. To do so, you can:

  • remove old snapshots (use mfsrmsnapshot for efficient removal)
  • delete unnecessary files
  • remove files from trash, if you have any (trash files can be very "heavy" if their original path in the filesystem was very long)
  • remove unnecessary extended attributes

It's strongly recommended not to connect any clients (not to restore normal I/O to the cluster) if the Master uses swap and only connect one client (local if possible) to perform the operations mentioned above.

After performing any (or all) of the operations listed above, restart your Master server(s).

If the RAM usage is still too high, you have to increase physical RAM in the machines or move Master(s) to machines with more RAM.

It's always strongly recommended to monitor the Master RAM usage and have a plan of action in case it goes over 80%-90% of available physical RAM.