Virtualization

Virtualization is widely used in modern computing and while most virtualization solutions try to be as compact as possible, there is still a great need for storage space for virtual machines' data. MooseFS can be the solution here, when configured properly.

When planning for virtualization, storage is critical. The amount of storage needed depends on the type and number of virtual machines (VMs), the operating systems and applications they run, as well as the chosen virtualization methods. MooseFS offers unique solutions that can help in saving extra space in some areas, while keeping high performance in other areas.

Types of data stored and their i/o requirements:

Virtual Machine Disk Files

These files (like .vmdk for VMware, .vhdx for Hyper-V, .qcow2 for KVM) store the virtualized OS, applications, and user data. Each VM typically requires 20–100+ GB depending on its role. These files will see a lot of i/o, but changes (write operations) will be limited to certain areas of the file (system logs, tmp directories, working directories of proceeses run on these VMs), while other areas will be mostly read only.

Snapshots and Clones

Snapshots preserve the state of a VM at a given time. They consume additional storage, often rapidly if changes to the VM are frequent, but are stored for read only purposes (to use when a rollback is needed).

ISO Files and Templates

These are used for OS installations and VM provisioning, requiring additional space, typically ranging from 1–10 GB per ISO/template. These files are also stored for read only purposes.

Logs and Backups

Logs are needed for monitoring and troubleshooting, while VM backups can significantly increase storage needs depending on retention policies. These files are stored for read only purposes.

Swap or Paging Files

Depending on the host configuration, swap files for overcommitted memory can use additional disk space. These files, when needed, will observe the most varied i/o usage (so both reads and writes).

So how to configure your MooseFS instance to store your virtualization environment?

1. Space is crucial, so using the Erasure Codes (EC) format to store some of the data can improve your overall efficiency. But for EC to work properly, your MooseFS instance will need more rather than less chunkserver machines - plan accordingly.
2. Make good use of Storage Classes - we will show you some examples.
3. You can use both MooseFS Client (aka the mount process) to access your files and MooseFS block device (mfsbdev) specifically for VMs disk files.
4. If your chosen virtualization solution does not have snapshots or you are not happy with how they are implemented, you can check out the snapshot mechanism provided by MooseFS and see if this will work better for you.

Storage Classes examples for virtualization environments:

All examples are asssuming Redundancy Level (RL) 2, which means the system can survive two independent concurrent hardware failures (a machine failure or a harddrive failure in one machine) of data storage servers (chunkservers) and all the data will still be accessible. In some cases (like log files) you may decide to lower that level, while in other (some crucial backups) you may decide to up that level. The general purpose of the proposed classes will stay the same.

1. Virtual Machine Disk Files

For keeping images of working virtual machines, you can pick one of two options:

• if you have a lot of space, you can keep them in copy format, which means fast read and write access to all areas of the file, but no space saving
• if you want to save some space, you can convert part of your image file to EC format; this format saves a lot of space without lowering your data safety (redundancy) level and provides as fast read access as the copy format, but slightly slower initial write access

To keep all your virtual machines images in copy format, simply define this class:

mfsscadmin create -K 3* my_machine_images

This command creates a storage class named my\_machine\_images, that stores all chunks of file's data in 3 (three) copies.

To use EC format, you can use one of two recommended options:

mfsscadmin create -K 3* -A@8+2 -o p -d 1h my_machine_images
mfsscadmin create -K 3* -A@4+2 -o p -d 1h my_machine_images

These commands create a storage class named my\_machine\_images, that initially stores all chunks of file's data in 3 (three) copies, but each chunk that has not been modified for the specified amount of time - in our example 1 hour - will be converted to EC format, with 8 (first example) or 4 (second example) data parts and 2 parity parts. That means that chunks (parts of your file of size 64MiB) that are not modified very often, or not at all, will be stored with the same redundancy (safety), but will occupy only 125% (first example) or 150% (second example) of storage space, compared to the chunks kept in copy format, which occupy 300% of storage space.

You can adjust the time needed for format convertion to EC to suit your needs, 1 hour is the minimum possible, but it can be extended, resulting in less convertions if your files have areas that are modified not so frequently, but still on a regular basis.

Notice, that the specified delay time is the minimum time after which the convertion can occur. The chunks will be considered able to be converted after this time has passed, but the actual convertion can be further delayed, depending on availability of resources in your MooseFS instance (chunk loop speed and replication limits).

2. Snapshots, clones, iso files, templates and backups.

These files, once created, are almost never modified and only accessed for read purposes when necessary. Therefore you can afford to instruct MooseFS to conver them to EC format as soon as possible. Example storage class:

mfsscadmin create -K 3* -A@8+2 -o f my_backup_files
mfsscadmin create -K 3* -A@4+2 -o f my_backup_files

These commands create a storage class named my_backup_files, that initially stores all chunks of file's data in 3 (three) copies, but each chunk is converted to EC storage format as soon as possible, with 8 (first example) or 4 (second example) data parts and 2 parity parts.

3. Logs

Logs are text files, that grow over time. While the initial content is never modified, content is added at the end frequently, so one has to be careful with convertion of chunks to EC format. Since log files grow in a linear way and always at the end, when a full chunk (64MiB) is written, it will never be modified again. So it makes sense to use the "per chunk modification date" policy for archiving. Also, sometimes log files are really small and there is no point in even attempting to convert them, as very small file gain nothing in terms of saving space (and sometimes even use more space!) when converted to EC format. So the example storage classes in this case would be:

mfsscadmin create -K 3* -A@8+2 -o p -d 6h -s 2Mi my_log_files
mfsscadmin create -K 3* -A@4+2 -o p -d 6h -s 2Mi my_log_files

These commands create a storage class named my_machine_images, that initially stores all chunks of file's data in 3 (three) copies, but each chunk that has not been modified for the specified amount of time - in our example 6 hours - will be converted to EC format, with 8 (first example) or 4 (second example) data parts and 2 parity parts. Files shorter than 2MiB will never be converted to EC storage format.

4. Swap or Paging files

These files, when used, need full performance, but are very temporary in their nature (their content becomes irrelevant when a virtual machine is stopped or restarted). They do not require backups nor transfers to another host machine. Therefore, we recommend:

• not using swap when it's not necessary (your virtual machine's tasks will require limited RAM usage)
• when using swap, to configure it, whenever possible, as a separate file and to use local, fast access disk (possibly entire device or a partition with direct access, not via a file) to store them

Using specific hardware for different types of files

Sometimes you may want to use more sophisticated hardware (e.g. SSD drives) for files that need fast access (like active VM disk images), while storing logs, backups and additional files on slower drives. MooseFS will help you, with the use of labels.

If you have a number of machines with SSD drives and a number of machines with SSD drives, simply label them (set the LABELS variable in mfschunkserver.cfg file) and define your storage classes to keep specific files on specific chunkservers (with specific drive type). Let's say you marked all your chunkservers with SSD or NVMe drives with F label (like fast) and all your chunkservers with mechanical drives with label S (like slow). Then the example storage classes will look like this:

1. Virtual Machine Disk Files

   mfsscadmin create -K 3F my_machine_images
   mfsscadmin create -K 3F -A@8+2,F -o p -d 1h my_machine_images
   mfsscadmin create -K 3F -A@4+2,F -o p -d 1h my_machine_images

2. Snapshots, clones, iso files, templates and backups.

   mfsscadmin create -K 3S -A@8+2,S -o f my_backup_files
   mfsscadmin create -K 3S -A@4+2,S -o f my_backup_files

3. Logs

   mfsscadmin create -K 3S -A@8+2,S -o p -d 6h -s 2Mi my_log_files
   mfsscadmin create -K 3S -A@4+2.S -o p -d 6h -s 2Mi my_log_files

When at first there was a * sign, that told MooseFS it can keep the chunks on whichever chunkserver it chose, now we are instructing MooseFS to keep files belonging to specific storage classes on specific chunkservers, labeled with either F or S label.

Hardware requirements and tips

To use the EC format, you need a specific number of chunk server machines. In our examples, we used EC 8+3 and EC 4+3, which require the minimum of 14 or 11 chunkservers, respectively, and we don't recommend to use the minimum. Always have at least one more chunkserver, try to have two more, for efficiency reasons, especially in case one machine fails.

If you want to use labels as in the example above, you need to have the required minimum number of chunkservers with each label. That would normally double your number of chunkserver machines.

You can, however, have both "slow" (mechanical) and "fast" (SSD or NVMe) drives in one physical machine and use that physical machine to run two chunkserver processes. You assign all your fast drives to one process (and label it with a specific label, e.g. F), and all your "slow" drives to another process (and label it it with a specific label, e.g. S). And you use the classes as described above, but with one important modification: you have to setSTRICTT mode for each class, so that MooseFS never records a copy or EC part of a chunk to the same physical machine. You can set the STRICT mode by adding -m s parameter to each storage class definition.

If you setup two chunkserver processes on one physical machine, remember, that each has to have its own local working directory (DATA_PATH), list of drives (paths) for chunks storage (HDD_CONF_FILENAME) and port to listen for clients (mounts) connections (CSSERV_LISTEN_PORT).