One of the main bottlenecks when running high performance virtualization systems is the harddisk. Now, you could of course switch your entire system over to SSDs, but that is costly and you’ll end up with a lot less storage, or a massive RAID array.

Modern filesystems like ZFS have solved this problem by allowing for ‘hybrid’ systems. These use the traditional harddisks for persistent storage, and use SSD drives in front of them to cache the read and write queries. This way you get the best of both worlds. Nearly SSD performance and the storage size of a traditional drive.

At Flosoft.biz we use Proxmox to power our VPS offers, which uses LVM and EXT4 for it’s filesystem which doesn’t have a ‘SSD caching’ method built into it. Facebook seems to have had a similar issue, so they created FlashCache. Flashcache is a kernel module that allows you to add a block caching partition in front of any other partition of your system, resulting in an amazing speedup of your system.

After having spent a night or two on getting this to work on Proxmox 2, I decided to write a small tutorial here. I’d also like to thank @toxicnaan for his l33t hax0r skillz.

Updating your system

Get your system up to date and make sure you’ve got the latest Kernel.

apt-get update
apt-get dist-upgrade
apt-get install dkms build-essential git
reboot

Kernel Headers

You will now need to install the Kernel Headers for your Kernel so that you can compile the module. Make sure you install the correct version of the headers. These need to be the same as the Kernel you’re running.

uname -a # to get your kernel version
apt-get install pve-headers-2.6.32-17-pve # to install the headers for version 2.6.32-17

Get FlashCache

Now that we’ve got the Kernel tools, we can get FlashCache and build it.

git clone git://github.com/facebook/flashcache.git
cd flashcache/

make -f Makefile.dkms boot_conf
make install

Load FlashCache

Next we need to load FlashCache into our running Kernel and make sure it’s loaded upon boot.

modprobe flashcache
echo flashcache >> /etc/modules

Re-purposing the SSD drives

Now it’s time to find a new use for our SSD drives, namely as cache. You can skip this step if your server doesn’t have the SSD drives mounted as /var/lib/vz

umount /var/lib/vz
vgremove pve
pvremove /dev/md2

Re-purposing the 2 HDD drives

Now let’s prepare the 2 HDD drives to be used as the storage for /var/lib/vz.

umount /data
pvcreate /dev/md0
lvcreate -l 100%VG -n storage pve
mkfs.ext4 /dev/mapper/pve-storage

Creating the FlashCache partition

Now let’s create the FlashCache partition on the SSD drives & mount it.

flashcache_create -p back pvec-storage /dev/md2 /dev/mapper/pve-storage
mount /dev/mapper/pvec-storage /var/lib/vz
echo 1 > /proc/sys/dev/flashcache/md2+pve-storage/fast_remove

Editing /etc/fstab

Next step is to edit /etc/fstab and remove the /data and /var/lib/vz mounts. If you forget to do this (as I did for quite a while), your server will struggle to boot on it’s own, and you’ll end up with the datacenter techs thinking you’re an idiot 🙂

vi /etc/fstab

The init.d file

This next step is important. We need to add an init.d file to do some operations, like mounting the filesystem and cleaning it up. It will also unmount the drive before shutting down, as if you don’t do this, your kernel will freeze on shutdown. Make sure you edit your file according to your needs.

#!/bin/sh

# Start or stop Flashcache

### BEGIN INIT INFO
# Provides:          flashcache
# Required-Start:
# Required-Stop:     $remote_fs $network pvedaemon
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Flashcache SSD caching
# Description:       Flashcache SSD caching
### END INIT INFO

PATH=/bin:/usr/bin:/sbin:/usr/sbin

flashcache_start() {
if df -h | grep /var/lib/vz > /dev/null
then
echo "Flashcache allready running"
else
flashcache_load /dev/md2
mount /dev/mapper/pvec-storage /var/lib/vz
#mount /dev/mapper/pve-backup /mnt/backup
echo 1 > /proc/sys/dev/flashcache/md2+pve-storage/fast_remove
echo "Flashcache started"
fi
}

flashcache_stop() {
if df -h | grep /var/lib/vz > /dev/null
then
#umount /mnt/backup
umount /var/lib/vz
dmsetup remove pvec-storage
echo "Flashcache stopped"
else
echo "Flashcache not running"
fi
}

case "$1" in
    start)
flashcache_start
    ;;

    stop)
flashcache_stop
    ;;

    restart)
        $0 stop
        $0 start
    ;;
esac

exit 0

Enabling the init.d file.

Now we need to make the file executable and make sure it’s run on boot.

chmod +x /etc/init.d/flashcache
update-rc.d flashcache defaults

Give it a spin

Right, that should do it. Reboot your machine and see if it comes back.

reboot

If all went well, your drive should be mounted with FlashCache in between.

root@vh43:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
none                   32G  256K   32G   1% /dev
/dev/md1               10G  1.3G  8.2G  14% /
tmpfs                  32G     0   32G   0% /lib/init/rw
tmpfs                  32G     0   32G   0% /dev/shm
/dev/fuse              30M   12K   30M   1% /etc/pve
/dev/mapper/pvec-storage
                      1.8T  196M  1.7T   1% /var/lib/vz

You can also see the statistics of FlashCache by running:

cat /proc/flashcache/md2+pve-storage/flashcache_stats

That’s it! Your Proxmox system should now have it’s VMs on the FlashCache drive.

If you have any questions or feedback, just leave them below.

References: