It’s a question many will ask at some point. You’ve got Xen set up, used a graphical tool to configure some domUs (or downloaded some pre-built images, or followed a howto). But now you want to know where your virtual machines are actually stored. It’s a good question – and it has a slightly complicated answer.
There are actually several aspects to this question – various parts of a domU’s state are stored in different locations. Strictly, a “domain” originally referred to a running VM, where a VM represented a persistant virtual machine with a name, hard drive state, etc (this is a little similar to the UNIX “a process is a running program” distinction). However, “domain” is a bit of an overloaded term these days and we often refer to “my VM” as “my domU”. This distinction is sometimes relevant but I’ll generally use the terms interchangeably.
A virtual machine / domU may have the following bits of state stored somewhere:
- the memory image and CPU state
- the configuration file
- the virtual disk state
In most people’s Xen setups, these elements are stored in different locations on dom0’s disk. If you want to fully back up a domU, you really need to back up all these components.
Memory image and CPU state
The domain’s memory image and CPU state may not be present (if the domain isn’t running or suspended), or may be currently in memory (running domain) or on disk (saved domain) in a user specified file at an arbitrary location. [1] Combined with the domain’s storage and network connections at the same instant, it describes everything the domain was currently doing at a given point in time.
Configuration file
The VM configuration is normally stored in a file inside /etc/xen/, although some tools may choose other locations. By convention, the config files are often named after the domain but that’s not actually necessary. [2] If you xm save or xm migrate a domain, the current configuration is packaged into the save image so that the virtual machine can be resumed without you having to specify a configuration file. This has the disadvantage that changes to the on-disk configuration in /etc/xen/ aren’t picked up for a resumed guest. There’s another problem here that sometimes trips people up: Xend isn’t recording suspend / resume status in the configuration file or anywhere else. Xend doesn’t know if a virtual machine is currently meant to be suspended or properly shut down – it doesn’t even know where you might have chosen to save the suspend file. If a domU is not running, it’s outside Xend’s world. As a result, there’s nothing to stop you saving a guest, then booting it directly using xm create, shutting it down, then resuming the earlier saved image at a later date. This is quite likely to hose the domain’s filesystem because the resumed guest won’t know the on-disk data has moved around!
Virtual disks
The domain’s virtual disk state may be stored in a number of ways. Often it’s just a file on a hard drive somewhere. It may be an LVM logical volume, a partition, a whole disk drive, or a network block device. [3] The virtual disks are listed in the configuration file and may also be hotplugged at runtime. Some distros require the disks to be in a special path (e.g. on CentOS / RHEL, due to SELinux).
Gotchas
This default setup of explicit, separate configuration files has a few drawbacks: one obvious drawback is that the config file in /etc/xen/ is never actually updated by Xen(d). As a result, if you xm block-attach a disk to a domain whilst it’s running, that disk will be gone if you then destroy and restart the domain. If you want to hotplug and permanently add a disk to a domain you have to issue xm block-attach as well as modifying the configuration file. There’s also the problem mentioned above with xm save / xm create conflicts – and probably a few other gotchas I’ve not mentioned.
Conclusions
The state of a Xen virtual machine is spread across dom0’s storage. The key parts of this state are the memory and CPU state, the configuration file and the virtual disks. In order to correctly backup and restore a domU, you should take a copy of the virtual disks and the configuration file, and also the memory and CPU state file if the domain has been saved. You should never resume a saved domain image if anything has modified it’s original virtual disk state, including subsequent boots of the domain using xm create, or mounting the virtual disk in dom0. Backing up a running domain has some pitfalls of its own, which I’ll discuss in a future article – in the meantime, the simplest way to guarantee a safe backup is to cleanly shut down a domain before backing up its state.
Some of the problems I’ve described in this article are mitigated by the Xend “managed domains” feature, which provides more advanced management functionality within the normal Open Source Xend. I may be able to discuss this in a future article.
Feedback on these articles is very welcome – please leave comments or e-mail me if you spot mistakes, or if they is useful to you, or if you have any suggestions.
Footnotes
[1] It may even be in the process of being duplicated onto another machine, in the case of live migration.
[2] A single config file can start multiple different domains, if you want to get that advanced. I don’t ever do this myself but I imagine it can be useful to simplify configuration control.
[3] Xen also provides some scripts that can automatically connect to NBD servers, etc, so that it’s easy to set up remote storage.