Dom0 Memory -- Where It Has Not Gone

If you are upgrading domain 0 Linux kernel from a non-pvops (classic, 2.6.18/2.6.32/etc.) kernel to a pvops one (3.0 or later), you may find that the amount of free memory inside dom0 has decreased significantly.  This is because of changes in the way kernel handles the memory given to it by Xen.  With some updates and configuration changes, the “lost” memory can be recovered.

tl;dr: If you previously had ‘dom0_mem=2G’ as a command line option to Xen, change this to ‘dom0_mem=2G,max:2G’.  If that didn’t help, read on.

What Changed?

When a domain is started, it is provided with:

1. A number of pages of memory. These are contiguous in the guest’s pseudo-physical address space and start at address 0.
2. The maximum number of pages the domain is allowed to have (the maximum reservation).  This may be more than the initial number of pages to allow the the guest to balloon up (increase its number of pages).
3. The e820 memory map.  For dom0, the physical e820 map is used so dom0 can access all BIOS data areas and PCI device MMIO regions.

The principle differences between the new and old kernels are:

1. The new kernel allocates page tables and struct page’s for all the pages up to the maximum reservation, the older kernels only allocate enough for the current number of pages.  On systems with lots of RAM relative to the number of pages used by dom0 this can result in lots of memory wasted.
2. The new kernel tries harder to release pages back to Xen that are unusable (because they are behind reserved regions or holes in the memory map).

Give Me Back My Memory!
To stop the kernel from wasting memory on page tables etc. for memory it will never use:

1. Use Xen dom0_mem=max:LLL command line option to set the maximum reservation to LLL (the M and G suffixes can be used for for MiB and GiB) .  e.g., If you previously had dom0_mem=XXXG, change this to dom0_mem=XXXG,max:XXXG.

This requires Xen 4.1.2 or later (which sets the maximum reservation based on the command line) and Linux 3.0.5 or later (which makes use of it)..
To recover the memory released during boot:

1. Ensure the balloon driver is enabled in the kernel (CONFIG_XEN_BALLOON).
2. Set the balloon driver’s target to the desired amount of RAM. Either with the mechanism provided by your toolstack or directly with (XXX is the amount of RAM in KiB):

   echo XXX > /sys/devices/system/xen_memory/xen_memory0/target_kb

I’m Lazy, Make it So
A series of patches by Konrad Rzeszutek Wilk will eliminate the need for the balloon driver.  With these patches, the otherwise unusable pages will be moved instead of released.  These patches are expected in 3.5.