Linux 3.14 and PVH

The Linux v3.14 will sport a new mode in which the Linux kernel can run thanks to Mukesh Rathor (Oracle).
Called ‘ParaVirtualized Hardware,’ it allows the guest to utilize many hardware features – while at the same time having no emulated devices. It is the next step in PV evolution, and it is pretty fantastic.
Here is a great blog that explains the background and history in detail at:
The Paravirtualization Spectrum, Part 2: From poles to a spectrum.
The short description is that Xen guests can run as HVM or PV. PV is a mode where the kernel lets the hypervisor program page-tables, segments, etc. With EPT/NPT capabilities in current processors, the overhead of doing this in an HVM (Hardware Virtual Machine) container is much lower than the hypervisor doing it for us. In short, we let a PV guest run without doing page-table, segment, syscall, etc updates through the hypervisor – instead it is all done within the guest container.
It is a hybrid PV – hence the ‘PVH’ name – a PV guest within an HVM container.

The benefits of this – less code to maintainer, faster performance for syscall (no context switches into the hypervisor), less traps on various operations etc. – in short better and faster response.
The code going in will allow users to use it with Xen 4.4 (which as of Jan 2014 is in RC3). From a standpoint of what the guest does compared to the normal PV – it is almost
no different except that it reports itself as an HVM guest (without any PCI devices) and is much faster. Naturally, since we are still working through the kinks of the PVH ABI, there is nothing set in stone.
That means the next version of Xen might not even run with this version of Linux PVH (or vice-versa). It is very experimental and unstable. Naturally we want to make this production quality, and we are working furiously toward that goal.
It helps us immensely if users also try it out so we can track bugs and issues we have not even thought of.
HOW TO USE IT
The only things needed to make this work as PVH are:

• Get the latest version of Xen and compile/install it. See http://wiki.xen.org/wiki/Compiling_Xen_From_Source for details or http://wiki.xen.org/wiki/Xen_4.4_RC3_test_instructions
• Get the latest version of Linux , see < a href="http://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs#Configuring_the_Kernel">http://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs#Configuring_the_Kernel for details. The steps are:
  
cd $HOME
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux

• Compile with CONFIG_XEN_PVH=y
  • Based on your current distro.
cp /boot/config-`uname -r `$HOME/linux/.config
make menuconfig
Where one will select Processor type and features —> Linux guest support —>Support for running as a PVH guest (NEW)
  • Or from scratch:
    make localmodconfig

    You should see:
    “ Support for running as a PVH guest (XEN_PVH) [N/y] (NEW)“In case you missed it:
    
make menuconfig
Processor type and features ---; Linux guest support ---;
Xen guest support (which will now show you:)
Support for running as a PVH guest (NEW)

  • If you prefer to edit .config, these should be enabled:
    
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_XEN=y
CONFIG_XEN_PVH=y

    You will also have to enable the block, network drivers, console, etc which are in different submenus.
  • Install it. Usually doing:
    
make modules_install && install

    is suffice. It should generate the initramfs.cpio.gz and the kernel and stash them in /boot directory.
• Launch it with ‘pvh=1‘ in your guest config (for example):
extra="console=hvc0 debug kgdboc=hvc0 nokgdbroundup initcall_debug debug"
kernel="/boot/vmlinuz-3.13+"
ramdisk="/boot/initramfs-3.13+.cpio.gz"
memory=1024
vcpus=4
name="pvh"
vif = [ 'mac=00:0F:4B:00:00:68' ] vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1'] disk=['phy:/dev/sdb1,xvda,w'] pvh=1
on_reboot="preserve"
on_crash="preserve"
on_poweroff="preserve

  using ‘xl’. Xend ‘xm’ does not have PVH support.

It will bootup as a normal PV guest, but ‘xen-detect’ will report it as an HVM guest.
Items that have not been tested extensively or at all:

• Migration (xl save && xl restore for example).
• 32-bit guests (won’t even present you with a CONFIG_XEN_PVH option)
• PCI passthrough
• Running it in dom0 mode (as the patches for that are not yet in Xen upstream). If you want to try that, you can merge/pull Mukesh’s branch:
  
cd $HOME/xen
git pull git://oss.oracle.com/git/mrathor/xen.git dom0pvh-v7

  And use on the Xen command line the dom0pvh=1 bootup parameter. Remember to recompile and install the new version of Xen. The patches in Linux do not contain the necessary code to setup guests.
• Memory ballooning
• Multiple VBDs, NICs, etc.

Things that are broken:

• CPUID filtering. There are no filtering done at all which means that certain cpuid flags are exposed to the guest. The x2apic will cause a crash if the NMI handler is invoked. The APERF will cause inferior scheduling decisions.
• Does not work with AMD hardware.
• Does not work with 32-bit guests.

If you encounter errors, please email with the following (please note that the guest config has “on_reboot=”preserve”, on_crash=”preserve” – which you should have in your guest config to contain the memory of the guest):

• xl dmesg
• xl list
• xenctx -s $HOME/linux/System.map -f -a -C <domain id>
  [xenctx is sometimes found in /usr/lib/xen/bin/xenctx ]
• The console output from the guest
• Anything else you can think off.

to xen-devel@lists.xenproject.org
Stash away your vmlinux file (it is too big to send via email) – as we might need it later on.
That is it!
Thank you!