Less is More in the New Xen Project 4.5 Release

If we used code-names, the Xen 4.5 release should be called Panda on Diet! We have 78K new code with 141K deleted. In effect this release has -63KLOC code than the previous one.

The net effect of a skinnier Xen Project Hypervisor code base is increased usability, simplicity and innovation. This is all by design and one of many steps we’ll continue to take to fine-tune our development and release cycle.

For example, we shed the Python toolstack – including xend which we deprecated in 4.3. This comprised the majority of the code deleted in today’s release, which is a big boon for developers who now have less code to maintain and can spend more time on new features.

And 4.5 is more feature-rich than any release in Xen Project’s history.

Today we are announcing specific patches in Xen Project Hypervisor 4.5 that span from architecture (x86 and ARM), platforms (different ARM, AMD or Intel boards), to generic code. The release also creates new opportunity to incorporate Xen virtualization into software stacks in markets like embedded computing, automotive, drones, avionics and more.

Virtualization and open source are more relevant than ever in today’s evolving, more software-centric data center too. New developments with hyper scale-out computing, Internet of Things, NFV/SDN, and next-generation ARM-based products are driving increased demand for better resource sharing and utilization with enough flexibility to efficiently grow well into the future. What isn’t likely to change anytime soon is the diversity of OSes, multi-tenant architectures, security concerns and storage and network challenges that cloud providers and enterprises must contend with to run their applications. Undeniably, abstraction at the VM level is necessary for superior performance and security in these environments.

Despite these impressive and rapid changes, or perhaps because of them, Xen Project developers are motivated to continually stay ahead of the market with performance, speed, agility and security improvements. Our traditional customers also inspire us; organizations such as Alibaba, Amazon Web Services, IBM Softlayer, Rackspace, Oracle and others are some of the most savvy and innovative users around.
To learn more about the release and for ease of reading, I’ve grouped the summary of updates into four major categories:

• Hypervisor specific

• Toolstack

• External users of toolstack

• Linux, FreeBSD, and other OSes that can utilize the new features.

x86 Hypervisor-Specific Updates
On the x86 side, development has focused on improving performance on various fronts:

• The HPET has been modified to provide faster and better resolution values.

• Memory is scrubbed in parallel on bootup, giving a huge time boost for large-scale machines (1TB or more).

• PVH initial domain support for Intel has been added and now supports running as dom0 and FreeBSD with Linux platforms. PVH is an extension to the classic Xen Project Paravirtualization (PV) that uses the hardware virtualization extensions available on modern x86 processor servers. Requiring no additional support other than the hypervisor, PVH boots as the first guest and takes on the responsibilities of the initial domain known as dom0. This means Xen Project Hypervisor is able to take advantage of contemporary hardware features like virtual machine extensions (VMX) to significantly expedite execution of the initial domain. Instead of asking the hypervisor to handle certain operations, the dom0 can execute operations natively without compromising security. For more background, Virtualization Spectrum is an excellent introduction to PVH.

• Lower interrupt latency for PCI passthrough on large-scale machines (more than 2 sockets).

• Multiple IO-REQ services for guests, which is a technique to have many QEMUs assigned for one domain. This allows speed up of guests operation by having multiple backends (QEMUs) deal with different emulations.

We also expanded support for:

• Soft affinity for vCPUs: Xen has had NUMA- aware scheduling (http://wiki.xen.org/wiki/Xen_on_NUMA_Machines) since 4.3. In Xen 4.5, we build on that to make it more general and useful on non-NUMA systems. In fact, it is now possible for the sysadmin to define an arbitrary set of physical CPUs on which vCPUs prefer to run on, and Xen will try as hard as possible to follow this indication.

• Security improvements – guest introspection expansion: VM introspection using Intel EPT / AMD RVI hardware virtualization functionality builds on Xen Project Hypervisor Memory Inspection APIs introduced in 2011. This addresses a number of security issues from outside the guest OS, without relying on functionality that can be rendered unreliable by advanced malware. The approach works by auditing access of sensitive memory areas using HW support in guests with minimal overhead and allows control software running within a dedicated VM to allow or deny attempts to access sensitive memory based on policy and security heuristics. You can find an excellent introduction on the topic of VM introspection here and a video on Youtube (a recording of this presentation) explaining the new functionality in Xen 4.5.

• Serial support for debug purposes. This covers PCIe cards (Oxford ones) and newer Broadcom ones found on blades.

• Experimental support for Real-Time Scheduling: a new, multicore-enabled, real-time scheduler, called RTDS is part of Xen 4.5 as an experimental feature. Virtualization will soon become the norm rather than the exception in automotive, avionics, mobile and multimedia, and other fields where predictability and high-end, real-time support are critical. Xen wants to play a big role in this, and this new scheduler will allow for such, which is why we introduced it in 4.5 while still under development. More information here: Youtube video, Linux Foundation presentation and related blog.

Intel Hypervisor-Specific Updates

• Broadwell Supervisor Mode Access Prevention. This LWN article has an excellent explanation of it – but a short summary is that it restricts the kernel from accessing the user-space pages. This feature in Xen also added alternative assembler support to patch the hypervisor during run-time (so that we won’t be running these operations on older hardware).

• Haswell Server Cache QoS Monitoring, aka Intel Resource Director Technology, is a “new area of architecture extension that seeks to provide better information and control of applications running on Intel processors. The feature, ” … documented in the Software Developers’ Manual, relates to monitoring application thread LLC usage, to provide a means of directing such usage and provide more information on the amount of memory traffic out of the LLC,” according to xen-devel.

• SandyBridge (vAPIC) extensions.  Xen 4.3 added support for VT-d Posted Interrupts, and  in Xen 4.5 we added extensions for PVHVM guests to take advantage of VT-d Posted Interrupts. Instead of using vector callback, the guest can utilize the vAPIC to lower its VMEXIT overhead, leading to lower interrupt latency and performance improvements for I/O intensive workloads in PVHMM guests.

AMD Hypervisor-Specific Updates

• Fixes in the microcode loading.

• Data Breakpoint Extensions and further MSR masking support for Kabini, Kaveri and newer. This allows “.. to specify cpuid masks to help with cpuid levelling across a pool of hosts,” from the xen-command-line manual.

ARM Hypervisor-Specific Updates

The ARM ecosystem operates differently than the x86 architecture – in which ARM licensees design new chipsets and features and OEMs manufacture platforms based on these specifications. OEMs designing ARM-based platforms determine what they need on the SoC – that is the System On Chip. As such, they can selectively enable or disable certain functionality that they consider important (or unimportant). ARM provides the Intellectual Property (IP) and standards from which OEMs can further specialize and optimize. Therefore the features Xen Project Hypervisor supports on ARM are not for a specific platform – but rather for functionality SoCs provide. New updates include:

• Support for up to 1TB for guests.

• The Generic Interrupt Controller (GIC) v3 is supported in Xen 4.5. v3 is very important because it introduces support for Message Signaled Interrupts (MSI), emulation of GICv3 for guests – and most importantly – for more than 8 CPUS. Many of the new features are not used by Xen yet but the driver is on par with v2.

• Power State Coordination Interface 0.2 (PSCI) is important in embedded environments where power consumption needs to be kept to the absolute minimum. It allows us to power down/up CPUS, suspend them, etc.

• UEFI booting. On ARM64 servers both U-Boot and UEFI can be used to boot the OS.

• IOMMU support (SMMUv1). For isolation between guests, ARM platforms can come with an IOMMU chipset based on the SMMU specification.

• Super Pages (2MB) support in Xen. Using super pages for the guest pseudo-physical to physical translation tables significantly improves overall guest performance.

• Passthrough – the PCI passthrough features did not make it on time, but doing passthrough of MMIO regions did. In the ARM world, it is quite common to have no PCIe devices and to only access devices using MMIO regions. As such this feature allows us to have driver domains be in charge of network or storage devices.

• Interrupt latency reduction: By removing maintenance interrupts, we get rid of an expensive trap into Xen for each interrupt EOI. Please see Stefano’s slides.

With these new features, the following motherboards are now supported in Xen Project Hypervisor 4.5:

• AMD Seattle

• Broadcom 7445D0 A15

• Midway (Calxeda)

• Vexpress (ARM Ltd.)

• OMAP5, DRA7 (Texas Instrument)

• Exynos5250 (Exynos 5 Dual), Odroid-Xu, and Exynos 5420 (Exynos Octa) (Samsung SoC for Arndale and various smartphones and tablets)

• SunXI (AllWinner), aka A20/A21, CubieTruck, CubieBoard

• Mustang (Applied Micro-X-Gene, the ARMv8 SoC)

• McDivitt aka HP Moonshot cartridge (Applied Micro X-Gene)

• The Xen Project also maintains this list of ARM boards that work with Xen Project software.

Toolstack Updates

Xen Project software is now using a C-based toolstack called xl or libxl, replacing the obsolete Python toolstack called xend.  This more modern architecture is easier to easier maintain, and users will not be affected by the move since xm and xl offer feature parity. In fact, the switch greatly simplifies managing Xen instances as other toolstack, such as libvirt are C based and less complex. libvirt and XAPI are now using libxl as well. For more background, check out our new hands-on tutorial “XM to XL: A Short, but Necessary, Journey.”

Additional toolstack changes include:

• VM Generation ID. This allows Windows 2012 Server and later active directory domain controllers to be migrated.

• Remus initial support provides high availability by check pointing guests states at high frequency.

• Libxenlight (libxl) JSON infrastructure support. This allows libxenlight to use JSON to communicate with other toolstacks.

• Libxenlight to keep track of domain configuration. It now uses the JSON infrastructure to keep track of domain configuration. The is feature parity with Xend.

• Systemd support. This allows one source base to contain the systemd files, which can be used by various distributions instead of them having to generate them.

• vNUMA,while still in progress,  is coming along nicely thanks to sponsorship from . Virtual NUMA allows Xen to expose to the guest the NUMA topology (either based on the host or made-up) for the guest.

On the libvirt side, changes include:

• PCI/SR-IOV passthrough, including hot{un}plug

• Migration support

• Improved concurrency through job support in the libxl driver – no more locking entire driver when modifying a domain

• Improved domxml-{to,from}-native support, e.g. for converting between xl config and libvirt domXML and vise-versa

• PV console support

• Improved qdisk support

• Support for <interface type=’network’> – allows using libvirt-managed networks in the libxl driver

• Support PARAVIRT and ACPI shutdown flags

• Support PARAVIRT reboot flag

• Support for domain lifecycle event configuration, e.g. on_crash, on_reboot, etc

• A few improvements for ARM

• Lots of bug fixes

QEMU Updates

Xen Project 4.5 will ship with QEMU v2.0 and SeaBIOS v1.7.5 with the following updates:

• Bigger PCI hole in QEMU via the mmio_hole parameter in guest config. This allows users to pack more legacy PCI devices for passthrough in an guest.

• QEMU is now built for ARM providing backend support for framebuffer (VNC).

OSes

The 4.5 release also takes advantage of new features in Linux and FreeBSD such as PVH support (which is considered experimental)

Summary

With 43 major new features, 4.5 includes the most updates in our project’s history. That’s not even counting 22 new enablers in up-streams (Linux and QEMU). The Project is also taking a more holistic, proactive approach to managing dependencies such as Linux and QEMU, as well as downstream functionality such as libvirt. In 2015, we plan to build on this even further up the stack to include OpenStack and other key projects. For the first time, our Project’s development process is robust, active and mature enough to systematically focus on these strategic growth opportunities. It also reflects enhanced responsiveness to community feedback; for example, we’re improving usability and performing broader testing for specific use cases with new releases.

During this development and release we’ve seen a steady influx of folks helping, contributing, testing and reporting. As the Release Manager, I would like to thank everybody and call out major contributions coming from AMD, Bitdefender, Citrix, Fujitsu, GlobalLogic, Intel, Linaro, Oracle, SuSE and Cavium, as well as several individual and academic institutions.
The sources are located in the git tree or one can download the tarball:

• xen: with a recent enough git (>= 1.7.8.2) just pull from the proper tag (RELEASE-4.5.0) from the main repo directly:

• git clone -b RELEASE-4.5.0 git://xenbits.xen.org/xen.git

• With an older git version (and/or if that does not work, e.g., complaining with a message like this: Remote branch RELEASE-4.5.0 not found in upstream origin, using HEAD instead), do the following:

• git clone git://xenbits.xen.org/xen.git ; cd xen ; git checkout RELEASE-4.5.0

• tarball: here it is a 4.5.0 and its signature.

Release Documentation can be found on our wiki.