In this post we will focus on the state of virtualization for Windows Hyper-V on ARM64, checking if we can run Windows and Linux virtual machines.
This is the second part of the ‘State of Windows on ARM64’ series. Click here for the first part.
The test lab consists of 3 Ampere Computing EMAG servers (Lenovo HR330A – https://amperecomputing.com/emag), each with 32 ARM64 processors, 128 GB of RAM and 512 GB SSD.
Install Windows 10 PRO ARM64 on the Ampere Computing eMAG ARM servers
To begin with, please follow the instruction in the previous blog post to install Windows 10 on an ARM server. Windows Server ARM builds are not yet publicly available, so we have to stick to Windows 10 for now, but that’s enough for our purposes.
Enabling Hyper-V
In the Windows 10 build we used, trying to enable Hyper-V via the control panel GUI results in an error about virtualization not being available, but the command line installation works as expected:
|
1 |
dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All |
As usual a reboot is required for the Hyper-V feature to be enabled.
Small detour – PowerShell on Windows 10 ARM64
By default, Windows 10 PRO ARM64 comes with PowerShell for x86 platform, which is very slow (to put it nicely).
Downloading the native PowerShell 7 for ARM64 from https://github.com/PowerShell/PowerShell/releases greatly improved the sysadmin experience on this Windows version. Now we can use the Hyper-V cmdlets straight from the native version with no extra steps required.
Windows VMs on Hyper-V
Creating a Windows VM on Hyper-V is straightforward. We used the same Windows Insiders VHDX as the host and created a VM.
For the Windows VM creation, you can either use Hyper-V Manager or PowerShell:
|
1 2 3 |
$vm = New-VM "Windows ARM 1" -MemoryStartupBytes 4GB -SwitchName 'Default Switch' -VHDPath .\WindowsInsiders.vhdx $vm | Set-VMProcessor -Count 2 $vm | Start-VM |
Note that on this hardware, only Generation 2 UEFI virtual machines are supported.
Side note: while the Hyper-V processes (VMMS.exe, VMWP.exe) are native, the Hyper-V Manager and the VM console are X86 emulated, which means again that they are very, very slow for managing your VMs. x64 emulation is in the works for the next version of Windows 10 for ARM64, I am eager to see if some of the tooling gets upgraded to x64 if not to the native versions.
Linux VMs on Hyper-V
For this step, we are going to test an Ubuntu 20.04 ARM64 cloud image, downloaded from http://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.img. Using qemu-img, we can convert the qcow2 image to the VHDX format needed for Hyper-V.
Here I used an x86 32 bit qemu-img version that can run emulated on Windows ARM64, making the conversion step possible without resorting to Linux.
Next, I created the Hyper-V VM, disabled Secure boot and started the VM.
Aaand bummer!, the kernel was loaded by GRUB2 correctly, then it just hanged. That meant that the default Linux kernel version 5.4.x from Ubuntu 20.04 was not compatible with Hyper-V on ARM64 out of the box.
Luckily there are some pending Linux Kernel patches that add support for Linux running on Hyper-V ARM64:
- https://lore.kernel.org/patchwork/patch/1287593/
- https://patchwork.kernel.org/project/linux-arm-kernel/list/?series=337693&submitter=&state=*&q=&archive=both&delegate=
I had to build the Linux Kernel using the above patches on a Linux box (I used a x86_64 Ubuntu 18.04 VM):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# install toolchain sudo apt install \ libncurses5-dev xz-utils libssl-dev libelf-dev bc ccache kernel-package \ devscripts build-essential lintian debhelper git wget bc fakeroot crudini flex bison \ asciidoc libdw-dev systemtap-sdt-dev libunwind-dev libaudit-dev libslang2-dev \ libperl-dev python-dev binutils-dev libiberty-dev liblzma-dev libnuma-dev openjdk-8-jdk \ libbabeltrace-ctf-dev libbabeltrace-dev rename gawk # install cross build tools for ARM64 sudo apt install crossbuild-essential-arm64 # clone linux git clone https://github.com/ader1990/linux --single-branch --branch hyperv_arm64 cd linux wget https://raw.githubusercontent.com/LIS/lis-pipeline/master/scripts/package_building/deps-lis/kernel_config/Microsoft/config-azure make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig # build debian packages ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j8 deb-pkg # a file named linux-image-5.9.0+_5.9.0+-1_arm64.deb should be present in the parent folder |
Prebuilt debian packages for the patched Linux kernel can be found here: https://github.com/ader1990/linux/releases.
Next, I chrooted the image, as it was the easiest way to access it, with a small twist. I needed to install the qemu-arm-static package and use chroot like this:
|
1 2 3 4 5 6 |
chroot /mnt/location-of-mounted-ubuntu-20-04 /usr/bin/qemu-aarch64-static /bin/bash dpkg -i install linux-image-5.9.0+_5.9.0+-1_arm64.deb # the dpkg install fails, because of the zz-update-grub hook, but the kernel is correctly installed # manually tweak /boot/grub/grub.cfg with the new kernel / initrd names (linux-image-5.9.0-127961-geffcd948a62d) |
Now I moved the newly tweaked image to the Hyper-V server, created a new VM using the image, remembered to disable ‘Secure Boot’ and all was set, we have a running ARM64 Linux VM on Hyper-V!
|
1 |
$vm | Set-VMFirmware -EnableSecureBoot Off |
At a first glance we experienced some soft lockup issues, which possibly hint at some additional work to be done in the Hyper-V Linux Integration Services (LIS) to improve performance, but no show stopper so far.
Wrapping up
Enabling Hyper-V and creating Windows VMs on Windows 10 PRO ARM64 was pretty straightforward.
Creating Linux VMs required a tweaked Linux kernel, but I am confident that those patches will land in the upstream kernel in the near future.
The built-in emulated tooling for Hyper-V is slow, but when the newer native PowerShell ARM64 version is installed, the experience improves.
Some features like nested virtualization are not yet available for Hyper-V on ARM64.
