ARM servers are more and more present in our day to day life, their usage varying from minimal IoT devices to huge computing clusters. So we decided to put the Windows support for ARM64 cloud images to the test, with two primary focuses:
- Toolchain ecosystem – Building and running Cloudbase-Init on Windows ARM64
- Virtualization – Running Windows and Linux virtual machines on Windows ARM64
Our friends from https://amperecomputing.com kindly provided the computing resources that we used to check the current state of Windows virtualization on ARM64.
The test lab consisted of 3 Ampere Computing EMAG servers (Lenovo HR330A – https://amperecomputing.com/emag), each with 32 ARM64 processors, 128 GB of RAM and 512 GB SSD.
Toolchain ecosystem on Windows ARM64: building and running Cloudbase-Init
Cloudbase-Init is a provisioning agent designed to initialize and configure guest operating systems on various platforms: OpenStack, Azure, Oracle Cloud, VMware, Kubernetes CAPI, OpenNebula, Equinix Metal (formerly: Packet), and many others.
Building and running Cloudbase-Init requires going through multiple layers of an OS ecosystem, as it needs a proper build environment, C compiler for Python and Python extensions, Win32 and WMI wrappers, a Windows service wrapper and an MSI installer.
This complexity made Cloudbase-Init the perfect candidate for checking the state of the toolchain ecosystem on Windows ARM64.
Install Windows 10 PRO ARM64 on the EMAG ARM servers
EMAG servers come with CentOS 7 preinstalled, so the first step was to have a Windows ARM64 OS installed on them.
Windows Server ARM64 images are unfortunately not publicly available, so the best option consists in using Windows Insider (https://insider.windows.com/), Windows 10 PRO ARM64 images available for download.
As there is no ISO available on the Windows Insiders website, we had to convert the VHDX to a RAW file using qemu-img.exe, boot a Linux Live ISO which had dd binary tool on it (Ubuntu is great for this) on the EMAG server and copy the RAW file content directly on the primary disk.
For the dd step, we needed a Windows machine where to download / convert the Windows 10 PRO ARM64 VHDX and two USB sticks. One USB stick for the Ubuntu Live ISO and one for the Windows 10 PRO ARM64 RAW file.
Rufus was used for creating the Ubuntu Live ISO USB and copying the RAW file to the other USB stick. Note that one USB stick must be at least 32 GB in size to cover for the ~25GB of the Windows RAW file.
Tools used for the dd step:
- qemu-img.exe for VHDX -> RAW conversion: https://cloudbase.it/qemu-img-windows/
- Ubuntu 20.04 Focal Fossa ISO for ARM64: https://cdimage.ubuntu.com/focal/daily-live/current/focal-desktop-arm64.iso
- Rufus for burning ISO and dd-ing the RAW file on USB sticks: https://github.com/pbatard/rufus/releases
After the dd process succeeds, a server reboot was required. The first boot took a while for the Windows device initialization followed by the usual “Out of the box experience”.
The following steps show how we built Cloudbase-Init for ARM64. As a side note, Windows 10 ARM64 has a builtin emulator for x86, but not for x64. Practically, we could run the x86 version of Cloudbase-Init on the system, but it would have run very slow and some features would have been limited by the emulation (starting native processes).
Gather information on the toolchain required to build Cloudbase-Init
The Cloudbase-Init ecosystem consists of these main building blocks:
- Python for Windows ARM64
- Python setuptools
- Python pip
- Python PyWin32
- Cloudbase-Init
- OpenStack Service Wrapper executable
Toolchain required:
- Visual Studio with ARM64 support (2017 or 2019)
- git
Python for Windows ARM64
Python 3.x for ARM64 can be built using Visual Studio 2017 or 2019. In our case, we used the freely available Visual Studio 2019 Community Edition, downloadable from https://visualstudio.microsoft.com/downloads/.
The required toolchain / components for Visual Studio can be installed using this vsconfig.txt. This way, we make sure that the build environment is 100% reproducible.
Python source code can be found here: https://github.com/python/cpython.
To make the build process even easier, we leveraged GitHub Actions to easily build Python for ARM64. An example workflow can be found here: https://github.com/cloudbase/cloudbase-init-arm-scripts/blob/main/.github/workflows/build.yml.
Also, prebuilt archives of Python for Windows ARM64 are available for download here: https://github.com/ader1990/CPython-Windows-ARM64/releases.
Notes:
- We used Python 3.8.x ARM64.
- A requirement for Cloudbase-Init (oslo.config) is not yet supported on Python 3.9.x > 3.9.rc2, otherwise 3.9 is ok as well.
- A CPython patch is required to build the PyWin32 Cloudbase-Init requirement https://github.com/ader1990/cpython/commit/b8c59c9b96a7ad11094224b5631aae3b89323f7a
Python setuptools
Python setuptools is a Python package that handles the “python setup.py install” workflow.
Source code can be found here: https://github.com/pypa/setuptools.
The following patches are required for setuptools to work:
- https://github.com/ader1990/setuptools/commit/3e936d223e78952e86a5501742adfd2f1b154cb0
- https://github.com/ader1990/setuptools/commit/4345aa40888ab15e2789487c5df215fa42504948
- https://github.com/ader1990/setuptools/commit/64c10b9a5facd8e0cdabc57bc5fc56f67b2890fa
Installation steps for setuptools (Python and Visual Studio are required):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
set VCVARSALL="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" set CL_PATH="C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX86\ARM64\cl.exe" set MC_PATH="C:\Program Files (x86)\Windows Kits\10\bin\10.0.17763.0\arm64\mc.exe" call %VCVARSALL% amd64_arm64 10.0.17763.0 & set git clone https://github.com/ader1990/setuptools 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 pushd setuptools git checkout am_64 echo "Installing setuptools" python.exe bootstrap.py 1>nul 2>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 %CL_PATH% /D "GUI=0" /D "WIN32_LEAN_AND_MEAN" /D _ARM64_WINAPI_PARTITION_DESKTOP_SDK_AVAILABLE launcher.c /O2 /link /MACHINE:ARM64 /SUBSYSTEM:CONSOLE /out:setuptools/cli-arm64.exe IF %ERRORLEVEL% NEQ 0 EXIT 1 python.exe setup.py install 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 popd |
Python pip
Python pip is required for easier management of Cloudbase-Init’s requirements installation and wheels building.
Python’s wheel package is required to build wheels. Wheels are the pre-built versions of Python packages. There is no need to have a compiler to install the package from source on the exact system version the wheel has been built for.
Pip sources can be found here: https://github.com/pypa/pip.
The following pip patch is required: https://github.com/ader1990/pip/commit/0559cd17d81dcee43433d641052088b690b57cdd.
The patch introduces two binaries required for ARM64, which were built from: https://github.com/ader1990/simple_launcher/tree/win_arm64
This patched version of pip can use the wheel to create proper binaries for ARM64 (like setuptools).
Installation steps for wheel (Python is required):
|
1 2 3 |
echo "Installing pip" python.exe -m easy_install https://github.com/ader1990/pip/archive/20.3.dev1.win_arm64.tar.gz 1>nul 2>nul IF %ERRORLEVEL% NEQ 0 EXIT |
Python PyWin32
Python PyWin32 package is a wrapper for (almost) all Win32 APIs from Windows. It is a behemoth from the source code perspective, with Cloudbase-Init using a limited amount of Win32 APIs via PyWin32.
Source code can be found here: https://github.com/mhammond/pywin32.
The following patches are required:
- https://github.com/ader1990/pywin32/commit/e7864cdeb9474eb7d5e672ea5aee989f9457362d
- https://github.com/ader1990/pywin32/commit/f6795ff6cb91c0c613c23e5bb3897f2e7d45b9f5
- https://github.com/ader1990/pywin32/commit/8d02293df764e69fdafd225316922a9a806e25b7
Installation steps for PyWin32 (Python 3.8 and Visual Studio 2019 are required):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
echo "Installing pywin32" git clone https://github.com/ader1990/pywin32 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 pushd pywin32 git checkout win_arm64 IF %ERRORLEVEL% NEQ 0 EXIT 1 pushd "win32\src" %MC_PATH% -A PythonServiceMessages.mc -h . popd pushd "isapi\src" %MC_PATH% -A pyISAPI_messages.mc -h . popd mkdir "build\temp.win-arm64-3.8\Release\scintilla" 1>nul 2>nul echo '' > "build\temp.win-arm64-3.8\Release\scintilla\scintilla.dll" python.exe setup.py install --skip-verstamp IF %ERRORLEVEL% NEQ 0 EXIT 1 popd |
The build process takes quite a lot of time, at least half an hour, so we took a(nother) cup of coffee and enjoyed the extra time.
The patches hardcode some compiler quirks for Visual Studio 2019 and remove some unneeded extensions from the build. There is work in progress to prettify and upstream the changes.
Cloudbase-Init
Now, as all the previous steps have been completed, it is time to finally build Cloudbase-Init. Thank you for your patience.
Source code can be found here: https://github.com/cloudbase/cloudbase-init
Installation steps for Cloudbase-Init (Python and Visual Studio are required):
|
1 2 3 4 5 6 7 8 9 10 |
echo "Installing Cloudbase-Init" git clone https://github.com/cloudbase/cloudbase-init 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 pushd cloudbase-init echo "Installing Cloudbase-Init requirements" python.exe -m pip install -r requirements.txt 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 python.exe -m pip install . 1>nul IF %ERRORLEVEL% NEQ 0 EXIT 1 popd |
After the installation steps were completed, the cloudbase-init.exe AMR64 executable wrapper will be available.
OpenStack Service Wrapper executable
Cloudbase-Init usually runs as a service at every boot. As cloudbase-init.exe is a normal executable, it needs a service wrapper for Windows. A service wrapper is a small program that implements the hooks for the Windows service actions, like start, stop and restart.
Source code can be found here: https://github.com/cloudbase/OpenStackService
The following patch was required: https://github.com/ader1990/OpenStackService/commit/a48c4e54b3f7db7d4df163a6d7e13aa0ead4a58b
For an easier build process, a GitHub actions workflow file can be found here: https://github.com/ader1990/OpenStackService/blob/arm64/.github/workflows/build.yml
A prebuilt release binary for OpenStackService ARM64 is available for download here: https://github.com/ader1990/OpenStackService/releases/tag/v1.arm64
Epilogue
Now we are ready to use Cloudbase-Init for guest initialization on Windows 10 PRO ARM64.
Main takeaways:
- The main building blocks (Python and Visual Studio) are in great shape to be used for ARM64 applications
- Some of the Python packages required for Cloudbase-Init still need minor tweaks when it comes to the build process on ARM64.
