OpenStack on ARM64 - LBaaS - Cloudbase Solutions

In part 2 of this series about OpenStack on ARM64, we got to the point where our cloud is fully deployed with all the Compute (VMs), Software Defined Networking (SDN) and Software Defined Storage (SDS) up and running. One additional component that we want to add is a Load Balancer as a Service (LBaaS), which is a key requirement for pretty much any high available type of workload and a must-have feature in any cloud.

OpenStack’s current official LBaaS component is called Octavia, which replaced the older Neutron LBaaS v1 project, starting with the Liberty release. Deploying and configuring requires a few steps, which explains the need for a dedicated blog post.

Octavia’s reference implementation uses VM instances called Amphorae to perform the actual load balancing. The octavia-worker service takes care of communicating with the amphorae and to do that we need to generate a few X509 CAs and certificates used to secure the communications. The good news is that starting with the Victoria release, kolla-ansible simplifies a lot this task. Here’s how to:

# Change the following according to your organization
echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml

# This is the kolla-ansible virtual env created in the previous blog post
cd kolla
source venv/bin/activate

sudo chown $USER:$USER /etc/kolla
kolla-ansible octavia-certificates

# Change the following according to your organization

echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml

echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml

echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml

echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml

# This is the kolla-ansible virtual env created in the previous blog post

cd kolla

source venv/bin/activate

sudo chown $USER:$USER /etc/kolla

kolla-ansible octavia-certificates

The communication between Octavia and the Amphorae needs an isolated network, as we don’t want to share it with the tenant network for security reasons. A simple way to accomplish that is to create a provider network with a dedicated VLAN ID, which is why we enabled Neutron provider networks and OVS VLAN segmentation in the previous post. Again, starting with Victoria, this got much easier with kolla-ansible.

# This is a dedicated network, outside your management LAN address space, change as needed
OCTAVIA_MGMT_SUBNET=192.168.43.0/24
OCTAVIA_MGMT_SUBNET_START=192.168.43.10
OCTAVIA_MGMT_SUBNET_END=192.168.43.254
OCTAVIA_MGMT_HOST_IP=192.168.43.1/24
OCTAVIA_MGMT_VLAN_ID=107

sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_network:
  name: lb-mgmt-net
  provider_network_type: vlan
  provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID
  provider_physical_network: physnet1
  external: false
  shared: false
  subnet:
    name: lb-mgmt-subnet
    cidr: "$OCTAVIA_MGMT_SUBNET"
    allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"
    allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"
    gateway_ip: "$OCTAVIA_MGMT_HOST_IP"
    enable_dhcp: yes
EOT

# This is a dedicated network, outside your management LAN address space, change as needed

OCTAVIA_MGMT_SUBNET=192.168.43.0/24

OCTAVIA_MGMT_SUBNET_START=192.168.43.10

OCTAVIA_MGMT_SUBNET_END=192.168.43.254

OCTAVIA_MGMT_HOST_IP=192.168.43.1/24

OCTAVIA_MGMT_VLAN_ID=107

sudo tee -a /etc/kolla/globals.yml << EOT

octavia_amp_network:

provider_network_type: vlan

provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID

provider_physical_network: physnet1

external: false

shared: false

subnet:

cidr: "$OCTAVIA_MGMT_SUBNET"

allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"

allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"

gateway_ip: "$OCTAVIA_MGMT_HOST_IP"

enable_dhcp: yes

EOT

Unless there is a dedicated network adapter, a virtual ethernet one can be used. This needs to be configured at boot and added to the OVS br-ex switch.

# This sets up the VLAN veth interface
# Netplan doesn't have support for veth interfaces yet
sudo tee /usr/local/bin/veth-lbaas.sh << EOT
#!/bin/bash
sudo ip link add v-lbaas-vlan type veth peer name v-lbaas
sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas
sudo ip link set v-lbaas-vlan up
sudo ip link set v-lbaas up
EOT
sudo chmod 744 /usr/local/bin/veth-lbaas.sh

sudo tee /etc/systemd/system/veth-lbaas.service << EOT
[Unit]
After=network.service

[Service]
ExecStart=/usr/local/bin/veth-lbaas.sh

[Install]
WantedBy=default.target
EOT
sudo chmod 644 /etc/systemd/system/veth-lbaas.service

sudo systemctl daemon-reload
sudo systemctl enable veth-lbaas.service
sudo systemctl start veth-lbaas.service

docker exec openvswitch_vswitchd ovs-vsctl add-port \
  br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID

# This sets up the VLAN veth interface

# Netplan doesn't have support for veth interfaces yet

sudo tee /usr/local/bin/veth-lbaas.sh << EOT

#!/bin/bash

sudo ip link add v-lbaas-vlan type veth peer name v-lbaas

sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas

sudo ip link set v-lbaas-vlan up

sudo ip link set v-lbaas up

EOT

sudo chmod 744 /usr/local/bin/veth-lbaas.sh

sudo tee /etc/systemd/system/veth-lbaas.service << EOT

[Unit]

After=network.service

[Service]

ExecStart=/usr/local/bin/veth-lbaas.sh

[Install]

WantedBy=default.target

EOT

sudo chmod 644 /etc/systemd/system/veth-lbaas.service

sudo systemctl daemon-reload

sudo systemctl enable veth-lbaas.service

sudo systemctl start veth-lbaas.service

docker exec openvswitch_vswitchd ovs-vsctl add-port \

br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID

A few more Octavia kolla-ansible configurations…

echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml

# Flavor used when booting an amphora, change as needed
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_flavor:
  name: "amphora"
  is_public: no
  vcpus: 1
  ram: 1024
  disk: 5
EOT

sudo mkdir /etc/kolla/config/octavia
# Use a config drive in the Amphorae for cloud-init
sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT
[controller_worker]
user_data_config_drive = true
EOT

echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml

echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml

# Flavor used when booting an amphora, change as needed

sudo tee -a /etc/kolla/globals.yml << EOT

octavia_amp_flavor:

is_public: no

vcpus: 1

ram: 1024

disk: 5

EOT

sudo mkdir /etc/kolla/config/octavia

# Use a config drive in the Amphorae for cloud-init

sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT

[controller_worker]

user_data_config_drive = true

EOT

…and we can finally tell kolla-ansible to deploy Octavia:

kolla-ansible -i all-in-one deploy --tags common,horizon,octavia

1	kolla-ansible -i all-in-one deploy --tags common,horizon,octavia

Octavia uses a special VM image for the Amphorae, which needs to be built for ARM64. We prepared Dockerfiles for building either an Ubuntu or a CentOS image, you can choose either one in the following snippets. We use containers to perform the build in order to isolate the requirements and be independent from the host OS.

git clone https://github.com/cloudbase/openstack-kolla-arm64-scripts
cd openstack-kolla-arm64-scripts/victoria

# Choose either Ubuntu or CentOS (not both!)

# Ubuntu
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Ubuntu \
  -t amphora-image-build-arm64-ubuntu

# Centos
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Centos \
  -t amphora-image-build-arm64-centos

git clone https://github.com/cloudbase/openstack-kolla-arm64-scripts

cd openstack-kolla-arm64-scripts/victoria

# Choose either Ubuntu or CentOS (not both!)

# Ubuntu

docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Ubuntu \

-t amphora-image-build-arm64-ubuntu

# Centos

docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Centos \

-t amphora-image-build-arm64-centos

ARM64 needs a trivial patch in the diskimage-create.sh build script (we also submitted it upstream):

git clone https://opendev.org/openstack/octavia -b stable/victoria
# Use latest branch Octavia to create Ubuntu image
cd octavia
# diskimage-create.sh includes armhf but not arm64
git apply ../0001-Add-arm64-in-diskimage-create.sh.patch
cd ..

git clone https://opendev.org/openstack/octavia -b stable/victoria

# Use latest branch Octavia to create Ubuntu image

cd octavia

# diskimage-create.sh includes armhf but not arm64

git apply ../0001-Add-arm64-in-diskimage-create.sh.patch

cd ..

Build the image (this will take a bit):

# Again, choose either Ubuntu or CentOS (not both!)

# Note the mount of /mnt and /proc in the docker container
# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,
# making the host unusable
docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \
  -v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu

# Create CentOS 8 Amphora image
docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \
  -ti amphora-image-build-arm64-centos

# Again, choose either Ubuntu or CentOS (not both!)

# Note the mount of /mnt and /proc in the docker container

# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,

# making the host unusable

docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \

-v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu

# Create CentOS 8 Amphora image

docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \

-ti amphora-image-build-arm64-centos

Add the image to Glance, using the octavia user in the service project. The amphora tag is used by Octavia to find the image.

. /etc/kolla/admin-openrc.sh

# Switch to the octavia user and service project
export OS_USERNAME=octavia
export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service

openstack image create amphora-x64-haproxy.qcow2 \
  --container-format bare \
  --disk-format qcow2 \
  --private \
  --tag amphora \
  --file octavia/diskimage-create/amphora-x64-haproxy.qcow2

# We can now delete the image file
rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2

. /etc/kolla/admin-openrc.sh

# Switch to the octavia user and service project

export OS_USERNAME=octavia

export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')

export OS_PROJECT_NAME=service

export OS_TENANT_NAME=service

openstack image create amphora-x64-haproxy.qcow2 \

--container-format bare \

--disk-format qcow2 \

--private \

--tag amphora \

--file octavia/diskimage-create/amphora-x64-haproxy.qcow2

# We can now delete the image file

rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2

Currently, we need a small patch in Octavia to properly render the userdata for the Amphorae:

# Patch the user_data_config_drive_template
cd octavia
git apply  ../0001-Fix-userdata-template.patch
# For now just update the octavia-worker container, no need to restart it
docker cp octavia/common/jinja/templates/user_data_config_drive.template \
  octavia_worker:/usr/lib/python3/dist-packages/octavia/common/jinja/templates/user_data_config_drive.template

# Patch the user_data_config_drive_template

cd octavia

git apply ../0001-Fix-userdata-template.patch

# For now just update the octavia-worker container, no need to restart it

docker cp octavia/common/jinja/templates/user_data_config_drive.template \

octavia_worker:/usr/lib/python3/dist-packages/octavia/common/jinja/templates/user_data_config_drive.template

Finally, let’s create a load balancer to make sure everything works fine:

# To create the loadbalancer
. /etc/kolla/admin-openrc.sh

openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet

# Check the status until it's marked as ONLINE
openstack loadbalancer list

# To create the loadbalancer

. /etc/kolla/admin-openrc.sh

openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet

# Check the status until it's marked as ONLINE

openstack loadbalancer list

Congratulations! You have a working LBaaS in your private cloud!!

Troubleshooting

In case something goes wrong, finding the root cause might be tricky. Here are a few suggestions to ease up the process.

# Check for errors
sudo tail -f /var/log/kolla/octavia/octavia-worker.log

# SSH into amphora
# Get amphora VM IP either from the octavia-worker.log or from:
openstack server list --all-projects

ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu
ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos

# Instances stuck in pending create cannot be deleted
# Password: grep octavia_database_password /etc/kolla/passwords.yml
docker exec -ti mariadb mysql -u octavia -p octavia
update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';
exit;

# Check for errors

sudo tail -f /var/log/kolla/octavia/octavia-worker.log

# SSH into amphora

# Get amphora VM IP either from the octavia-worker.log or from:

openstack server list --all-projects

ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu

ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos

# Instances stuck in pending create cannot be deleted

# Password: grep octavia_database_password /etc/kolla/passwords.yml

docker exec -ti mariadb mysql -u octavia -p octavia

update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';

exit;

OpenStack on ARM64 – LBaaS

Troubleshooting

Author: Alessandro Pilotti

Related posts