If you are planning to deploy SCION alongside an existing network, you want answers long before any hardware is installed: How does the Anapaya appliance fit into your topology? How does it interact with the routers you already run? Do your configurations and automation actually work?
Containerlab is an open-source tool for deploying local container-based networking labs that can be used to validate interactions between network appliances from different vendors. Network operating systems from Nokia, Arista, Cisco, Juniper, and many others can already run as Containerlab nodes. In this post, we show how we run the Anapaya appliance, a full virtual machine, as a Containerlab node. Starting from one of our published VM images, we first bake a ready-to-use appliance disk and then build topologies on top of it, ending up with a virtual copy of our physical test rack. If you are an operator evaluating the appliance or preparing a deployment, this matters in two ways:
We can verify the appliance against the devices you use. Since the appliance is just another node in a lab topology, we can test its correctness and compatibility against other vendors' equipment that Containerlab supports, without having that hardware in our rack.
You can build the same lab yourself. Everything in this post starts from the appliance VM images we already publish. This post shows how to package the appliance as a Containerlab node, so that you can test it locally in a topology that matches your network.
Every deployment starts from the same clean appliance, so you can easily try out different configurations. And because a lab is a single file, you can version it and share it with your team.
Last year, we started publishing performance and compatibility evaluations as part of our certification program at Anapaya. The evaluations run on our in-office rack, driven by a testing orchestrator that installs specific versions of our appliance software on a selection of hosts, configures them, and runs the tests. See our testing methodology for details.
The problem: we have a single rack, and only one person can use it at a time. Every new orchestrator feature required engineers to coordinate access (and iterating on real hardware is slow). We wanted a virtualized copy of the rack, so that every engineer could test against it on their own workstation in parallel, and only move to the real rack once the obvious bugs were gone.
Above is the topology we want to virtualize, where DUT1 and DUT2 are Anapaya appliances and TG is the traffic generator.
We needed a solution that would:
Our research led us to Containerlab.
Containerlab (or clab) allows us to locally deploy arbitrary network topologies with a single configuration file. The power of clab comes from the multitude of virtual appliances it already supports out of the box.
A topology definition file is the only input to clab. We first define the nodes, and then the link endpoints. Clab uses this file to deploy the nodes and build links between them. Here is an example configuration file, taken directly from the Containerlab documentation:
This file defines two nodes (sw1 and sw2) and a single link between them. Each node has a kind field that tells the clab runtime which strategy to use to deploy and manage the node. For example, linux is a generic container, for which clab simply deploys a Docker container with the given image. For every link, clab creates a veth pair and moves its ends into the network namespaces of the containers identified as the endpoints. For management, clab connects each container to a management network (a Docker bridge on the host), and it adds /etc/hosts entries and an SSH client configuration on the host, so that the nodes can be reached by name.
There is no dedicated “Anapaya appliance” kind, so we have to build our own using the generic_vm kind. Just like linux, this kind describes a Docker container, but it hints to clab that we will run QEMU inside the container. In the next section, we explain how we packaged our appliance as a Docker image that can be directly used in a clab topology.
The recommended way to run virtual machines in clab is to use vrnetlab. Our solution follows the same approach. Nothing in this recipe is specific to our appliance: it works for any VM-based network appliance that Containerlab does not support natively. If you want to follow along, all you need is Containerlab, Docker, a host with KVM, genisoimage, and one of our published appliance VM images.
The idea is to wrap our virtual machine in a Docker image that runs QEMU. This runner image contains QEMU and a launch script, but not the VM disk itself, it lives on the host and is mounted into the container. We use the runner image in two steps:
qcow2 image.Build topologies on top of the baked disk, as many and as often as we like.
We start from the BIOS variant of the qcow2 image we publish on our releases page. On top of this image, we complete the appliance installation and bake in our SSH keys, so that any engineer can use the disk directly on their own workstation.
The launch script (/launch_qemu.sh) starts the VM with the disk, the cloud-init seed image (more on that below) as a CD-ROM, and a management port. For the management port, we use QEMU's user-mode networking, and we forward the SSH port of the container to the SSH port of the virtual machine. The launch script looks like this:
The runner image itself only needs QEMU and the networking tools:
Build it with docker build -t anapaya-appliance:latest ..
To provision SSH keys, we use cloud-init. We write the key to ~/.ssh/authorized_keys2, which the appliance leaves untouched, so access keeps working after the first appliance configuration is applied. A minimal user-data file looks like this:
Pack it, together with an empty meta-data file, into a seed image labeled cidata:
Define a topology with a single node that mounts the disk and the seed image into the container:
Run clab deploy -t bake.clab.yml and wait until cloud-init has run (see when cloud-init runs), installation has completed, and ssh root@clab-bake-appliance accepts your key. Then shut the VM down cleanly with ssh root@clab-bake-appliance poweroff, and remove the topology with clab destroy -t bake.clab.yml. All changes were written directly to disk.qcow2, which is now our baked appliance disk.
With the baked disk in place, we can build topologies on top of it. From now on, we never want to modify the disk again: every deployment should start from the same clean appliance, and several appliances should be able to share the same disk file. This is what QEMU's -snapshot flag does: all writes go to a temporary overlay that is discarded when the VM stops. For this step, we therefore change the launch script: we add -snapshot, and we remove the cloud-init seed image, which is no longer needed. In the topology, we mount the baked disk read-only. A topology with two interconnected Anapaya appliances could look something like this:
Saving the topology as topology.clab.yml and running clab deploy -t topology.clab.yml will:
Create two containers, clab-bench-1.dev.fsnets.local and clab-bench-2.dev.fsnets.local, from the anapaya-appliance:latest Docker image,
Attach each container's management interface (eth0) to the clab management network, a Docker bridge on the host (your dev machine), and
Create one veth pair connecting the containers to each other (for the link that we defined).
This is all good, but notice that the virtual machines cannot talk to each other yet: clab only interconnects the containers, and the virtual machines running inside them do not see these links. If we shell into one of the containers, we can see two interfaces besides the loopback:
Both eth0 and eth1 are created by clab: eth0 is the management interface, which the VM already reaches through user-mode networking, and eth1 is one end of the veth pair that clab has created for us. The data interface eth1 needs to be mapped into the virtual machine. Compared to the launch script from step 1, the launch script for step 2 differs in three ways:
The seed image (/mnt/host/init.img) is no longer attached,
-snapshot is added, so that the baked disk is never modified, and
A data port is added for eth1 (DATA_PORT).
The full launch script looks like this:
For the data port, we use a tap interface, tap1 (see the QEMU networking documentation). Notice that we set ctrl_vlan=off to disable VLAN filtering. This way, the virtual NIC will receive all VLAN traffic. Finally, we provide a link-up script, /etc/tc-tap-ifup, that brings tap1 up and forwards traffic in both directions between eth1 and tap1:
Replace launch_qemu.sh with the new launch script, add the link-up script to the runner image with COPY tc-tap-ifup /etc/tc-tap-ifup and RUN chmod +x /etc/tc-tap-ifup, and rebuild the image. The baked disk is not affected, so there is no need to bake again (if you ever do, rebuild the image with the launch script from step 1). After redeploying, we can SSH into the appliance and see our links:
enp1s0 is the management interface, and ens2 is the data port as seen from within the VM. Once the data ports are configured on both appliances, a quick connectivity check with ping tells us that the appliances are connected.
The last piece of the puzzle is the traffic generator. We use TRex, which has community-maintained Docker images that can be used directly in a clab topology. In this topology, each appliance has a second data interface, eth2, so the launch script gets a second data port for eth2 (tap2), with its own link-up script, in the same way as for eth1. The final topology file looks something like this:
This is the same topology we use for testing in our in-office rack. With a single clab deploy, we can virtualize our rack on our workstations and build our testing orchestration software against this virtualized environment.
Our virtualized rack is now used extensively for testing our orchestrator, and sometimes even for debugging our appliance locally.
The main lesson is that a VM-based appliance can easily be used as a regular Containerlab node: Wrapping the appliance VM in a small Docker image that runs QEMU, and declaring it with the generic_vm kind, is enough to place it in any topology next to containers and other vendors' nodes. We bake the disk once, using cloud-init to provision the SSH keys, and then deploy it as many times as we like: -snapshot and a read-only mount let every deployment start from the same clean appliance, and let several appliances share one disk file.
In return, we depend much less on real hardware: testing against the virtual rack first catches a significant portion of the issues before anyone needs to touch the physical rack. And because a lab is just a file, it can be versioned, shared, and generated, which makes testing against many different devices practical.
Finally, everything starts from the appliance VM images we already publish, so you can reproduce this setup in your own lab and extend it to match your network.