Terraform(ing) Docker hosts with LinuxKit on-premise (VMware vCenter)
Table of contents
Preface
This blog post aims not be a fully-fledged step by step tutorial on how you can create and bootstrap a Docker Swarm cluster with VMware vCenter on-premise. Instead, it should give you an idea about what is possible and why we do it this way. There are different ways to achieve different goals and the way we explain here shows only one way of how you can do it.
Today we are running around 30 Docker hosts which are installed as classic VMWare virtual machines and these Docker hosts are all based upon Ubuntu Linux. This leads to the fact that we have to update our Docker hosts every three months manually to get in touch with the (annual) up-to-date Docker-CE version. Pointless work. We are provisioning our Docker hosts with puppet. Even if we are provisioning our hosts with puppet, it would take time to create new Docker hosts to let the Docker Swarm workers rotate. Yes, we could create a blue-green infrastructure to rotate them, but this will use computing resources if we create them beforehand or let them run all the time. If we would create them on-demand, every three month, it would take time to bring them up and running, update them, and so on. This is now the point to introduce new players. Docker LinuxKit as Docker host OS and HashiCorp Terraform as IaC (Infrastructure as Code). Yes, there is also Docker InfraKit and we are currently evaluating it, but there will be more work to do.
LinuxKit
LinuxKit is the operating system project started by Docker to provide a toolkit for building custom minimal, immutable Linux distributions. The benefit you get, if you choose to go with LinuxKit, is not only to have a custom made Linux distribution which reflects your needs, furthermore you get a platform which enables you to push a resulting iso-image to a VMWare datastore for example. There are already different cloud providers build into the LinuxKit toolkit.
LinuxKit example
Downloading and building LinuxKit is very well described on the LinuxKit GitHub page. After you have built the LinuxKit binary you need a yaml file that describes the compositions of the LinuxKit operating system you would like to create. The example linuxkit.yaml included in the sources is a good starting point. But after some work you will recognize that the simple example will maybe be not enough. Therefore, the next lines will show our basic example where we are starting from to include some additional packages we need.
|
|
LinuxKit example explanation
The first thing you need to know is, that LinuxKit uses {bash}containerd{/bash} as container engine to power most of the needed operating system daemons. As you can see in the {bash}linuxkit.yml{/bash}, the different sections are using {bash}images{/bash} known as container-images to build up the system. Important: This images are using the OCI open container initiative image standard. I will reference back to this point later, so just keep this information in mind.
The first line of the {bash}linuxkit.yml{/bash} takes a Linux kernel image and loads it. The file is not using any {bash}:latest{/bash} tag as image definition and you would like to avoid them too because you would like to know what versions are operating in your operating system. Please do not copy the file as it is, because it will be outdated nearly immediately!
After the definition of the {bash}kernel{/bash} there comes the {bash}init{/bash} section. In this section, there are those things located which will be needed immediately, for example the containerd image. This image will be responsible for the upcoming service containers.
Next in the row, there are the services you will probably need for your environment. We need the open-vm-tools image, because we are running the resulting image on the VMWare ESXi infrastructure and without the tools it would not be possible to retrieve the ip address information from the VMWare vCenter and this is a must as we are going to build the virtual machines with Terraform later.
The ssh daemon should be self explaining, but you will need some root/.ssh/authorized_keys to access the running LinuxKit OS. Therefore, look at the files section, where you can describe your configuration needs.
Now we are installing the Docker engine because we would like to build up a Docker Swarm worker and this Docker Swarm worker should join an existing Docker swarm manager later. Important: You will notice, that for the Docker engine, we are using the DockerHub image as usual, no special LinuxKit image! But how does this work? As I said before, the containerd is using OCI standard images which are not the same as Docker images. Lets have a look into it.
LinuxKit persistent disk magic
When it comes to the point, that you have to startup your Docker swarm cluster or even a Docker swarm host with LinuxKit, you will sooner or later ask yourself how you can manage it to persist your Docker Swarm data or at least the information which Docker containers are started.
The Docker data lives inside the /var/lib/docker folder. Therefore, if you persist this folder, you are able to persist the current state of the Docker host.
The collegues at Docker have done their work the right way. Look at the lines, where the images format and mount are loaded. This two images are doing the magic which enables LinuxKit to persist it’s data. You can lookup the documentation at github for details. For the impatient, here’s the summary.
The format image will take the first block device it finds and if there is no Linux partition on it, it will format it. If it finds a Linux partition, nothing happens. Very comfortable! After the disk format is done (or not) the partition gets mounted via the mount image and the corresponding mountie configuration lines. Et voila, there you go. Magical persistence with LinuxKit iso-turbo-boost mode. Genius!
This is one of the most important features, when it comes to infrastructure as code because regardless what you take, InfraKit or Terraform, you might eventually need some kind of persistence to get a reliable infrastructure.
LinuxKit OCI and Docker images
The really cool stuff about LinuxKit is, that it is a toolset. This means, that during the linux build command, which we will see later, the image components as described by the linuxkit.yml are downloaded from DockerHub and afterwards the contents of the used images are transformed to OCI compatible root image filesystems. This means, you can use all the DockerHub images directly without worrying about the image format. Neat! Now you know why you need a Docker environment to build LinuxKit and to build your LinuxKit OS afterwards.
LinuxKit build your image
To build your LinuxKit iso-image, you can use the following command. We have created a separate docker.yml file, to reflect the changes. The resulting iso-image will therefore be named docker.iso automatically.
./linuxkit build -format iso-bios docker.yml
LinuxKit push image to VMware
After you have build your iso-image, you can push it to a VMware datastore with the following command. Important: There is a small problem located in the actual version of LinuxKit. You cannot push to a VMware datacenter if there are multiple hosts located in it. Thanks to Dan Finneran @thebsdbox who helped me a lot! There is already a github PR merged which makes it possible to push without a problem. The PR will be included in the next version (0.2) of LinuxKit.
|
|
Terraform
Sure, at some point in time we will maybe use InfraKit to get our things up and running. But as for today there are hardly alternatives to Terraform. Terraform is Open Source Software but you can purchase an enterprise license if you need. When you start to dig into the parts and pieces of Terraform, you will recognize, that it is not the easiest piece of software but incredibly powerful.
Just download Terraform from the website and unpack it. The only thing you will see after extraction is just a binary called terraform.
Before you can do anything with it you will need a configuration which describes what you would like to receive. Due to this concept, Terraform is working with states. You plan a state, you apply a state and you will destroy a state. Terraform will try to keep your resources consistent. Now it comes to the config.
Terraform example
provider "vsphere" {
user = "youruser"
password = "yourpass"
vsphere_server = "yourserver"
# if you have a self-signed cert
allow_unverified_ssl = true
}
variable "instance_hostnames" {
default = {
"0" = "tf-sw-0"
"1" = "tf-sw-1"
"2" = "tf-sw-2"
}
}
data "vsphere_datacenter" "dc" {
name = "Spittal"
}
data "vsphere_datastore" "datastore" {
name = "yourdatastore"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
data "vsphere_resource_pool" "pool" {
name = "yourresourcepool"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
data "vsphere_network" "network" {
name = "yourlan"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
output "instance_ips" {
value = ["${vsphere_virtual_machine.vm.*.guest_ip_addresses}"]
}
resource "vsphere_folder" "parent" {
path = "TERRAFORM/Linux"
type = "vm"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
resource "vsphere_folder" "folder" {
path = "${vsphere_folder.parent.path}/linuxkit-dev"
type = "vm"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
resource "vsphere_virtual_machine" "vm" {
count = "1"
name = "${lookup(var.instance_hostnames, count.index)}"
folder = "${vsphere_folder.folder.path}"
resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
datastore_id = "${data.vsphere_datastore.datastore.id}"
wait_for_guest_net_timeout = false
num_cpus = 2
memory = 1024
guest_id = "ubuntu64Guest"
network_interface {
network_id = "${data.vsphere_network.network.id}"
adapter_type = "vmxnet3"
}
cdrom {
datastore_id = "${data.vsphere_datastore.datastore.id}"
path = "docker/docker.iso"
}
disk {
name = "tf.vmdk"
size = 10
}
}If you run this example with ./terraform plan and terraform, you will get three LinuxKit VMWare virtual machines which are saving their persistent state to the hard drive.
Conclusion
You can use the information from this blog post to build up a basic infrastructure. But to build up a running Docker Swarm cluster with managers and workers there is more work to do. There are a few glitches which makes it a little bit complicated at the moment to use the VMWare customization against the LinuxKit running virtual machines for example. In turn, it might be better to go with VMWare templates at the moment because you will have more possibilities. Maybe we will post an update on this soon. Stay tuned!
Doing Linux since 2000 and containers since 2009. Like to hack new and interesting stuff. Containers, Python, DevOps, automation and so on. Interested in science and I like to read (if I found the time). My motto is "𝗜𝗺𝗮𝗴𝗶𝗻𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗺𝗼𝗿𝗲 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝘁𝗵𝗮𝗻 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲. [Einstein]". Interesting contacts are always welcome - nice to meet you out there - if you like, do not hesitate and contact me!
