Kubernetes and Weave - a no go per default

The default Weave deployment for Kubernetes is not as secure as it should and could be. What does this mean? In fact it does mean, that if you setup a Kubernetes cluster via kubeadm and install Weave Works overlay network via the recommended installation guide, any host that installs Weave manually can join this overlay network!

Details

We are reviewing Kubernetes every couple of month, because there is some (small) kind of uncertainty if Docker Swarm will stay alive or not. Therefore it is always good to know if there are alternatives, like Kubernetes, and how they fit in into an existing environment. Docker Swarm is unmatched simply to manage, we drive more than 2300 containers today, but it is not as hyped as Kubernetes. So we have to stay up-to-date how we can use Kubernetes to replace Docker Swarm if we have to. And every single day when we have a look at it, our review boils down to two main problems, networking and storage. Today we will focus on the network integration, CNI for detail.

There are are lot of Kubernetes CNI plugins out there, but some of them are widely used: Calico, Weave and Flannel. Weave is, of course, a nice solution to get up and running with CNI capabilities inside Kubernetes easily. But there is a major drawback.

The problem

If you are following the default installation guide for Weave within Kubernetes there is no password used to protect your overlay network against suspicious network peers!

In the following output, the host atlxkube474 is not part of the Kubernetes cluster. But it can join the created Kubernetes Weave network easily by specifying one of the main peers during weave launch.

Kubernetes cluster nodes:

[19:52 atlxkube471 ~]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
atlxkube471   Ready    master   12d   v1.15.3
atlxkube472   Ready    <none>   11d   v1.15.3
atlxkube473   Ready    <none>   11d   v1.15.3

Suspicious Weave host:

19:54 atlxkube474 ~]# kubeclt

Command 'kubeclt' not found, did you mean:

  command 'kubectl' from snap kubectl (1.15.3)

See 'snap info <snapname>' for additional versions.

[19:55 atlxkube474 ~][127]# weave launch 10.x.x.1
... truncated output ...
INFO: 2019/09/16 17:55:22.483698 sleeve ->[10.x.x.2:6783|c6:cf:11:33:a1:ca(atlxkube472)]: Effective MTU verified at 1438
[19:56 atlxkube474 ~]# eval $(weave env)
[19:56 atlxkube474 ~]# weave status

        Version: 2.5.2 (up to date; next check at 2019/09/17 01:10:02)

        Service: router
       Protocol: weave 1..2
           Name: da:09:b8:ee:77:3e(atlxkube474)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 1
    Connections: 3 (3 established)
          Peers: 4 (with 12 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

        Service: dns
         Domain: weave.local.
       Upstream: 10.x.x.50, 10.x.x.51, 10.x.x.52
            TTL: 1
        Entries: 0

        Service: proxy
        Address: unix:///var/run/weave/weave.sock

        Service: plugin (legacy)
     DriverName: weave

And now, everyone can run a container that joins the singe weave overlay network and can do anything that is possible:

[20:00 atlxkube471 ~]# kubectl get pods -n deployment-v1 -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP          NODE          NOMINATED NODE   READINESS GATES
nginx-deployment-5754944d6c-z486x   1/1     Running   0          6d9h   10.44.0.1   atlxkube472   <none>           <none>

[19:59 atlxkube474 ~]# docker run --name a1 -ti weaveworks/ubuntu
root@a1:/# ping 10.44.0.1
PING 10.44.0.1 (10.44.0.1) 56(84) bytes of data.
64 bytes from 10.44.0.1: icmp_seq=1 ttl=64 time=2.00 ms
64 bytes from 10.44.0.1: icmp_seq=2 ttl=64 time=0.672 ms
^C
--- 10.44.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.672/1.337/2.002/0.665 ms
root@a1:/#

This example shows that any host can really, really easily join the Weave overlay network! This is not a secure by default design!

Fix it

I know, that it is possible to set a password for Weave, which is used to encrypt the network traffic and to protect unknown host to join the Kubernetes created Weave overlay network. This is described here.

Lets do this for our Kubernetes installation right now. Thanks to summerswallow-whi for opening a KOps issue which is already addressing this. The issue is still open (May 2018), 🙁, but it provides a lot of information how you can iron your Weave setup.

I tried it on my own and the following steps are enough to bring some kind of protection to your Weave overlay.

First, create a password for your Weave overlay and save it to a file:

# < /dev/urandom tr -dc A-Za-z0-9 | head -c16 > weave-password

Now create a Kubernetes secret:

# kubectl create secret -n kube-system generic weave-password --from-file=./weave-password

Add this setting to the Weave Kubernetes daemonset by editing it (under the weave-net container spec):

kubectl create secret -n kube-system generic weave-password --from-file=./weave-password
...
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: weave-net
    spec:
      containers:
      - command:
        - /home/weave/launch.sh
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: WEAVE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: weave-password
              name: weave-password
       image: docker.io/weaveworks/weave-kube:2.5.2

If you now try to join the Weave overlay network, you will see the following failure:

[20:19 atlxkube474 ~]# weave launch 10.x.x.1
24ef2192c30e3c5d9372469eb1fb456e348cdd9efe4b8cda27c3ba1e756ba73c
[20:19 atlxkube474 ~]# docker logs weave
...
INFO: 2019/09/16 18:19:50.677943 ->[10.x.x.1:6783] connection shutting down due to error during handshake: no password specificed, but peer requested an encrypted connection

Yes! This is want we want! And nothing less!

Conclusion

Weave is using, I my opinion, a non secure default setup. This violates the GDPR article 25 at least. Encryption is a default today! This is one of the points where Docker Swarm is much, much better. Docker swarm creates a VXLAN overlay network for each service per default (not just one single overlay for anything like Weave does)! 😎 Furthermore you cannot(!) join a Docker Swarm without knowing the join token and therefore you cannot infiltrate a existing Docker Swarm overlay network! It is secure by default! No additional screws needed!

It is hard to believe, but even Weave Works itself does not provide a Kubernetes documentation about how to enhance your Kubernetes Weave setup (Kubernetes Secrets, Kubernetes Daemonset,…) Even the actual edition of the book Kubernetes: Up and running, Second Edition does not mention something like this in any way.

I don’t like to know, how many insecure setups are out there - hopefully everyone trust his or her own network. 🙄

Sometimes we think, that the whole industry is just fueling up the whole Kubernetes thing to sell more and more consulting services…

But, lets see…