Docker MTU issues on Openstack

While testing in a docker container running on a VM in my Openstack cluster, I encountered a weird issue when trying to connect to services over TLS. For example, I could curl from within the container, but not DNS was working, routing was working. I could ping I just couldn’t establish a TLS connection.

After some packet tracing, I noticed that TLS ClientHello was never leaving the VM in the case.

Looking at the MTU for the docker bridge vs the virtual ethernet adapter

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:81:9d:b8 brd ff:ff:ff:ff:ff:ff
inet brd scope global dynamic eth0
valid_lft 83165sec preferred_lft 83165sec
inet6 fe80::f816:3eff:fe81:9db8/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:a1:a2:06 brd ff:ff:ff:ff:ff:ff
inet scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fea1:a206/64 scope link
valid_lft forever preferred_lft forever
29: vethda551f3@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether fe:53:b9:85:37:b8 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::fc53:b9ff:fe85:37b8/64 scope link
valid_lft forever preferred_lft forever

Openstack creates virtual ethernet devices with an MTU of 1450, I assume to accommodate encapsulation overheads. When the docker bridge is created, it does not detect the MTU of the underlying adapter. It just defaults to 1500. That MTU is propagated to the veth interfaces in the containers. That causes workloads inside the container to occasionally generate packets that are too large to be forwarded by the host and are dropped.

After going into my /etc/sysconfig/docker and adding the --mtu option
# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald --mtu=1450'

systemctl restart docker and order was restored. So if TLS connections are failing in your docker containers, think MTU.

One thought on “Docker MTU issues on Openstack”

  1. Good tip, but on Ubuntu 16.04 there’s no `/etc/sysconfig`. Instead, I added the `–mtu=1450` option this way in `/etc/default/docker`: `DOCKER_OPTS=”–mtu=1450″`.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.