We are scaling our application across three different cloud VMs and need the containers to talk to each other as if they were on the same local network. I’ve started using Docker Swarm, but my overlay network seems to have high latency and frequent disconnects. What are the best practices for MTU settings and firewall rules to ensure a stable overlay network in 2024?
3 answers
You can also use the --attachable flag when creating your overlay network if you need non-swarm containers to join the same network later
Overlay networking issues in the cloud are almost always related to MTU (Maximum Transmission Unit) mismatches or blocked ports. The default Docker MTU is 1500, but the overlay encapsulation adds a 50-byte overhead. If your cloud provider's network also has a 1500 limit, your packets will get fragmented or dropped. Try setting your Docker network MTU to 1450. Additionally, ensure your cloud security groups allow UDP port 4789 (for the data plane), TCP/UDP port 7946 (for control plane gossip), and TCP port 2377 (for cluster management). Without these, the overlay network will never be stable.
Are you using an external key-value store like Consul or Etcd, or are you relying on the built-in Swarm mode raft store for your network state?
Brian, we are sticking with the built-in Swarm raft store to keep things simple. Nancy, the MTU tip was a lifesaver! Our cloud provider (OpenStack-based) had a strict 1500 limit, and the fragmentation was causing our database syncs to fail. Dropping the Docker MTU to 1400 completely stopped the disconnects. I also realized we hadn't opened port 4789 for UDP, which explains why the containers could see each other's metadata but couldn't actually pass any traffic.
Good point, Deborah. The attachable flag is crucial for hybrid setups where you might have standalone containers interacting with a Swarm service.