ray
81 lines
2.9 KiB
Markdown
81 lines
2.9 KiB
Markdown
# Docker User-Defined Bridge IP Loss
|
|
|
|
## Scenario
|
|
|
|
After migrating or partially recreating containers (e.g. `docker compose up -d <single-service>` or `docker compose restart`), the host can't reach the container via its published port. `ss -tlnp` shows the port listening, `docker-proxy` is running, but curl establishes a TCP connection then hangs with no response.
|
|
|
|
## Root Cause
|
|
|
|
Docker user-defined bridge networks (created by `docker compose` for each project) assign `172.18.0.1/16` (varies) to a `br-*` interface on the host. After partial container recreation, this bridge interface can lose its IPv4 address while remaining UP. With no address on the host side, docker-proxy can't route traffic — the SYN reaches the container but the SYN-ACK never comes back.
|
|
|
|
The route also disappears:
|
|
```bash
|
|
ip route | grep 172.18
|
|
# → (nothing)
|
|
```
|
|
|
|
## Diagnosis
|
|
|
|
```bash
|
|
# 1. Check the bridge interface
|
|
ip addr show br-* | grep -A 2 "^[0-9]"
|
|
# → UP but no "inet" line = no IPv4 address assigned
|
|
|
|
# 2. Check for the route
|
|
ip route | grep <bridge-subnet>
|
|
# → missing = no route from host to containers
|
|
|
|
# 3. Confirm no reachability from host
|
|
ping -c 2 <container-ip>
|
|
# → 100% packet loss (even though containers on same bridge reach each other)
|
|
|
|
# 4. Verify container is actually listening
|
|
docker exec <container> sh -c "cat /proc/net/tcp | grep <port-in-hex>"
|
|
# → e.g. port 2283 = 08EB, shows LISTEN (state 0A) on 00000000 (all interfaces)
|
|
|
|
# 5. Confirm inter-container reachability
|
|
docker run --rm --network <project_default> curlimages/curl:latest \
|
|
-s --max-time 5 http://<service-name>:<port>/
|
|
# → works fine! The container itself is healthy
|
|
```
|
|
|
|
## Fix
|
|
|
|
Full `docker compose down && docker compose up -d` for the entire project. This rebuilds the bridge network from scratch, assigning the gateway IP correctly.
|
|
|
|
```bash
|
|
cd /opt/<project>
|
|
docker compose down
|
|
docker compose up -d
|
|
```
|
|
|
|
After this:
|
|
```bash
|
|
ip addr show br-* | grep "inet "
|
|
# → inet 172.18.0.1/16 scope global br-...
|
|
|
|
ip route | grep 172.18
|
|
# → 172.18.0.0/16 dev br-... proto kernel scope link src 172.18.0.1
|
|
```
|
|
|
|
## Prevention
|
|
|
|
Always use `docker compose down && docker compose up -d` (full lifecycle) when:
|
|
- Changing env vars in `.env`
|
|
- Changing port mappings in compose
|
|
- After a migration where compose files were copied to a new machine
|
|
- When the bridge network is being used by a single service that was recreated
|
|
|
|
`docker compose restart` or `docker compose up -d <service>` is safe for code/configuration changes inside a running container (new env vars loaded from outside, etc.) but NOT for network-level configuration changes.
|
|
|
|
## Emergency Workaround (no downtime)
|
|
|
|
If you can't take the project down but need access immediately:
|
|
|
|
```bash
|
|
sudo ip addr add <gateway-ip> dev br-<id>
|
|
# e.g. sudo ip addr add 172.18.0.1/16 dev br-e5a0b03f3857
|
|
```
|
|
|
|
This restores connectivity immediately without restarting containers. But it won't survive a reboot — the proper fix (full down/up) is still needed for persistence.
|