ray
117 lines
3.4 KiB
Markdown
117 lines
3.4 KiB
Markdown
# Docker User-Defined Bridge: Lost Gateway IPv4 Address
|
|
|
|
## Symptom
|
|
|
|
A Docker container is running, `docker ps` shows it as healthy, and `docker exec` can reach it from inside. But accessing the published port from the **host** or **other machines on the network** hangs/times out.
|
|
|
|
```
|
|
$ curl -v http://localhost:2283/
|
|
* Trying 127.0.0.1:2283...
|
|
* Connected to localhost (127.0.0.1) port 2283
|
|
> GET / HTTP/1.1
|
|
> ...
|
|
* Request completely sent off
|
|
* Operation timed out after 10001 milliseconds with 0 bytes received
|
|
```
|
|
|
|
Docker-proxy is running and listening on the port, but responses never come back.
|
|
|
|
## Diagnosis
|
|
|
|
```bash
|
|
# docker-proxy is running and forwarding correctly
|
|
ss -tlnp | grep <port>
|
|
# → LISTEN 0.0.0.0:<port>
|
|
|
|
# Docker NAT rule exists
|
|
sudo iptables -t nat -L DOCKER -n | grep <port>
|
|
# → DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port> to:172.XX.0.X:<port>
|
|
|
|
# Docker filter allows traffic
|
|
sudo iptables -L DOCKER -n | grep <container_ip>
|
|
|
|
# But the host can't reach the container's IP
|
|
ping -c 2 <container_ip>
|
|
# → 100% packet loss
|
|
|
|
# The bridge interface has NO IPv4 address
|
|
ip addr show br-<hash>
|
|
# Only shows inet6 fe80::... — NO inet 172.XX.0.1/16
|
|
```
|
|
|
|
## Root Cause
|
|
|
|
Docker's user-defined bridge networks (created by `docker compose`) need a **gateway IP** on the host-side bridge interface (`br-<hash>`). This IP is normally assigned when the network is first created.
|
|
|
|
If the bridge loses its IPv4 (e.g. after `docker compose restart` without a full `down`+`up`, or due to a Docker daemon restart), the host has **no route** to the container's subnet:
|
|
|
|
```bash
|
|
ip route | grep 172.18
|
|
# → (nothing — no route)
|
|
```
|
|
|
|
docker-proxy still listens and accepts connections, but it forwards to the container's IP via the bridge. Without the host having an IP on that bridge, the forwarded packets can't reach the container and responses can't come back.
|
|
|
|
## Fix
|
|
|
|
### Preferred: Full Network Recreate
|
|
|
|
```bash
|
|
cd /opt/<project>/
|
|
docker compose down
|
|
docker compose up -d
|
|
```
|
|
|
|
This recreates the network fresh, which assigns the gateway IP correctly. **All containers are recreated**, so the network starts clean.
|
|
|
|
### Quick Patch (if downtime is unacceptable)
|
|
|
|
```bash
|
|
sudo ip addr add <gateway_ip>/<mask> dev br-<hash>
|
|
```
|
|
|
|
Find the gateway IP from `docker network inspect`:
|
|
|
|
```bash
|
|
docker network inspect <name> | grep -A 5 '"Config"'
|
|
# "Gateway": "172.18.0.1"
|
|
```
|
|
|
|
Then:
|
|
|
|
```bash
|
|
sudo ip addr add 172.18.0.1/16 dev br-<hash>
|
|
```
|
|
|
|
This restores connectivity immediately but is **ephemeral** — it won't survive a reboot or Docker daemon restart. Use the full `down && up -d` for a permanent fix.
|
|
|
|
### Making the Quick Patch Persistent
|
|
|
|
If you need a stopgap before the full restart, create a oneshot systemd service:
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=Assign IP to Docker bridge
|
|
After=docker.service
|
|
Requires=docker.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStartPre=/bin/bash -c "sleep 5"
|
|
ExecStart=/sbin/ip addr add <gateway_ip>/<mask> dev br-<hash> 2>/dev/null || /bin/true
|
|
RemainAfterExit=yes
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
```bash
|
|
sudo systemctl enable --now <service>
|
|
```
|
|
|
|
## Prevention
|
|
|
|
- **Avoid `docker compose restart <service>`** for critical containers on user-defined bridge networks if you suspect the network might be degraded
|
|
- Use **full `docker compose down && docker compose up -d`** when something feels off with container networking
|
|
- After a **full machine reboot**, Docker recreates everything properly — this issue typically only appears during hot-fixes or partial restarts
|