diff --git a/CLAUDE.md b/CLAUDE.md index 75cae3e7..00ead63b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -176,38 +176,64 @@ This practice ensures: - Too many tasks to fix immediately (113+) - Focus on new code having proper names -### 2. dnscrypt-proxy Service Failures +### 2. DNS Architecture and Common Issues + +#### Understanding local_service_ip +- Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface +- This IP (`local_service_ip`) is where dnscrypt-proxy should listen +- Requires `net.ipv4.conf.all.route_localnet=1` sysctl for VPN clients to reach loopback IPs +- This is by design for consistency across VPN types (WireGuard + IPsec) + +#### dnscrypt-proxy Service Failures **Problem:** "Unit dnscrypt-proxy.socket is masked" or service won't start - The service has `Requires=dnscrypt-proxy.socket` dependency - Masking the socket prevents the service from starting -- **Solution:** Configure socket properly instead of fighting it (see systemd section above) +- **Solution:** Configure socket properly instead of fighting it -### 3. DNS Not Accessible to VPN Clients -**Symptoms:** VPN connects but no internet access -- First check: `sudo ss -ulnp | grep :53` on the server -- If only showing 127.0.0.53 or 127.0.2.1, socket activation is misconfigured -- Check firewall allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}` -- **Never** allow DNS from all sources (0.0.0.0/0) - security risk! +#### DNS Not Accessible to VPN Clients +**Symptoms:** VPN connects but no internet/DNS access +1. **First check what's listening:** `sudo ss -ulnp | grep :53` + - Should show `local_service_ip:53` (e.g., 172.24.117.23:53) + - If showing only 127.0.2.1:53, socket override didn't apply +2. **Check socket status:** `systemctl status dnscrypt-proxy.socket` + - Look for "configuration has changed while running" - needs restart +3. **Verify route_localnet:** `sysctl net.ipv4.conf.all.route_localnet` + - Must be 1 for VPN clients to reach loopback IPs +4. **Check firewall:** Ensure allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}` + - **Never** allow DNS from all sources (0.0.0.0/0) - security risk! -### 4. Multi-homed Systems and NAT +### 3. Multi-homed Systems and NAT **DigitalOcean and other providers with multiple IPs:** - Servers may have both public and private IPs on same interface - MASQUERADE needs output interface: `-o {{ ansible_default_ipv4['interface'] }}` - Don't overengineer with SNAT - MASQUERADE with interface works fine - Use `alternative_ingress_ip` option only when truly needed -### 5. Jinja2 Template Complexity +### 4. iptables Backend Changes (nft vs legacy) +**Critical:** Switching between iptables-nft and iptables-legacy can break subtle behaviors +- Ubuntu 22.04+ defaults to iptables-nft which may have implicit NAT behaviors +- Algo forces iptables-legacy for consistent rule ordering +- This switch can break DNS routing that "just worked" before +- Always test thoroughly after backend changes + +### 5. systemd Socket Activation Gotchas +- Interface-specific sysctls (e.g., `net.ipv4.conf.wg0.route_localnet`) fail if interface doesn't exist yet +- WireGuard interface only created when service starts +- Use global sysctls or apply settings after service start +- Socket configuration changes require explicit restart (not just reload) + +### 6. Jinja2 Template Complexity - Many templates use Ansible-specific filters - Test templates with `tests/unit/test_template_rendering.py` - Mock Ansible filters when testing -### 6. OpenSSL Version Compatibility +### 7. OpenSSL Version Compatibility ```yaml # Check version and use appropriate flags {{ (openssl_version is version('3', '>=')) | ternary('-legacy', '') }} ``` -### 7. IPv6 Endpoint Formatting +### 8. IPv6 Endpoint Formatting - WireGuard configs must bracket IPv6 addresses - Template logic: `{% if ':' in IP %}[{{ IP }}]:{{ port }}{% else %}{{ IP }}:{{ port }}{% endif %}` @@ -289,11 +315,13 @@ Each has specific requirements: ### Time Wasters to Avoid (Lessons Learned) **Don't spend time on these unless absolutely necessary:** 1. **Converting MASQUERADE to SNAT** - MASQUERADE works fine for Algo's use case -2. **Fighting systemd socket activation** - Configure it properly instead +2. **Fighting systemd socket activation** - Configure it properly instead of trying to disable it 3. **Debugging NAT before checking DNS** - Most "routing" issues are DNS issues -4. **Complex IPsec policy matching** - Keep NAT rules simple +4. **Complex IPsec policy matching** - Keep NAT rules simple, avoid `-m policy --pol none` 5. **Testing on existing servers** - Always test on fresh deployments -6. **Adding `-m policy --pol none`** - This breaks NAT, don't use it +6. **Interface-specific route_localnet** - WireGuard interface doesn't exist until service starts +7. **DNAT for loopback addresses** - Packets to local IPs don't traverse PREROUTING +8. **Removing BPF JIT hardening** - It's optional and causes errors on many kernels ## Working with Algo @@ -329,27 +357,105 @@ ansible-playbook users.yml -e "server=SERVER_NAME" ### Troubleshooting VPN Connectivity -#### "VPN connects but can't route traffic" - Check in this order: -1. **DNS first** - `sudo ss -ulnp | grep :53` - Is dnscrypt-proxy listening on VPN IPs? -2. **Packet counters** - `sudo iptables -L FORWARD -v -n | grep -E '10.49|10.48'` - Are packets reaching the firewall? -3. **NAT counters** - `sudo iptables -t nat -L POSTROUTING -v -n` - Is NAT happening? -4. **Service status** - `sudo systemctl status dnscrypt-proxy` - Is the DNS service running? +#### Debugging Methodology +When VPN connects but traffic doesn't work, follow this **exact order** (learned from painful experience): -**Important:** Most "routing" issues are actually DNS issues. Always check DNS first. +1. **Check DNS listening addresses first** + ```bash + ss -lnup | grep :53 + # Should show local_service_ip:53 (e.g., 172.24.117.23:53) + # If showing 127.0.2.1:53, socket override didn't apply + ``` -#### systemd and dnscrypt-proxy -- Ubuntu's dnscrypt-proxy package uses socket activation by default -- The default socket listens on 127.0.2.1:53, NOT the VPN service IPs -- Work WITH systemd, not against it: - ```yaml - # Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/override.conf - [Socket] - ListenStream= # Clear defaults - ListenStream=172.x.x.x:53 # Add VPN IP - ``` -- Use empty `listen_addresses = []` in dnscrypt-proxy.toml when using socket activation -- **Never** use `TriggeredBy=` in systemd units (it's not a valid directive) -- Don't mask sockets that services depend on - just disable them +2. **Check both socket AND service status** + ```bash + systemctl status dnscrypt-proxy.socket dnscrypt-proxy.service + # Look for "configuration has changed while running" warnings + ``` + +3. **Verify route_localnet is enabled** + ```bash + sysctl net.ipv4.conf.all.route_localnet + # Must be 1 for VPN clients to reach loopback IPs + ``` + +4. **Test DNS resolution from server** + ```bash + dig @172.24.117.23 google.com # Use actual local_service_ip + # Should return results if DNS is working + ``` + +5. **Check firewall counters** + ```bash + iptables -L INPUT -v -n | grep -E '172.24|10.49|10.48' + # Look for increasing packet counts + ``` + +6. **Verify NAT is happening** + ```bash + iptables -t nat -L POSTROUTING -v -n + # Check for MASQUERADE rules with packet counts + ``` + +**Key insight:** 90% of "routing" issues are actually DNS issues. Always check DNS first! + +#### systemd and dnscrypt-proxy (Critical for Ubuntu/Debian) +**Background:** Ubuntu's dnscrypt-proxy package uses systemd socket activation which **completely overrides** the `listen_addresses` setting in the config file. + +**How it works:** +1. Default socket listens on 127.0.2.1:53 (hardcoded in package) +2. Socket activation means systemd opens the port, not dnscrypt-proxy +3. Config file `listen_addresses` is ignored when socket activation is used +4. Must configure the socket, not just the service + +**Correct approach:** +```bash +# Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/10-algo-override.conf +[Socket] +ListenStream= # Clear ALL defaults first +ListenDatagram= # Clear UDP defaults too +ListenStream=172.x.x.x:53 # Add TCP on VPN IP +ListenDatagram=172.x.x.x:53 # Add UDP on VPN IP +``` + +**Config requirements:** +- Use empty `listen_addresses = []` in dnscrypt-proxy.toml for socket activation +- Socket must be restarted (not just reloaded) after config changes +- Check with: `systemctl status dnscrypt-proxy.socket` for warnings +- Verify with: `ss -lnup | grep :53` to see actual listening addresses + +**Common mistakes:** +- Trying to disable/mask the socket (breaks service with Requires= dependency) +- Only setting ListenStream (need ListenDatagram for UDP) +- Forgetting to clear defaults first (results in listening on both IPs) +- Not restarting socket after configuration changes + +## Architectural Decisions and Trade-offs + +### DNS Service IP Design +Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface for DNS (`local_service_ip`). This design has trade-offs: + +**Why it's done this way:** +- Provides a consistent DNS IP across both WireGuard and IPsec +- Avoids binding to VPN gateway IPs which differ between protocols +- Survives interface changes and restarts +- Works the same way across all cloud providers + +**The cost:** +- Requires `route_localnet=1` sysctl (minor security consideration) +- Adds complexity with systemd socket activation +- Can be confusing to debug + +**Alternatives considered but rejected:** +- Binding to VPN gateway IPs directly (breaks unified configuration) +- Using dummy interface instead of loopback (non-standard, more complex) +- DNAT redirects (doesn't work with loopback destinations) + +### iptables Backend Choice +Algo forces iptables-legacy instead of iptables-nft on Ubuntu 22.04+ because: +- nft reorders rules unpredictably, breaking VPN traffic +- Legacy backend provides consistent, predictable behavior +- Trade-off: Lost some implicit NAT behaviors that nft provided ## Important Context for LLMs