Update CLAUDE.md with comprehensive debugging lessons learned

Based on our extensive debugging session, this update adds critical documentation:

## DNS Architecture and Troubleshooting
- Explained the local_service_ip design and why it requires route_localnet
- Added detailed DNS debugging methodology with exact steps in order
- Documented systemd socket activation complexities and common mistakes
- Added specific commands to verify DNS is working correctly

## Architectural Decisions
- Added new section explaining trade-offs in Algo's design choices
- Documented why local_service_ip uses loopback instead of alternatives
- Explained iptables-legacy vs iptables-nft backend choice

## Enhanced Debugging Guidance
- Expanded troubleshooting with exact commands and expected outputs
- Added warnings about configuration changes that need restarts
- Documented socket activation override requirements in detail
- Added common pitfalls like interface-specific sysctls

## Time Wasters Section
- Added new lessons learned from this debugging session
- Interface-specific route_localnet (fails before interface exists)
- DNAT for loopback addresses (doesn't work)
- BPF JIT hardening (causes errors on many kernels)

This documentation will help future maintainers avoid the same debugging
rabbit holes and understand why things are designed the way they are.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

CLAUDE.md

@@ -176,38 +176,64 @@ This practice ensures:
 - Too many tasks to fix immediately (113+)
 - Focus on new code having proper names
 
-### 2. dnscrypt-proxy Service Failures
+### 2. DNS Architecture and Common Issues
+
+#### Understanding local_service_ip
+- Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface
+- This IP (`local_service_ip`) is where dnscrypt-proxy should listen
+- Requires `net.ipv4.conf.all.route_localnet=1` sysctl for VPN clients to reach loopback IPs
+- This is by design for consistency across VPN types (WireGuard + IPsec)
+
+#### dnscrypt-proxy Service Failures
 **Problem:** "Unit dnscrypt-proxy.socket is masked" or service won't start
 - The service has `Requires=dnscrypt-proxy.socket` dependency
 - Masking the socket prevents the service from starting
-- **Solution:** Configure socket properly instead of fighting it (see systemd section above)
+- **Solution:** Configure socket properly instead of fighting it
 
-### 3. DNS Not Accessible to VPN Clients
+#### DNS Not Accessible to VPN Clients
-**Symptoms:** VPN connects but no internet access
+**Symptoms:** VPN connects but no internet/DNS access
-- First check: `sudo ss -ulnp | grep :53` on the server
+1. **First check what's listening:** `sudo ss -ulnp | grep :53`
-- If only showing 127.0.0.53 or 127.0.2.1, socket activation is misconfigured
+   - Should show `local_service_ip:53` (e.g., 172.24.117.23:53)
-- Check firewall allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}`
+   - If showing only 127.0.2.1:53, socket override didn't apply
-- **Never** allow DNS from all sources (0.0.0.0/0) - security risk!
+2. **Check socket status:** `systemctl status dnscrypt-proxy.socket`
+   - Look for "configuration has changed while running" - needs restart
+3. **Verify route_localnet:** `sysctl net.ipv4.conf.all.route_localnet`
+   - Must be 1 for VPN clients to reach loopback IPs
+4. **Check firewall:** Ensure allows VPN subnets: `-A INPUT -s {{ subnets }} -d {{ local_service_ip }}`
+   - **Never** allow DNS from all sources (0.0.0.0/0) - security risk!
 
-### 4. Multi-homed Systems and NAT
+### 3. Multi-homed Systems and NAT
 **DigitalOcean and other providers with multiple IPs:**
 - Servers may have both public and private IPs on same interface
 - MASQUERADE needs output interface: `-o {{ ansible_default_ipv4['interface'] }}`
 - Don't overengineer with SNAT - MASQUERADE with interface works fine
 - Use `alternative_ingress_ip` option only when truly needed
 
-### 5. Jinja2 Template Complexity
+### 4. iptables Backend Changes (nft vs legacy)
+**Critical:** Switching between iptables-nft and iptables-legacy can break subtle behaviors
+- Ubuntu 22.04+ defaults to iptables-nft which may have implicit NAT behaviors
+- Algo forces iptables-legacy for consistent rule ordering
+- This switch can break DNS routing that "just worked" before
+- Always test thoroughly after backend changes
+
+### 5. systemd Socket Activation Gotchas
+- Interface-specific sysctls (e.g., `net.ipv4.conf.wg0.route_localnet`) fail if interface doesn't exist yet
+- WireGuard interface only created when service starts
+- Use global sysctls or apply settings after service start
+- Socket configuration changes require explicit restart (not just reload)
+
+### 6. Jinja2 Template Complexity
 - Many templates use Ansible-specific filters
 - Test templates with `tests/unit/test_template_rendering.py`
 - Mock Ansible filters when testing
 
-### 6. OpenSSL Version Compatibility
+### 7. OpenSSL Version Compatibility
 ```yaml
 # Check version and use appropriate flags
 {{ (openssl_version is version('3', '>=')) | ternary('-legacy', '') }}
 ```
 
-### 7. IPv6 Endpoint Formatting
+### 8. IPv6 Endpoint Formatting
 - WireGuard configs must bracket IPv6 addresses
 - Template logic: `{% if ':' in IP %}[{{ IP }}]:{{ port }}{% else %}{{ IP }}:{{ port }}{% endif %}`
 
@@ -289,11 +315,13 @@ Each has specific requirements:
 ### Time Wasters to Avoid (Lessons Learned)
 **Don't spend time on these unless absolutely necessary:**
 1. **Converting MASQUERADE to SNAT** - MASQUERADE works fine for Algo's use case
-2. **Fighting systemd socket activation** - Configure it properly instead
+2. **Fighting systemd socket activation** - Configure it properly instead of trying to disable it
 3. **Debugging NAT before checking DNS** - Most "routing" issues are DNS issues
-4. **Complex IPsec policy matching** - Keep NAT rules simple
+4. **Complex IPsec policy matching** - Keep NAT rules simple, avoid `-m policy --pol none`
 5. **Testing on existing servers** - Always test on fresh deployments
-6. **Adding `-m policy --pol none`** - This breaks NAT, don't use it
+6. **Interface-specific route_localnet** - WireGuard interface doesn't exist until service starts
+7. **DNAT for loopback addresses** - Packets to local IPs don't traverse PREROUTING
+8. **Removing BPF JIT hardening** - It's optional and causes errors on many kernels
 
 ## Working with Algo
 
@@ -329,27 +357,105 @@ ansible-playbook users.yml -e "server=SERVER_NAME"
 
 ### Troubleshooting VPN Connectivity
 
-#### "VPN connects but can't route traffic" - Check in this order:
+#### Debugging Methodology
-1. **DNS first** - `sudo ss -ulnp | grep :53` - Is dnscrypt-proxy listening on VPN IPs?
+When VPN connects but traffic doesn't work, follow this **exact order** (learned from painful experience):
-2. **Packet counters** - `sudo iptables -L FORWARD -v -n | grep -E '10.49|10.48'` - Are packets reaching the firewall?
-3. **NAT counters** - `sudo iptables -t nat -L POSTROUTING -v -n` - Is NAT happening?
-4. **Service status** - `sudo systemctl status dnscrypt-proxy` - Is the DNS service running?
 
-**Important:** Most "routing" issues are actually DNS issues. Always check DNS first.
+1. **Check DNS listening addresses first**
+   ```bash
+   ss -lnup | grep :53
+   # Should show local_service_ip:53 (e.g., 172.24.117.23:53)
+   # If showing 127.0.2.1:53, socket override didn't apply
+   ```
 
-#### systemd and dnscrypt-proxy
+2. **Check both socket AND service status**
-- Ubuntu's dnscrypt-proxy package uses socket activation by default
+   ```bash
-- The default socket listens on 127.0.2.1:53, NOT the VPN service IPs
+   systemctl status dnscrypt-proxy.socket dnscrypt-proxy.service
-- Work WITH systemd, not against it:
+   # Look for "configuration has changed while running" warnings
-  ```yaml
+   ```
-  # Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/override.conf
+
-  [Socket]
+3. **Verify route_localnet is enabled**
-  ListenStream=  # Clear defaults
+   ```bash
-  ListenStream=172.x.x.x:53  # Add VPN IP
+   sysctl net.ipv4.conf.all.route_localnet
-  ```
+   # Must be 1 for VPN clients to reach loopback IPs
-- Use empty `listen_addresses = []` in dnscrypt-proxy.toml when using socket activation
+   ```
-- **Never** use `TriggeredBy=` in systemd units (it's not a valid directive)
+
-- Don't mask sockets that services depend on - just disable them
+4. **Test DNS resolution from server**
+   ```bash
+   dig @172.24.117.23 google.com  # Use actual local_service_ip
+   # Should return results if DNS is working
+   ```
+
+5. **Check firewall counters**
+   ```bash
+   iptables -L INPUT -v -n | grep -E '172.24|10.49|10.48'
+   # Look for increasing packet counts
+   ```
+
+6. **Verify NAT is happening**
+   ```bash
+   iptables -t nat -L POSTROUTING -v -n
+   # Check for MASQUERADE rules with packet counts
+   ```
+
+**Key insight:** 90% of "routing" issues are actually DNS issues. Always check DNS first!
+
+#### systemd and dnscrypt-proxy (Critical for Ubuntu/Debian)
+**Background:** Ubuntu's dnscrypt-proxy package uses systemd socket activation which **completely overrides** the `listen_addresses` setting in the config file.
+
+**How it works:**
+1. Default socket listens on 127.0.2.1:53 (hardcoded in package)
+2. Socket activation means systemd opens the port, not dnscrypt-proxy
+3. Config file `listen_addresses` is ignored when socket activation is used
+4. Must configure the socket, not just the service
+
+**Correct approach:**
+```bash
+# Create socket override at /etc/systemd/system/dnscrypt-proxy.socket.d/10-algo-override.conf
+[Socket]
+ListenStream=              # Clear ALL defaults first
+ListenDatagram=            # Clear UDP defaults too
+ListenStream=172.x.x.x:53  # Add TCP on VPN IP
+ListenDatagram=172.x.x.x:53 # Add UDP on VPN IP
+```
+
+**Config requirements:**
+- Use empty `listen_addresses = []` in dnscrypt-proxy.toml for socket activation
+- Socket must be restarted (not just reloaded) after config changes
+- Check with: `systemctl status dnscrypt-proxy.socket` for warnings
+- Verify with: `ss -lnup | grep :53` to see actual listening addresses
+
+**Common mistakes:**
+- Trying to disable/mask the socket (breaks service with Requires= dependency)
+- Only setting ListenStream (need ListenDatagram for UDP)
+- Forgetting to clear defaults first (results in listening on both IPs)
+- Not restarting socket after configuration changes
+
+## Architectural Decisions and Trade-offs
+
+### DNS Service IP Design
+Algo uses a randomly generated IP in the 172.16.0.0/12 range on the loopback interface for DNS (`local_service_ip`). This design has trade-offs:
+
+**Why it's done this way:**
+- Provides a consistent DNS IP across both WireGuard and IPsec
+- Avoids binding to VPN gateway IPs which differ between protocols
+- Survives interface changes and restarts
+- Works the same way across all cloud providers
+
+**The cost:**
+- Requires `route_localnet=1` sysctl (minor security consideration)
+- Adds complexity with systemd socket activation
+- Can be confusing to debug
+
+**Alternatives considered but rejected:**
+- Binding to VPN gateway IPs directly (breaks unified configuration)
+- Using dummy interface instead of loopback (non-standard, more complex)
+- DNAT redirects (doesn't work with loopback destinations)
+
+### iptables Backend Choice
+Algo forces iptables-legacy instead of iptables-nft on Ubuntu 22.04+ because:
+- nft reorders rules unpredictably, breaking VPN traffic
+- Legacy backend provides consistent, predictable behavior
+- Trade-off: Lost some implicit NAT behaviors that nft provided
 
 ## Important Context for LLMs