Set up a personal VPN in the cloud
Find a file
Dan Guido c495307027
Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801)
* Fix DigitalOcean cloud-init compatibility issue causing SSH timeout on port 4160

This commit addresses the issue described in GitHub issue #14800 where DigitalOcean
deployments fail during the "Wait until SSH becomes ready..." step due to cloud-init
not processing the write_files directive correctly.

## Problem
- DigitalOcean's cloud-init shows "Unhandled non-multipart (text/x-not-multipart) userdata" warning
- write_files module gets skipped, leaving SSH on default port 22 instead of port 4160
- Algo deployment times out when trying to connect to port 4160

## Solution
Added proactive detection and remediation to the DigitalOcean role:
1. Check if SSH is listening on the expected port (4160) after droplet creation
2. If not, automatically apply the SSH configuration manually via SSH on port 22
3. Verify SSH is now listening on the correct port before proceeding

## Changes
- Added SSH port check with 30-second timeout
- Added fallback remediation block that:
  - Connects via SSH on port 22 to apply Algo's SSH configuration
  - Backs up the original sshd_config
  - Applies the correct SSH settings (port 4160, security hardening)
  - Restarts the SSH service
  - Verifies the fix worked

This ensures DigitalOcean deployments succeed even when cloud-init fails to process
the user_data correctly, maintaining backward compatibility and reliability.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Implement cleaner fix for DigitalOcean cloud-init encoding issue

This replaces the previous workaround with two targeted fixes that address
the root cause of the "Unhandled non-multipart (text/x-not-multipart) userdata"
issue that prevents write_files from being processed.

## Root Cause
Cloud-init receives user_data as binary/bytes instead of UTF-8 string,
causing it to fail parsing and skip the write_files directive that
configures SSH on port 4160.

## Cleaner Solutions Implemented

### Fix 1: String Encoding (user_data | string)
- Added explicit string conversion to user_data template lookup
- Ensures DigitalOcean API receives proper UTF-8 string, not bytes
- Minimal change with maximum compatibility

### Fix 2: Use runcmd Instead of write_files
- Replaced write_files approach with runcmd shell commands
- Bypasses the cloud-init parsing issue entirely
- More reliable as it executes direct shell commands
- Includes automatic SSH config backup for safety

## Changes Made
- `roles/cloud-digitalocean/tasks/main.yml`: Added | string filter to user_data
- `files/cloud-init/base.yml`: Replaced write_files with runcmd approach
- Removed complex SSH detection/remediation workaround (no longer needed)

## Benefits
-  Fixes root cause instead of working around symptoms
-  Much simpler and more maintainable code
-  Backward compatible - no API changes required
-  Handles both potential failure modes (encoding + parsing)
-  All tests pass, linters clean

This should resolve DigitalOcean SSH timeout issues while being much
cleaner than the previous workaround approach.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix cloud-init header format for DigitalOcean compatibility

The space in '# cloud-config' (introduced in PR #14775) breaks cloud-init
YAML parsing on DigitalOcean, causing SSH configuration to be skipped.

Cloud-init documentation requires '#cloud-config' without a space.

Fixes #14800

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Revert to write_files approach for SSH configuration

Using write_files is more maintainable and Ansible-native than runcmd.
The root cause was the cloud-config header format, not write_files itself.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix Ansible deprecation and variable warnings

- Replace deprecated network filters with ansible.utils equivalents:
  - ipaddr → ansible.utils.ipaddr
  - ipmath → ansible.utils.ipmath
  - ipv4 → ansible.utils.ipv4
  - ipv6 → ansible.utils.ipv6
  - next_nth_usable → ansible.utils.next_nth_usable

- Fix reserved variable name: no_log → algo_no_log

- Fix SSH user groups warning by explicitly specifying groups parameter

Addresses deprecation warnings that would become errors after 2024-01-01.
All linter checks pass with only cosmetic warnings remaining.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add comprehensive protection for cloud-config header format

- Add inline documentation explaining critical #cloud-config format requirement
- Exclude files/cloud-init/ from yamllint and ansible-lint to prevent automatic 'fixes'
- Create detailed README.md documenting the issue and protection measures
- Reference GitHub issue #14800 for future maintainers

This prevents regression of the critical cloud-init header format that
causes deployment failures when changed from '#cloud-config' to '# cloud-config'.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add test for cloud-init header format to prevent regression

This test ensures the cloud-init header remains exactly ''#cloud-config''
without a space. The regression in PR #14775 that added a space broke
DigitalOcean deployments by causing cloud-init YAML parsing to fail,
resulting in SSH timeouts on port 4160.

Co-authored-by: Dan Guido <dguido@users.noreply.github.com>

* Refactor SSH config template and fix MOTD task permissions

- Use dedicated sshd_config template instead of inline content
- Add explicit become: true to MOTD task to fix permissions warning

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix no_log variable references after renaming to algo_no_log

Update all remaining references from old 'no_log' variable to 'algo_no_log'
in WireGuard, SSH tunneling, and StrongSwan roles. This fixes deployment
failures caused by undefined variable references.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Correct YAML indentation in cloud-init template for DigitalOcean

The indent filter was not indenting the first line of the sshd_config content,
causing invalid YAML structure that cloud-init couldn't parse. This resulted
in SSH timeouts during deployment as the port was never changed from 22 to 4160.

- Add first=True parameter to indent filter to ensure all lines are indented
- Remove extra indentation in base template to prevent double-indentation
- Add comprehensive test suite to validate template rendering and prevent regressions

Fixes deployment failures where cloud-init would show:
"Invalid format at line X: expected <block end>, but found '<scalar>'"

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Dan Guido <dguido@users.noreply.github.com>
2025-08-03 14:25:47 -04:00
.github Bump docker/metadata-action from 5.5.1 to 5.8.0 (#14797) 2025-08-03 08:02:22 -04:00
configs ECDSA fixed 2016-07-24 14:44:59 +03:00
docs docs: Add sudo requirement for local installations (#14790) 2025-08-03 07:04:17 -04:00
files/cloud-init Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
library chore: Conservative dependency updates for Jinja2 security fix (#14792) 2025-08-03 07:45:26 -04:00
playbooks Apply ansible-lint improvements (#14775) 2025-08-03 01:33:15 -04:00
roles Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
scripts chore: Conservative dependency updates for Jinja2 security fix (#14792) 2025-08-03 07:45:26 -04:00
tests Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
venvs on-build python venvs (#1199) 2018-11-22 13:04:58 -05:00
.ansible-lint Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
.dockerignore updating .dockerignore file (#14559) 2023-03-03 00:54:49 -04:00
.gitignore fix: Correct Azure requirements file path to resolve deployment failures (#14781) 2025-08-03 04:56:06 -04:00
.yamllint Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
algo Refactor to support Ansible 2.8 (#1549) 2019-09-28 08:10:20 +08:00
algo-docker.sh chore: Conservative dependency updates for Jinja2 security fix (#14792) 2025-08-03 07:45:26 -04:00
algo-showenv.sh Refactor to support Ansible 2.8 (#1549) 2019-09-28 08:10:20 +08:00
ansible.cfg Refactor to support Ansible 2.8 (#1549) 2019-09-28 08:10:20 +08:00
CHANGELOG.md Update CHANGELOG.md 2020-08-06 19:32:23 +03:00
CLAUDE.md Add Claude Code GitHub Workflow (#14798) 2025-08-03 08:01:41 -04:00
cloud.yml Ansible upgrade 6.1 (#14500) 2022-07-30 15:01:24 +03:00
CODEOWNERS Add CODEOWNERS file (#14599) 2023-07-25 14:55:28 +03:00
config.cfg Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
CONTRIBUTING.md Doc improvements (#479) 2017-04-23 14:54:54 -04:00
deploy_client.yml Ansible upgrade 6.1 (#14500) 2022-07-30 15:01:24 +03:00
Dockerfile upgrade ansible to 9.1.0 (#14673) 2023-12-08 01:54:08 +03:00
input.yml Ubuntu 22.04 support (#14579) 2023-05-17 03:04:23 +03:00
install.sh fix: Fix shellcheck POSIX sh issue and make ansible-lint stricter (#14789) 2025-08-03 07:04:04 -04:00
inventory Refactor to support Ansible 2.8 (#1549) 2019-09-28 08:10:20 +08:00
LICENSE AGPLv3 change (#1351) 2019-03-17 11:19:24 -04:00
logo.png Closes #82, again 2017-02-07 16:35:23 -05:00
main.yml Fix DigitalOcean cloud-init compatibility and deprecation warnings (#14801) 2025-08-03 14:25:47 -04:00
Makefile Docker makefile (#1553) 2019-08-19 15:07:24 +02:00
PULL_REQUEST_TEMPLATE.md Update PULL_REQUEST_TEMPLATE.md 2019-10-07 13:11:33 +02:00
pyproject.toml chore: Conservative dependency updates for Jinja2 security fix (#14792) 2025-08-03 07:45:26 -04:00
README.md Clean up README.md donation options and badges (#14783) 2025-08-03 04:45:16 -04:00
requirements.txt chore: Conservative dependency updates for Jinja2 security fix (#14792) 2025-08-03 07:45:26 -04:00
requirements.yml fix: Fix shellcheck POSIX sh issue and make ansible-lint stricter (#14789) 2025-08-03 07:04:04 -04:00
SECURITY.md Create SECURITY.md (#14669) 2023-12-12 19:17:59 +03:00
server.yml ssh_config: ignore pre-existing SSH keys on client (#14646) 2023-09-27 18:15:35 +03:00
users.yml fix: Fix shellcheck POSIX sh issue and make ansible-lint stricter (#14789) 2025-08-03 07:04:04 -04:00
Vagrantfile Bump ansible from 2.9.20 to 4.4.0 (#14272) 2021-10-31 12:58:35 +03:00

Algo VPN

Twitter

Algo VPN is a set of Ansible scripts that simplify the setup of a personal WireGuard and IPsec VPN. It uses the most secure defaults available and works with common cloud providers. See our release announcement for more information.

Features

  • Supports only IKEv2 with strong crypto (AES-GCM, SHA2, and P-256) for iOS, macOS, and Linux
  • Supports WireGuard for all of the above, in addition to Android and Windows 11
  • Generates .conf files and QR codes for iOS, macOS, Android, and Windows WireGuard clients
  • Generates Apple profiles to auto-configure iOS and macOS devices for IPsec - no client software required
  • Includes a helper script to add and remove users
  • Blocks ads with a local DNS resolver (optional)
  • Sets up limited SSH users for tunneling traffic (optional)
  • Based on current versions of Ubuntu and strongSwan
  • Installs to DigitalOcean, Amazon Lightsail, Amazon EC2, Vultr, Microsoft Azure, Google Compute Engine, Scaleway, OpenStack, CloudStack, Hetzner Cloud, Linode, or your own Ubuntu server (for more advanced users)

Anti-features

  • Does not support legacy cipher suites or protocols like L2TP, IKEv1, or RSA
  • Does not install Tor, OpenVPN, or other risky servers
  • Does not depend on the security of TLS
  • Does not claim to provide anonymity or censorship avoidance
  • Does not claim to protect you from the FSB, MSS, DGSE, or FSM

Deploy the Algo Server

The easiest way to get an Algo server running is to run it on your local system or from Google Cloud Shell and let it set up a new virtual machine in the cloud for you.

  1. Setup an account on a cloud hosting provider. Algo supports DigitalOcean (most user friendly), Amazon Lightsail, Amazon EC2, Vultr, Microsoft Azure, Google Compute Engine, Scaleway, DreamCompute, Linode, or other OpenStack-based cloud hosting, Exoscale or other CloudStack-based cloud hosting, or Hetzner Cloud.

  2. Get a copy of Algo. The Algo scripts will be installed on your local system. There are two ways to get a copy:

    • Download the ZIP file. Unzip the file to create a directory named algo-master containing the Algo scripts.

    • Use git clone to create a directory named algo containing the Algo scripts:

      git clone https://github.com/trailofbits/algo.git
      
  3. Install Algo's core dependencies. Algo requires that Python 3.10 and at least one supporting package are installed on your system.

    • macOS: Big Sur (11.0) and higher includes Python 3 as part of the optional Command Line Developer Tools package. From Terminal run:

      python3 -m pip install --user --upgrade virtualenv
      

      If prompted, install the Command Line Developer Tools and re-run the above command.

      For macOS versions prior to Big Sur, see Deploy from macOS for information on installing Python 3 .

    • Linux: Recent releases of Ubuntu, Debian, and Fedora come with Python 3 already installed. If your Python version is not 3.10, then you will need to use pyenv to install Python 3.10. Make sure your system is up-to-date and install the supporting package(s):

      • Ubuntu and Debian:

        sudo apt install -y --no-install-recommends python3-virtualenv file lookup
        

        On a Raspberry Pi running Ubuntu also install libffi-dev and libssl-dev.

      • Fedora:

        sudo dnf install -y python3-virtualenv
        
    • Windows: Use the Windows Subsystem for Linux (WSL) to create your own copy of Ubuntu running under Windows from which to install and run Algo. See the Windows documentation for more information.

  4. Install Algo's remaining dependencies. You'll need to run these commands from the Algo directory each time you download a new copy of Algo. In a Terminal window cd into the algo-master (ZIP file) or algo (git clone) directory and run:

    python3 -m virtualenv --python="$(command -v python3)" .env &&
      source .env/bin/activate &&
      python3 -m pip install -U pip virtualenv &&
      python3 -m pip install -r requirements.txt
    

    On Fedora first run export TMPDIR=/var/tmp, then add the option --system-site-packages to the first command above (after python3 -m virtualenv). On macOS install the C compiler if prompted.

  5. Set your configuration options. Open the file config.cfg in your favorite text editor. Specify the users you wish to create in the users list. Create a unique user for each device you plan to connect to your VPN.

Note: [IKEv2 Only] If you want to add or delete users later, you must select yes at the Do you want to retain the keys (PKI)? prompt during the server deployment. You should also review the other options before deployment, as changing your mind about them later may require you to deploy a brand new server.

  1. Start the deployment. Return to your terminal. In the Algo directory, run ./algo and follow the instructions. There are several optional features available, none of which are required for a fully functional VPN server. These optional features are described in greater detail in here.

That's it! You will get the message below when the server deployment process completes. Take note of the p12 (user certificate) password and the CA key in case you need them later, they will only be displayed this time.

You can now set up clients to connect to your VPN. Proceed to Configure the VPN Clients below.

    "#                          Congratulations!                            #"
    "#                     Your Algo server is running.                     #"
    "#    Config files and certificates are in the ./configs/ directory.    #"
    "#              Go to https://whoer.net/ after connecting               #"
    "#        and ensure that all your traffic passes through the VPN.      #"
    "#                     Local DNS resolver 172.16.0.1                    #"
    "#        The p12 and SSH keys password for new users is XXXXXXXX       #"
    "#        The CA key password is XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX       #"
    "#      Shell access: ssh -F configs/<server_ip>/ssh_config <hostname>  #"

Configure the VPN Clients

Certificates and configuration files that users will need are placed in the configs directory. Make sure to secure these files since many contain private keys. All files are saved under a subdirectory named with the IP address of your new Algo VPN server.

Apple Devices

WireGuard is used to provide VPN services on Apple devices. Algo generates a WireGuard configuration file, wireguard/<username>.conf, and a QR code, wireguard/<username>.png, for each user defined in config.cfg.

On iOS, install the WireGuard app from the iOS App Store. Then, use the WireGuard app to scan the QR code or AirDrop the configuration file to the device.

On macOS Mojave or later, install the WireGuard app from the Mac App Store. WireGuard will appear in the menu bar once you run the app. Click on the WireGuard icon, choose Import tunnel(s) from file..., then select the appropriate WireGuard configuration file.

On either iOS or macOS, you can enable "Connect on Demand" and/or exclude certain trusted Wi-Fi networks (such as your home or work) by editing the tunnel configuration in the WireGuard app. (Algo can't do this automatically for you.)

Installing WireGuard is a little more complicated on older version of macOS. See Using macOS as a Client with WireGuard.

If you prefer to use the built-in IPSEC VPN on Apple devices, or need "Connect on Demand" or excluded Wi-Fi networks automatically configured, then see Using Apple Devices as a Client with IPSEC.

Android Devices

WireGuard is used to provide VPN services on Android. Install the WireGuard VPN Client. Import the corresponding wireguard/<name>.conf file to your device, then setup a new connection with it. See the Android setup instructions for more detailed walkthrough.

Windows

WireGuard is used to provide VPN services on Windows. Algo generates a WireGuard configuration file, wireguard/<username>.conf, for each user defined in config.cfg.

Install the WireGuard VPN Client. Import the generated wireguard/<username>.conf file to your device, then setup a new connection with it.

Linux WireGuard Clients

WireGuard works great with Linux clients. See this page for an example of how to configure WireGuard on Ubuntu.

Linux strongSwan IPsec Clients (e.g., OpenWRT, Ubuntu Server, etc.)

Please see this page.

OpenWrt Wireguard Clients

Please see this page.

Other Devices

Depending on the platform, you may need one or multiple of the following files.

  • ipsec/manual/cacert.pem: CA Certificate
  • ipsec/manual/.p12: User Certificate and Private Key (in PKCS#12 format)
  • ipsec/manual/.conf: strongSwan client configuration
  • ipsec/manual/.secrets: strongSwan client configuration
  • ipsec/apple/.mobileconfig: Apple Profile
  • wireguard/.conf: WireGuard configuration profile
  • wireguard/.png: WireGuard configuration QR code

Setup an SSH Tunnel

If you turned on the optional SSH tunneling role, then local user accounts will be created for each user in config.cfg and SSH authorized_key files for them will be in the configs directory (user.pem). SSH user accounts do not have shell access, cannot authenticate with a password, and only have limited tunneling options (e.g., ssh -N is required). This ensures that SSH users have the least access required to setup a tunnel and can perform no other actions on the Algo server.

Use the example command below to start an SSH tunnel by replacing <user> and <ip> with your own. Once the tunnel is setup, you can configure a browser or other application to use 127.0.0.1:1080 as a SOCKS proxy to route traffic through the Algo server:

ssh -D 127.0.0.1:1080 -f -q -C -N <user>@algo -i configs/<ip>/ssh-tunnel/<user>.pem -F configs/<ip>/ssh_config

SSH into Algo Server

Your Algo server is configured for key-only SSH access for administrative purposes. Open the Terminal app, cd into the algo-master directory where you originally downloaded Algo, and then use the command listed on the success message:

ssh -F configs/<ip>/ssh_config <hostname>

where <ip> is the IP address of your Algo server. If you find yourself regularly logging into the server then it will be useful to load your Algo ssh key automatically. Add the following snippet to the bottom of ~/.bash_profile to add it to your shell environment permanently:

ssh-add ~/.ssh/algo > /dev/null 2>&1

Alternatively, you can choose to include the generated configuration for any Algo servers created into your SSH config. Edit the file ~/.ssh/config to include this directive at the top:

Include <algodirectory>/configs/*/ssh_config

where <algodirectory> is the directory where you cloned Algo.

Adding or Removing Users

If you chose to save the CA key during the deploy process, then Algo's own scripts can easily add and remove users from the VPN server.

  1. Update the users list in your config.cfg
  2. Open a terminal, cd to the algo directory, and activate the virtual environment with source .env/bin/activate
  3. Run the command: ./algo update-users

After this process completes, the Algo VPN server will contain only the users listed in the config.cfg file.

Additional Documentation

Setup Instructions for Specific Cloud Providers

Install and Deploy from Common Platforms

Setup VPN Clients to Connect to the Server

  • Setup Android clients
  • Setup Linux clients with Ansible
  • Setup Ubuntu clients to use WireGuard
  • Setup Linux clients to use IPsec
  • Setup Apple devices to use IPsec
  • Setup Macs running macOS 10.13 or older to use WireGuard

Advanced Deployment

If you've read all the documentation and have further questions, create a new discussion.

Endorsements

I've been ranting about the sorry state of VPN svcs for so long, probably about time to give a proper talk on the subject. TL;DR: use Algo.

-- Kenn White

Before picking a VPN provider/app, make sure you do some research https://research.csiro.au/ng/wp-content/uploads/sites/106/2016/08/paper-1.pdf ... or consider Algo

-- The Register

Algo is really easy and secure.

-- the grugq

I played around with Algo VPN, a set of scripts that let you set up a VPN in the cloud in very little time, even if you dont know much about development. Ive got to say that I was quite impressed with Trail of Bits approach.

-- Romain Dillet for TechCrunch

If youre uncomfortable shelling out the cash to an anonymous, random VPN provider, this is the best solution.

-- Thorin Klosowski for Lifehacker

Support Algo VPN

PayPal Patreon

All donations support continued development. Thanks!

  • We accept donations via PayPal and Patreon.
  • Use our referral code when you sign up to Digital Ocean for a $10 credit.
  • We also accept and appreciate contributions of new code and bugfixes via Github Pull Requests.

Algo is licensed and distributed under the AGPLv3. If you want to distribute a closed-source modification or service based on Algo, then please consider purchasing an exception . As with the methods above, this will help support continued development.