* Fix DigitalOcean cloud-init compatibility issue causing SSH timeout on port 4160 This commit addresses the issue described in GitHub issue #14800 where DigitalOcean deployments fail during the "Wait until SSH becomes ready..." step due to cloud-init not processing the write_files directive correctly. ## Problem - DigitalOcean's cloud-init shows "Unhandled non-multipart (text/x-not-multipart) userdata" warning - write_files module gets skipped, leaving SSH on default port 22 instead of port 4160 - Algo deployment times out when trying to connect to port 4160 ## Solution Added proactive detection and remediation to the DigitalOcean role: 1. Check if SSH is listening on the expected port (4160) after droplet creation 2. If not, automatically apply the SSH configuration manually via SSH on port 22 3. Verify SSH is now listening on the correct port before proceeding ## Changes - Added SSH port check with 30-second timeout - Added fallback remediation block that: - Connects via SSH on port 22 to apply Algo's SSH configuration - Backs up the original sshd_config - Applies the correct SSH settings (port 4160, security hardening) - Restarts the SSH service - Verifies the fix worked This ensures DigitalOcean deployments succeed even when cloud-init fails to process the user_data correctly, maintaining backward compatibility and reliability. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Implement cleaner fix for DigitalOcean cloud-init encoding issue This replaces the previous workaround with two targeted fixes that address the root cause of the "Unhandled non-multipart (text/x-not-multipart) userdata" issue that prevents write_files from being processed. ## Root Cause Cloud-init receives user_data as binary/bytes instead of UTF-8 string, causing it to fail parsing and skip the write_files directive that configures SSH on port 4160. ## Cleaner Solutions Implemented ### Fix 1: String Encoding (user_data | string) - Added explicit string conversion to user_data template lookup - Ensures DigitalOcean API receives proper UTF-8 string, not bytes - Minimal change with maximum compatibility ### Fix 2: Use runcmd Instead of write_files - Replaced write_files approach with runcmd shell commands - Bypasses the cloud-init parsing issue entirely - More reliable as it executes direct shell commands - Includes automatic SSH config backup for safety ## Changes Made - `roles/cloud-digitalocean/tasks/main.yml`: Added | string filter to user_data - `files/cloud-init/base.yml`: Replaced write_files with runcmd approach - Removed complex SSH detection/remediation workaround (no longer needed) ## Benefits - ✅ Fixes root cause instead of working around symptoms - ✅ Much simpler and more maintainable code - ✅ Backward compatible - no API changes required - ✅ Handles both potential failure modes (encoding + parsing) - ✅ All tests pass, linters clean This should resolve DigitalOcean SSH timeout issues while being much cleaner than the previous workaround approach. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix cloud-init header format for DigitalOcean compatibility The space in '# cloud-config' (introduced in PR #14775) breaks cloud-init YAML parsing on DigitalOcean, causing SSH configuration to be skipped. Cloud-init documentation requires '#cloud-config' without a space. Fixes #14800 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Revert to write_files approach for SSH configuration Using write_files is more maintainable and Ansible-native than runcmd. The root cause was the cloud-config header format, not write_files itself. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Ansible deprecation and variable warnings - Replace deprecated network filters with ansible.utils equivalents: - ipaddr → ansible.utils.ipaddr - ipmath → ansible.utils.ipmath - ipv4 → ansible.utils.ipv4 - ipv6 → ansible.utils.ipv6 - next_nth_usable → ansible.utils.next_nth_usable - Fix reserved variable name: no_log → algo_no_log - Fix SSH user groups warning by explicitly specifying groups parameter Addresses deprecation warnings that would become errors after 2024-01-01. All linter checks pass with only cosmetic warnings remaining. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add comprehensive protection for cloud-config header format - Add inline documentation explaining critical #cloud-config format requirement - Exclude files/cloud-init/ from yamllint and ansible-lint to prevent automatic 'fixes' - Create detailed README.md documenting the issue and protection measures - Reference GitHub issue #14800 for future maintainers This prevents regression of the critical cloud-init header format that causes deployment failures when changed from '#cloud-config' to '# cloud-config'. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add test for cloud-init header format to prevent regression This test ensures the cloud-init header remains exactly ''#cloud-config'' without a space. The regression in PR #14775 that added a space broke DigitalOcean deployments by causing cloud-init YAML parsing to fail, resulting in SSH timeouts on port 4160. Co-authored-by: Dan Guido <dguido@users.noreply.github.com> * Refactor SSH config template and fix MOTD task permissions - Use dedicated sshd_config template instead of inline content - Add explicit become: true to MOTD task to fix permissions warning 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix no_log variable references after renaming to algo_no_log Update all remaining references from old 'no_log' variable to 'algo_no_log' in WireGuard, SSH tunneling, and StrongSwan roles. This fixes deployment failures caused by undefined variable references. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Correct YAML indentation in cloud-init template for DigitalOcean The indent filter was not indenting the first line of the sshd_config content, causing invalid YAML structure that cloud-init couldn't parse. This resulted in SSH timeouts during deployment as the port was never changed from 22 to 4160. - Add first=True parameter to indent filter to ensure all lines are indented - Remove extra indentation in base template to prevent double-indentation - Add comprehensive test suite to validate template rendering and prevent regressions Fixes deployment failures where cloud-init would show: "Invalid format at line X: expected <block end>, but found '<scalar>'" 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Dan Guido <dguido@users.noreply.github.com> |
||
---|---|---|
.github | ||
configs | ||
docs | ||
files/cloud-init | ||
library | ||
playbooks | ||
roles | ||
scripts | ||
tests | ||
venvs | ||
.ansible-lint | ||
.dockerignore | ||
.gitignore | ||
.yamllint | ||
algo | ||
algo-docker.sh | ||
algo-showenv.sh | ||
ansible.cfg | ||
CHANGELOG.md | ||
CLAUDE.md | ||
cloud.yml | ||
CODEOWNERS | ||
config.cfg | ||
CONTRIBUTING.md | ||
deploy_client.yml | ||
Dockerfile | ||
input.yml | ||
install.sh | ||
inventory | ||
LICENSE | ||
logo.png | ||
main.yml | ||
Makefile | ||
PULL_REQUEST_TEMPLATE.md | ||
pyproject.toml | ||
README.md | ||
requirements.txt | ||
requirements.yml | ||
SECURITY.md | ||
server.yml | ||
users.yml | ||
Vagrantfile |
Algo VPN
Algo VPN is a set of Ansible scripts that simplify the setup of a personal WireGuard and IPsec VPN. It uses the most secure defaults available and works with common cloud providers. See our release announcement for more information.
Features
- Supports only IKEv2 with strong crypto (AES-GCM, SHA2, and P-256) for iOS, macOS, and Linux
- Supports WireGuard for all of the above, in addition to Android and Windows 11
- Generates .conf files and QR codes for iOS, macOS, Android, and Windows WireGuard clients
- Generates Apple profiles to auto-configure iOS and macOS devices for IPsec - no client software required
- Includes a helper script to add and remove users
- Blocks ads with a local DNS resolver (optional)
- Sets up limited SSH users for tunneling traffic (optional)
- Based on current versions of Ubuntu and strongSwan
- Installs to DigitalOcean, Amazon Lightsail, Amazon EC2, Vultr, Microsoft Azure, Google Compute Engine, Scaleway, OpenStack, CloudStack, Hetzner Cloud, Linode, or your own Ubuntu server (for more advanced users)
Anti-features
- Does not support legacy cipher suites or protocols like L2TP, IKEv1, or RSA
- Does not install Tor, OpenVPN, or other risky servers
- Does not depend on the security of TLS
- Does not claim to provide anonymity or censorship avoidance
- Does not claim to protect you from the FSB, MSS, DGSE, or FSM
Deploy the Algo Server
The easiest way to get an Algo server running is to run it on your local system or from Google Cloud Shell and let it set up a new virtual machine in the cloud for you.
-
Setup an account on a cloud hosting provider. Algo supports DigitalOcean (most user friendly), Amazon Lightsail, Amazon EC2, Vultr, Microsoft Azure, Google Compute Engine, Scaleway, DreamCompute, Linode, or other OpenStack-based cloud hosting, Exoscale or other CloudStack-based cloud hosting, or Hetzner Cloud.
-
Get a copy of Algo. The Algo scripts will be installed on your local system. There are two ways to get a copy:
-
Download the ZIP file. Unzip the file to create a directory named
algo-master
containing the Algo scripts. -
Use
git clone
to create a directory namedalgo
containing the Algo scripts:git clone https://github.com/trailofbits/algo.git
-
-
Install Algo's core dependencies. Algo requires that Python 3.10 and at least one supporting package are installed on your system.
-
macOS: Big Sur (11.0) and higher includes Python 3 as part of the optional Command Line Developer Tools package. From Terminal run:
python3 -m pip install --user --upgrade virtualenv
If prompted, install the Command Line Developer Tools and re-run the above command.
For macOS versions prior to Big Sur, see Deploy from macOS for information on installing Python 3 .
-
Linux: Recent releases of Ubuntu, Debian, and Fedora come with Python 3 already installed. If your Python version is not 3.10, then you will need to use pyenv to install Python 3.10. Make sure your system is up-to-date and install the supporting package(s):
-
Ubuntu and Debian:
sudo apt install -y --no-install-recommends python3-virtualenv file lookup
On a Raspberry Pi running Ubuntu also install
libffi-dev
andlibssl-dev
. -
Fedora:
sudo dnf install -y python3-virtualenv
-
-
Windows: Use the Windows Subsystem for Linux (WSL) to create your own copy of Ubuntu running under Windows from which to install and run Algo. See the Windows documentation for more information.
-
-
Install Algo's remaining dependencies. You'll need to run these commands from the Algo directory each time you download a new copy of Algo. In a Terminal window
cd
into thealgo-master
(ZIP file) oralgo
(git clone
) directory and run:python3 -m virtualenv --python="$(command -v python3)" .env && source .env/bin/activate && python3 -m pip install -U pip virtualenv && python3 -m pip install -r requirements.txt
On Fedora first run
export TMPDIR=/var/tmp
, then add the option--system-site-packages
to the first command above (afterpython3 -m virtualenv
). On macOS install the C compiler if prompted. -
Set your configuration options. Open the file
config.cfg
in your favorite text editor. Specify the users you wish to create in theusers
list. Create a unique user for each device you plan to connect to your VPN.
Note: [IKEv2 Only] If you want to add or delete users later, you must select
yes
at theDo you want to retain the keys (PKI)?
prompt during the server deployment. You should also review the other options before deployment, as changing your mind about them later may require you to deploy a brand new server.
- Start the deployment. Return to your terminal. In the Algo directory, run
./algo
and follow the instructions. There are several optional features available, none of which are required for a fully functional VPN server. These optional features are described in greater detail in here.
That's it! You will get the message below when the server deployment process completes. Take note of the p12 (user certificate) password and the CA key in case you need them later, they will only be displayed this time.
You can now set up clients to connect to your VPN. Proceed to Configure the VPN Clients below.
"# Congratulations! #"
"# Your Algo server is running. #"
"# Config files and certificates are in the ./configs/ directory. #"
"# Go to https://whoer.net/ after connecting #"
"# and ensure that all your traffic passes through the VPN. #"
"# Local DNS resolver 172.16.0.1 #"
"# The p12 and SSH keys password for new users is XXXXXXXX #"
"# The CA key password is XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX #"
"# Shell access: ssh -F configs/<server_ip>/ssh_config <hostname> #"
Configure the VPN Clients
Certificates and configuration files that users will need are placed in the configs
directory. Make sure to secure these files since many contain private keys. All files are saved under a subdirectory named with the IP address of your new Algo VPN server.
Apple Devices
WireGuard is used to provide VPN services on Apple devices. Algo generates a WireGuard configuration file, wireguard/<username>.conf
, and a QR code, wireguard/<username>.png
, for each user defined in config.cfg
.
On iOS, install the WireGuard app from the iOS App Store. Then, use the WireGuard app to scan the QR code or AirDrop the configuration file to the device.
On macOS Mojave or later, install the WireGuard app from the Mac App Store. WireGuard will appear in the menu bar once you run the app. Click on the WireGuard icon, choose Import tunnel(s) from file..., then select the appropriate WireGuard configuration file.
On either iOS or macOS, you can enable "Connect on Demand" and/or exclude certain trusted Wi-Fi networks (such as your home or work) by editing the tunnel configuration in the WireGuard app. (Algo can't do this automatically for you.)
Installing WireGuard is a little more complicated on older version of macOS. See Using macOS as a Client with WireGuard.
If you prefer to use the built-in IPSEC VPN on Apple devices, or need "Connect on Demand" or excluded Wi-Fi networks automatically configured, then see Using Apple Devices as a Client with IPSEC.
Android Devices
WireGuard is used to provide VPN services on Android. Install the WireGuard VPN Client. Import the corresponding wireguard/<name>.conf
file to your device, then setup a new connection with it. See the Android setup instructions for more detailed walkthrough.
Windows
WireGuard is used to provide VPN services on Windows. Algo generates a WireGuard configuration file, wireguard/<username>.conf
, for each user defined in config.cfg
.
Install the WireGuard VPN Client. Import the generated wireguard/<username>.conf
file to your device, then setup a new connection with it.
Linux WireGuard Clients
WireGuard works great with Linux clients. See this page for an example of how to configure WireGuard on Ubuntu.
Linux strongSwan IPsec Clients (e.g., OpenWRT, Ubuntu Server, etc.)
Please see this page.
OpenWrt Wireguard Clients
Please see this page.
Other Devices
Depending on the platform, you may need one or multiple of the following files.
- ipsec/manual/cacert.pem: CA Certificate
- ipsec/manual/.p12: User Certificate and Private Key (in PKCS#12 format)
- ipsec/manual/.conf: strongSwan client configuration
- ipsec/manual/.secrets: strongSwan client configuration
- ipsec/apple/.mobileconfig: Apple Profile
- wireguard/.conf: WireGuard configuration profile
- wireguard/.png: WireGuard configuration QR code
Setup an SSH Tunnel
If you turned on the optional SSH tunneling role, then local user accounts will be created for each user in config.cfg
and SSH authorized_key files for them will be in the configs
directory (user.pem). SSH user accounts do not have shell access, cannot authenticate with a password, and only have limited tunneling options (e.g., ssh -N
is required). This ensures that SSH users have the least access required to setup a tunnel and can perform no other actions on the Algo server.
Use the example command below to start an SSH tunnel by replacing <user>
and <ip>
with your own. Once the tunnel is setup, you can configure a browser or other application to use 127.0.0.1:1080 as a SOCKS proxy to route traffic through the Algo server:
ssh -D 127.0.0.1:1080 -f -q -C -N <user>@algo -i configs/<ip>/ssh-tunnel/<user>.pem -F configs/<ip>/ssh_config
SSH into Algo Server
Your Algo server is configured for key-only SSH access for administrative purposes. Open the Terminal app, cd
into the algo-master
directory where you originally downloaded Algo, and then use the command listed on the success message:
ssh -F configs/<ip>/ssh_config <hostname>
where <ip>
is the IP address of your Algo server. If you find yourself regularly logging into the server then it will be useful to load your Algo ssh key automatically. Add the following snippet to the bottom of ~/.bash_profile
to add it to your shell environment permanently:
ssh-add ~/.ssh/algo > /dev/null 2>&1
Alternatively, you can choose to include the generated configuration for any Algo servers created into your SSH config. Edit the file ~/.ssh/config
to include this directive at the top:
Include <algodirectory>/configs/*/ssh_config
where <algodirectory>
is the directory where you cloned Algo.
Adding or Removing Users
If you chose to save the CA key during the deploy process, then Algo's own scripts can easily add and remove users from the VPN server.
- Update the
users
list in yourconfig.cfg
- Open a terminal,
cd
to the algo directory, and activate the virtual environment withsource .env/bin/activate
- Run the command:
./algo update-users
After this process completes, the Algo VPN server will contain only the users listed in the config.cfg
file.
Additional Documentation
- FAQ
- Troubleshooting
- How Algo uses Firewalls
Setup Instructions for Specific Cloud Providers
- Configure Amazon EC2
- Configure Azure
- Configure DigitalOcean
- Configure Google Cloud Platform
- Configure Vultr
- Configure CloudStack
- Configure Hetzner Cloud
Install and Deploy from Common Platforms
- Deploy from macOS
- Deploy from Windows
- Deploy from Google Cloud Shell
- Deploy from a Docker container
Setup VPN Clients to Connect to the Server
- Setup Android clients
- Setup Linux clients with Ansible
- Setup Ubuntu clients to use WireGuard
- Setup Linux clients to use IPsec
- Setup Apple devices to use IPsec
- Setup Macs running macOS 10.13 or older to use WireGuard
Advanced Deployment
- Deploy to your own Ubuntu server, and road warrior setup
- Deploy from Ansible non-interactively
- Deploy onto a cloud server at time of creation with shell script or cloud-init
- Deploy to an unsupported cloud provider
- Deploy to your own FreeBSD server
If you've read all the documentation and have further questions, create a new discussion.
Endorsements
I've been ranting about the sorry state of VPN svcs for so long, probably about time to give a proper talk on the subject. TL;DR: use Algo.
-- Kenn White
Before picking a VPN provider/app, make sure you do some research https://research.csiro.au/ng/wp-content/uploads/sites/106/2016/08/paper-1.pdf ... – or consider Algo
-- The Register
Algo is really easy and secure.
-- the grugq
I played around with Algo VPN, a set of scripts that let you set up a VPN in the cloud in very little time, even if you don’t know much about development. I’ve got to say that I was quite impressed with Trail of Bits’ approach.
-- Romain Dillet for TechCrunch
If you’re uncomfortable shelling out the cash to an anonymous, random VPN provider, this is the best solution.
-- Thorin Klosowski for Lifehacker
Support Algo VPN
All donations support continued development. Thanks!
- We accept donations via PayPal and Patreon.
- Use our referral code when you sign up to Digital Ocean for a $10 credit.
- We also accept and appreciate contributions of new code and bugfixes via Github Pull Requests.
Algo is licensed and distributed under the AGPLv3. If you want to distribute a closed-source modification or service based on Algo, then please consider purchasing an exception . As with the methods above, this will help support continued development.