wallenstein/algo
1
0
Fork 0
mirror of https://github.com/trailofbits/algo.git synced 2025-08-02 10:53:01 +02:00
Code Issues Projects Releases Wiki Activity

Python3 draft

Browse source
This commit is contained in:
Jack Ivanov 2019-08-22 16:54:06 +02:00
parent 13a073ada5
commit 42a4a2aa58
11 changed files with 103 additions and 143 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.travis.yml
View file

@ -1,6 +1,6 @@
---
language: python
python: "2.7"
python: "3.7"
dist: xenial
services:
@ -12,7 +12,7 @@ addons:
- sourceline: 'ppa:ubuntu-lxc/stable'
- sourceline: 'ppa:wireguard/wireguard'
packages: &default_packages
- python-pip
- python3-pip
- lxd
- expect-dev
- debootstrap
@ -22,7 +22,7 @@ addons:
- build-essential
- libssl-dev
- libffi-dev
- python-dev
- python3-dev
- linux-headers-$(uname -r)
- wireguard
- libxml2-utils

10
Dockerfile
View file

@ -1,4 +1,4 @@
FROM python:2-alpine
FROM python:3-alpine
ARG VERSION="git"
ARG PACKAGES="bash libffi openssh-client openssl rsync tini"
@ -16,11 +16,11 @@ RUN mkdir -p /algo && mkdir -p /algo/configs
WORKDIR /algo
COPY requirements.txt .
RUN apk --no-cache add ${BUILD_PACKAGES} && \
python -m pip --no-cache-dir install -U pip && \
python -m pip --no-cache-dir install virtualenv && \
python -m virtualenv env && \
python3 -m pip --no-cache-dir install -U pip && \
python3 -m pip --no-cache-dir install virtualenv && \
python3 -m virtualenv env && \
source env/bin/activate && \
python -m pip --no-cache-dir install -r requirements.txt && \
python3 -m pip --no-cache-dir install -r requirements.txt && \
apk del ${BUILD_PACKAGES}
COPY . .
RUN chmod 0755 /algo/algo-docker.sh

22
README.md
View file

@ -33,12 +33,12 @@ The easiest way to get an Algo server running is to run it on your local machine
2. **[Download Algo](https://github.com/trailofbits/algo/archive/master.zip).** Unzip it in a convenient location on your local machine.
3. **Install Algo's core dependencies.** Open the Terminal. The `python` interpreter you use to deploy Algo must be python2. If you don't know what this means, you're probably fine. `cd` into the `algo-master` directory where you unzipped Algo, then run:
3. **Install Algo's core dependencies.** Open the Terminal. `cd` into the `algo-master` directory where you unzipped Algo, then run:
- macOS:
```bash
$ python -m ensurepip --user
$ python -m pip install --user --upgrade virtualenv
$ python3 -m ensurepip --user
$ python3 -m pip install --user --upgrade virtualenv
```
- Linux (deb-based):
```bash
@ -46,20 +46,20 @@ The easiest way to get an Algo server running is to run it on your local machine
build-essential \
libssl-dev \
libffi-dev \
python-dev \
python-pip \
python-setuptools \
python-virtualenv -y
python3-dev \
python3-pip \
python3-setuptools \
python3-virtualenv -y
```
- Linux (rpm-based): See the pre-installation documentation for [RedHat/CentOS 6.x](docs/deploy-from-redhat-centos6.md) or [Fedora](docs/deploy-from-fedora-workstation.md)
- Windows: See the [Windows documentation](docs/deploy-from-windows.md)
4. **Install Algo's remaining dependencies.** Use the same Terminal window as the previous step and run:
```bash
$ python -m virtualenv --python=`which python2` env &&
$ python3 -m virtualenv --python="$(command -v python3)" env &&
source env/bin/activate &&
python -m pip install -U pip virtualenv &&
python -m pip install -r requirements.txt
python3 -m pip install -U pip virtualenv &&
python3 -m pip install -r requirements.txt
```
On macOS, you may be prompted to install `cc`. You should press accept if so.
@ -177,7 +177,7 @@ where `user` is either `root` or `ubuntu` as listed on the success message, and
_If you chose to save the CA key during the deploy process,_ then Algo's own scripts can easily add and remove users from the VPN server.
1. Update the `users` list in your `config.cfg`
2. Open a terminal, `cd` to the algo directory, and activate the virtual environment with `source env/bin/activate`
2. Open a terminal, `cd` to the algo directory, and activate the virtual environment with `source .env/bin/activate`
3. Run the command: `./algo update-users`
After this process completes, the Algo VPN server will contain only the users listed in the `config.cfg` file.

6
algo-showenv.sh
View file

@ -68,10 +68,10 @@ elif [[ -f LICENSE && ${STAT} ]]; then
fi
# The Python version might be useful to know.
if [[ -x ./env/bin/python ]]; then
./env/bin/python --version 2>&1
if [[ -x ./.env/bin/python3 ]]; then
./.env/bin/python3 --version 2>&1
elif [[ -f ./algo ]]; then
echo "env/bin/python not found: has 'python -m virtualenv ...' been run?"
echo "env/bin/python3 not found: has 'python3 -m virtualenv ...' been run?"
fi
# Just print out all command line arguments, which are expected

51
docs/deploy-from-fedora-workstation.md
View file

@ -7,17 +7,16 @@ These docs were written based on experience on Fedora Workstation 30.
### DNF counterparts of apt packages
The following table lists `apt` packages with their `dnf` counterpart. This is purely informative.
Using `python2-*` in favour of `python3-*` as per [declared dependency](https://github.com/trailofbits/algo#deploy-the-algo-server).
| `apt` | `dnf` |
| ----- | ----- |
| `build-essential` | `make automake gcc gcc-c++ kernel-devel` |
| `libssl-dev` | `openssl-devel` |
| `libffi-dev` | `libffi-devel` |
| `python-dev` | `python2-devel` |
| `python-pip` | `python2-pip` |
| `python-setuptools` | `python2-setuptools` |
| `python-virtualenv` | `python2-virtualenv` |
| `python3-dev` | `python3-devel` |
| `python3-pip` | `python3-pip` |
| `python3-setuptools` | `python3-setuptools` |
| `python3-virtualenv` | `python3-virtualenv` |
### Install requirements
@ -31,22 +30,20 @@ Next, install the required packages:
````
dnf install -y \
ansible \
automake \
gcc \
gcc-c++ \
kernel-devel \
openssl-devel \
libffi-devel \
libselinux-python \
python2-devel \
python2-pip \
python2-setuptools \
python2-virtualenv \
python2-crypto \
python2-pyyaml \
python2-pyOpenSSL \
python2-libselinux \
python3-devel \
python3-pip \
python3-setuptools \
python3-virtualenv \
python3-crypto \
python3-pyyaml \
python3-pyOpenSSL \
python3-libselinux \
make
````
@ -56,7 +53,7 @@ dnf install -y \
[Download](https://github.com/trailofbits/algo/archive/master.zip) or clone:
````
git clone git@github.com:trailofbits/algo.git
git clone https://github.com/trailofbits/algo.git
cd algo
````
@ -68,24 +65,14 @@ We'll assume from this point forward that our working directory is the `algo` ro
Some steps are needed before we can deploy our Algo VPN server.
### Check `pip`
Run `pip -v` and check the python version it is using:
````
$ pip -V
pip 19.0.3 from /usr/lib/python2.7/site-packages (python 2.7)
````
`python 2.7` is what we're looking for.
### Setup virtualenv and install requirements
````
python2 -m virtualenv --system-site-packages env
source env/bin/activate
pip -q install --user -r requirements.txt
````
```
python3 -m virtualenv --python="$(command -v python3)" .env
source .env/bin/activate
python3 -m pip install -U pip virtualenv
python3 -m pip install -r requirements.txt
```
## Configure

85
docs/deploy-from-redhat-centos6.md
View file

@ -5,8 +5,8 @@ Many people prefer RedHat or CentOS 6 (or similar variants like Amazon Linux) fo
## Step 1: Prep for RH/CentOS 6.8/Amazon
```shell
yum -y -q update
yum -y -q install epel-release
yum -y update
yum -y install epel-release
```
Enable any kernel updates:
@ -17,53 +17,64 @@ reboot
## Step 2: Install Ansible and launch Algo
Fix GPG key warnings during Ansible rpm install:
RedHat/CentOS 6.x uses Python 2.6 by default, which is explicitly deprecated and produces many warnings and errors, so we must install a safe, non-invasive 3.6 tool set which has to be expressly enabled (and will not survive login sessions and reboots):
- Install the Software Collections Library (to enable Python 3.6)
```shell
rpm --import https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
yum -y install centos-release-SCL
yum -y install \
openssl-devel \
libffi-devel \
automake \
gcc \
gcc-c++ \
kernel-devel \
rh-python36-python \
rh-python36-python-devel \
rh-python36-python-setuptools \
rh-python36-python-pip \
rh-python36-python-virtualenv \
rh-python36-python-crypto \
rh-python36-PyYAML \
libselinux-python \
python-crypto \
wget \
unzip \
nano
```
Fix GPG key warning during official Software Collections (SCL) package install:
```shell
rpm --import https://raw.githubusercontent.com/sclorg/centos-release-scl/master/centos-release-scl/RPM-GPG-KEY-CentOS-SIG-SCLo
- 3.6 will not be used until explicitly enabled, per login session. Enable 3.6 default for this session (needs re-run between logins & reboots)
```
scl enable rh-python36 bash
```
RedHat/CentOS 6.x uses Python 2.6 by default, which is explicitly deprecated and produces many warnings and errors, so we must install a safe, non-invasive 2.7 tool set which has to be expressly enabled (and will not survive login sessions and reboots):
- We're now defaulted to 3.6. Upgrade required components
```
python3 -m pip install -U pip virtualenv pycrypto setuptools
```
```shell
# Install the Software Collections Library (to enable Python 2.7)
yum -y -q install centos-release-SCL
# 2.7 will not be used until explicitly enabled, per login session
yum -y -q install python27-python-devel python27-python-setuptools python27-python-pip
yum -y -q install openssl-devel libffi-devel automake gcc gcc-c++ kernel-devel wget unzip ansible nano
# Enable 2.7 default for this session (needs re-run between logins & reboots)
# shellcheck disable=SC1091
source /opt/rh/python27/enable
# We're now defaulted to 2.7
# Upgrade pip itself
pip -q install --upgrade pip
# python-devel needed to prevent setup.py crash
pip -q install pycrypto
# pycrypto 2.7.1 needed for latest security patch
pip -q install setuptools --upgrade
# virtualenv to make installing dependencies easier
pip -q install virtualenv
wget -q https://github.com/trailofbits/algo/archive/master.zip
- Download and uzip Algo
```
wget https://github.com/trailofbits/algo/archive/master.zip
unzip master.zip
cd algo-master || echo "No Algo directory found"
```
# Set up a virtualenv and install the local Algo dependencies (must be run from algo-master)
virtualenv env && source env/bin/activate
pip -q install -r requirements.txt
- Set up a virtualenv and install the local Algo dependencies (must be run from algo-master)
```
python3 -m virtualenv --python="$(command -v python3)" .env
source .env/bin/activate
python3 -m pip install -U pip virtualenv
python3 -m pip install -r requirements.txt
```
# Edit the userlist and any other settings you desire
- Edit the userlist and any other settings you desire
```
nano config.cfg
# Now you can run the Algo installer!
```
- Now you can run the Algo installer!
```
./algo
```

2
docs/deploy-from-windows.md
View file

@ -21,7 +21,7 @@ Wait a minute for Windows to install a few things in the background (it will eve
Install additional packages:
```shell
sudo apt-get update && sudo apt-get install git build-essential libssl-dev libffi-dev python-dev python-pip python-setuptools python-virtualenv -y
sudo apt-get update && sudo apt-get install git build-essential libssl-dev libffi-dev python3-dev python3-pip python3-setuptools python3-virtualenv -y
```
Clone the Algo repository:

46
docs/troubleshooting.md
View file

@ -105,25 +105,13 @@ Command /usr/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/p
Storing debug log for failure in /Users/algore/Library/Logs/pip.log
```
You are running an old version of `pip` that cannot download the binary `cryptography` dependency. Upgrade to a new version of `pip` by running `sudo pip install -U pip`.
### Error: "TypeError: must be str, not bytes"
You tried to install Algo and you see many repeated errors referencing `TypeError`, such as `TypeError: '>=' not supported between instances of 'TypeError' and 'int'` and `TypeError: must be str, not bytes`. For example:
```
TASK [Wait until SSH becomes ready...] *****************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: must be str, not bytes
fatal: [localhost -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/nvr61v455qq98vp22k5r5vm40000gn/T/ansible_6sdjysth/ansible_module_wait_for.py\", line 538, in <module>\n main()\n File \"/var/folders/x_/nvr61v455qq98vp22k5r5vm40000gn/T/ansible_6sdjysth/ansible_module_wait_for.py\", line 483, in main\n data += response\nTypeError: must be str, not bytes\n", "module_stdout": "", "msg": "MODULE FAILURE"}
```
You may be trying to run Algo with Python3. Algo uses [Ansible](https://github.com/ansible/ansible) which has issues with Python3, although this situation is improving over time. Try running Algo with Python2 to fix this issue. Open your terminal and `cd` to the directory with Algo, then run: ``virtualenv -p `which python2.7` env && source env/bin/activate && pip install -r requirements.txt``
You are running an old version of `pip` that cannot download the binary `cryptography` dependency. Upgrade to a new version of `pip` by running `sudo python3 -m pip install -U pip`.
### Error: "ansible-playbook: command not found"
You tried to install Algo and you see an error that reads "ansible-playbook: command not found."
You did not finish step 4 in the installation instructions, "[Install Algo's remaining dependencies](https://github.com/trailofbits/algo#deploy-the-algo-server)." Algo depends on [Ansible](https://github.com/ansible/ansible), an automation framework, and this error indicates that you do not have Ansible installed. Ansible is installed by `pip` when you run `python -m pip install -r requirements.txt`. You must complete the installation instructions to run the Algo server deployment process.
You did not finish step 4 in the installation instructions, "[Install Algo's remaining dependencies](https://github.com/trailofbits/algo#deploy-the-algo-server)." Algo depends on [Ansible](https://github.com/ansible/ansible), an automation framework, and this error indicates that you do not have Ansible installed. Ansible is installed by `pip` when you run `python3 -m pip install -r requirements.txt`. You must complete the installation instructions to run the Algo server deployment process.
### Could not fetch URL ... TLSV1_ALERT_PROTOCOL_VERSION
@ -137,9 +125,9 @@ No matching distribution found for SecretStorage<3 (from -r requirements.txt (li
It's time to upgrade your python.
`brew upgrade python2`
`brew upgrade python3`
You can also download python 2.7.x from python.org.
You can also download python 3.7.x from python.org.
### Bad owner or permissions on .ssh
@ -414,32 +402,6 @@ Certain cloud providers (like AWS Lightsail) don't assign an IPv6 address to you
Manually disconnecting and then reconnecting should restore your connection. To solve this, you need to either "force IPv4 connection" if available on your phone, or install an IPv4 APN, which might be available from your carrier tech support. T-mobile's is available [for iOS here under "iOS IPv4/IPv6 fix"](https://www.reddit.com/r/tmobile/wiki/index), and [here is a walkthrough for Android phones](https://www.myopenrouter.com/article/vpn-connections-not-working-t-mobile-heres-how-fix).
### Error: name 'basestring' is not defined
```
TASK [cloud-digitalocean : Creating a droplet...] *******************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NameError: name 'basestring' is not defined
fatal: [localhost]: FAILED! => {"changed": false, "msg": "name 'basestring' is not defined"}
```
If you get something like the above it's likely you're not using a python2 virtualenv.
Ensure running `python2.7` drops you into a python 2 shell (it looks something like this)
```
user@homebook ~ $ python2.7
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
```
Then rerun the dependency installation explicitly using python 2.7
```
python2.7 -m virtualenv --python=`which python2.7` env && source env/bin/activate && python2.7 -m pip install -U pip && python2.7 -m pip install -r requirements.txt
```
### IPsec: Difficulty connecting through router
Some routers treat IPsec connections specially because older versions of IPsec did not work properly through [NAT](https://en.wikipedia.org/wiki/Network_address_translation). If you're having problems connecting to your AlgoVPN through a specific router using IPsec you might need to change some settings on the router.

14
install.sh
View file

@ -27,10 +27,10 @@ installRequirements() {
build-essential \
libssl-dev \
libffi-dev \
python-dev \
python-pip \
python-setuptools \
python-virtualenv \
python3-dev \
python3-pip \
python3-setuptools \
python3-virtualenv \
bind9-host \
jq -y
}
@ -39,11 +39,11 @@ getAlgo() {
[ ! -d "algo" ] && git clone "https://github.com/${REPO_SLUG}" -b "${REPO_BRANCH}" algo
cd algo
python -m virtualenv --python="$(command -v python2)" .venv
python3 -m virtualenv --python="$(command -v python3)" .venv
# shellcheck source=/dev/null
. .venv/bin/activate
python -m pip install -U pip virtualenv
python -m pip install -r requirements.txt
python3 -m pip install -U pip virtualenv
python3 -m pip install -r requirements.txt
}
publicIpFromInterface() {

2
inventory
View file

@ -1,2 +1,2 @@
[local]
localhost ansible_connection=local ansible_python_interpreter=python
localhost ansible_connection=local ansible_python_interpreter=python3

2
main.yml
View file

@ -25,7 +25,7 @@
msg: >
Ansible version is {{ ansible_version.full }}.
You must update the requirements to use this version of Algo.
Try to run python -m pip install -U -r requirements.txt
Try to run python3 -m pip install -U -r requirements.txt
- name: Include prompts playbook
import_playbook: input.yml