Provisioning remote machines using Ansible

Systems provisioning with Ansible

Various ways of provisioning systems have taken root within the IT industry during the last decade. Exponentially growing IT infrastructure required a paradigm shift away from manual intervention towards automating provisioning and its tasks to the point where it has become a straightforward enterprise.

The purpose of Configuration Management

Configuration management systems have taken the repetitiveness out of systems provisioning and administration. Managing more than one server, you will find that you need to deploy very similar configurations on several of those hosts. Some configuration files need to be present on most of them. Most tasks need to be executed in a particular order, without any deviations from the script. The same packages will deployed to multiple hosts, and many more.

configuration management systems have taken the repetitiveness out of systems provisioning and administration. Managing more than one server, you will notice sooner or later that the configuration of most hosts does not vary too much – with the same configuration files deployed to most of them, and the same packages installed as a basis. Some tasks need to be run in a specific order to accomplish their goals, and deviations from that script would lead to inconsistency that configuration management systems avoid.

Even heterogeneous systems can be dealt with appropriately. They will present you with a variety of host-specific variables to be used for conditionally applying changes only to a subset of the entirety of your servers.

In short, provisioning and configuration management systems have improved the productivity of the IT sector significantly. They are part of the backoffice operations and never visible to frontend users.

How Ansible works

Ansible is a configuration management software implemented in Python and licensed under the GPL version 3.0 (GNU Public License). It sequentially runs a predefined set of tasks on a set of hosts and provides abstractions for selectively running different commands on a per-use case basis.

Instead of defining a state you want the machine to be in after execution, you write tasks that bring about that desired state. This makes Ansible an ideal system not only for prototyping but also for more complex systems involving several tens of hosts serving different functions.

Commands are executed by using a “push” model. You will run Ansible on your computer, on a jumphost you ssh into, or through a continuous integration system you trust. Ansible then pushes commands to the remote host.

This requires you to store SSH keys, passwords and other security-relevant credentials on the machine you execute Ansible from. You will want to pay close attention to hardening access and tightening security onto that trusted machine.

Preparations

Install Ansible on your development machine. In the following I assume that you are running Ansible 2.0 or later, however most of the syntax will be compatible with Ansible 1.9.

Preparing your new FreeBSD server

  1. Bootstrap pkg
    # pkg
    
  2. Install Ansible’s requirements

    In addition to installing Ansible on the machine you are running it from, the remote server will have to have python 2.7 (or later) and sudo installed.

    pkg install sudo python27
    
  3. Add a user. adduser will prompt you for a variety of settings that can be configured on a per-user basis. Make sure you invite the user into the wheel group, and set a password.

  4. Enable password authentication for sshd

    You will want to be able to log in to your server using password authentication until you can use Ansible to set up SSH keys. In /etc/ssh/sshd_config, locate the line that contains PasswordAuthentication, outcomment it if necessary, and set it to yes:

    # Change to yes to enable built-in password authentication.
    PasswordAuthentication no
    
  5. Harden ssh

    There are too many ways to list how to go about hardening ssh daemons. The stribika guide gives you a few pointers.

  6. Restart sshd

    To apply the changes in sshd’s configuration file, restart it:

    # service sshd restart
    

Preparing your new Ubuntu server

  1. Add a user.
    adduser will prompt you for a variety of settings that can be configured on a per-user basis.

  2. Add the user to the sudo group.

    # usermod -a -G sudo xmj
    
  3. (optional) Harden ssh

    As in the FreeBSD example, please refer to the stribika guide for pointers.

  4. (optional) Restart sshd

    To apply the changes in sshd’s configuration file, restart it:

    # service sshd restart
    

Preparing Ansible

Now that your servers are prepared, you will want to create a configuration for Ansible. This will include a skeleton Ansible Playbook (site.yml) and the Ansible configuration file (`ansible.cfg). We will start with the latter.

Ansible Configuration File

Let’s start with the Ansible configuration file. My default config usually contains a [default] section applying to all hosts, and a [ssh_connection] section to make SSH connections more robust and slightly easier to work with.

In the [default] section, I generally let Ansible ask for ssh, sudo and vault passwords as they vary by environment. I further bar Ansible from using cowsay, and use sudo to obtain root access. I also specify a path to my SSH key, and enable warnings to see which part of my Ansible playbook could be optimized.

[defaults]
nocows=1
ask_pass=true
ask_sudo_pass=true
ask_vault_pass=true
private_key_file = /path/to/your/.ssh/key
command_warnings = True
hostfile = ./hosts
become_method = sudo

My [ssh_connection] settings are as minimal as possible, and are there to make ssh connections persist for five minutes. I append the following after the [default] section:

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=300s -o ForwardAgent=yes

Ansible playbook

A small playbook may contain a very limited number of lines to make it run in a homogenous environment. The more heterogeneous the environment, the more likely it will be that you need further abstractions. The following playbook instructs Ansible to apply to all hosts configured in hosts. Ansible will use sudo to become a privileged user, and execute a placeholder task

- hosts: all
become: true
become_method: sudo
vars_files:
- vars/ssl_key.yml
tasks:
- name: Placeholder task name
shell: echo Placeholder task

Using Ansible to install packages

Using Ansible to install packages on remote servers is a very convenient way to handle a large number of servers using different operating systems.

Ansible does not abstract away the packaging completely. This gives you an easy way to access OS-specific functions accurately. To deploy to two or more systems powered by different operating systems, use conditionals. The next lines will be included in the site.yml file you have started in the previous section. Please remove the placeholder task we defined, and append the following:

- name: Install Nginx on Ubuntu
apt: name=nginx state=latest
when: ansible_os_family == "Debian"

- name: Install Jenkins for FreeBSD
pkgng: name=nginx state=present
when: ansible_os_family == "FreeBSD"

Securely storing and providing SSL certificates

In recent years, the popularity of privacy and cryptography has risen incredibly. It has long since been the standard for eCommerce transactions, and is becoming increasingly common for accessing websites and email remotely.

We have already learned how to install the nginx webserver in the previous chapter. This chapter focuses on securely storing SSL certificates within ansible’s group_vars through ansible-vault, and how to hook them into nginx. For brevity, we will assume that you have purchased an SSL certificate through other means (or, used LetsEncrypt to obtain one).

First, we will set up the variable that contains your SSL certificate and private key:

$ ansible-vault create vars/ssl_key.yml

Enter a password with enough bits to be safe (I use an alias to generate them, `alias genpw='openssl rand -base64 9'`) twice, and add it to your keystore.

Enter your SSL certificate and key as well as your Certificate Authority’s CA chain:

ssl_certificate: |
-----BEGIN CERTIFICATE-----
<SSL Certificate goes here>
-----END CERTIFICATE-----

ssl_certificate_key: |
-----BEGIN RSA PRIVATE KEY-----
<SSL Private Key goes here>
-----END RSA PRIVATE KEY-----

ssl_ca_certificate: |
-----BEGIN CERTIFICATE-----
<CA Certificate chain goes here>
-----END CERTIFICATE-----

Save and exit the editor. Your SSL certificate and keys stored safely from
prying eyes. However, it is not a good idea to publish it even in encrypted
form without complete access control. You are responsible for your own server’s
safety as well as the data your users transmit to it.

As a next step, we continue with extending the Ansible playbook (site.yml) you started previously.

First, tell the playbook to use the new, vaulted variable file. Between become_method: sudo and tasks:, insert a vars_files: list with - vars/ssl_key.yml as its only item:

become_method: sudo
vars_files:
- vars/ssl_key.yml
tasks:

This allows the following directives to make use of the ssl_certificate,
ssl_certificate_key and ssl_ca_certificate variables you hav defined in the
vaulted variable file.

Continue by adding deployment tasks to the playbook, as items in the tasks: list.

- name: Ensure the Nginx conf.d directory exists.
file:
dest: "/usr/local/etc/nginx/ssl"
mode: 0755
owner: root
state: directory

- name: Ensure the SSL directory exists.
file:
dest: "/usr/local/etc/nginx/ssl"
mode: 0755
owner: root
state: directory

- name: Deploy SSL Certificate
copy:
content: "{{ ssl_certificate }}"
dest: "/usr/local/etc/nginx/ssl/ssl_certificate"
mode: 0644
owner: root

- name: Deploy SSL Certificate Key
copy:
content: "{{ ssl_certificate_key }}"
dest: "/usr/local/etc/nginx/ssl/ssl_certificate_key"
mode: 0600
owner: root

- name: Deploy SSL CA Certificate
copy:
content: "{{ ssl_ca_certificate }}"
dest: "/usr/local/etc/nginx/ssl/ca_cert_chain"
mode: 0644
owner: root

Great, your SSL certificate is stored safely on your computer, and has been
deployed onto your server. Now let’s hook it into Nginx.

Create a file ssl.conf inside a new directory files/, and add a secure SSL
configuration:

ssl_protocols TLSv1.2;
ssl_certificate /usr/local/etc/nginx/ssl/ssl_certificate;
ssl_certificate_key /usr/local/etc/nginx/ssl/ssl_certificate_key;
ssl_trusted_certificate /usr/local/etc/nginx/ssl/ca_cert_chain;
ssl_verify_depth 2;

ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_session_tickets off;

ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ssl_prefer_server_ciphers on;

add_header Strict-Transport-Security max-age=15768000;

Include this SSL configuration in server { listen 443 ssl ; ... } blocks that
contain your site-specific configuration files, and deploy both through
ansible’s copy module through extending your site.yml once more.

- name: Deploy Nginx SSL configuration
copy:
src: ssl.conf
dest: "/usr/local/etc/nginx/conf.d/ssl.conf"
mode: 0755
owner: root

You may now include this secure SSL configuration from within your own Nginx
site definitions, and be able to change it in one place. This will save
critical time should future vulnerabilities be discovered and should ciphers
need to be deactivated.

Running Ansible with Harrow

As mentioned above, Ansible is usually run from your computer, with full control over the setup and easy ways of providing the password to Ansible’s vault.

Due to security concerns when sharing credentials with your colleagues, you may want to use a dedicated Virtual Machine to run Ansible scripts from. This is made convenient with Harrow. At the click of a button, it:

  1. acquires a fresh virtual machine (VM)
  2. clones your repositories onto that machine
  3. runs a script you choose on this machine

Harrow allows you to safely store credentials, and implements role-based access to members of your team.

We will use this feature to securely store the Ansible vault passphrase.

The process of achieving this goal with Harrow and Ansible is as follows:

  1. You store the Ansible vault passphrase as an Environment Secret on Harrow. This makes the passphrase available as an environment variable available in the
    shell executing Ansible.

  2. Create a script on Harrow which

  3. puts the vault passphrase into a file, so that Ansible can pick it up from there
  4. runs Ansible, instructing it to use the passphrase from the file created above.

Once you have signed up and logged into Harrow, adding an Environment Secret is easy: in your project settings, select “Environments”. Click “Edit” next to the environment you want to run your script in. you will be able to add new secrets in the area highlighted here:

Adding a secret in Harrow

Assuming we name our secret ANSIBLE_VAULT_PASSWORD, we implement the script as follows:


#!/bin/bash -eu # Install python and pip sudo apt-get -y install python2.7 python-pip python-dev # Install Ansible from Cheeseshop sudo pip install ansible # Write the Vault passphrase into a temporary file printf "%s\n" "$ANSIBLE_VAULT_PASSWORD" > /tmp/ansible-vault-password # Change into the repository containing the Ansible playbook cd $(find ~/repositories -name site.yml | head -1 | xargs dirname) # Run the playbook and point Ansible to the vault passphrase ansible-playbook site.yml --vault-password-file /tmp/ansible-vault-password # Remove the temporary file just in case rm -f /tmp/ansible-vault-password

Having put this script in place, anyone with access to your Harrow project use it to have Ansible deploy things to production.

Personally, I find that pushing Ansible playbooks to GitHub or BitBucket, then having Harrow automatically run them on every push to the master branch a very convenient way to make continuous integration effortless.


Want to try Harrow.io? Start immediately with a 14 day free trial or learn more

Harrow.io is a platform for Continuous Integration, testing and deployment, built by the team behind Capistrano.
Add powerful automation and collaboration capabilities to your existing tools.

  • Automate any software, in any language.
  • Create self-documenting, repeatable web-based tasks
  • Make them available for your team-mates
  • Trigger tasks automatically based on Git changes and webhooks.
  • Get notified by email, slack, etc.
  • Free for small projects

Test, deploy and collaborate online easily, using tools you already know and love.
Works seamlessly for PHP, Node.js, Ansible, Python, Go, Capistrano and more!

Learn more Start your 14 day free trial

No credit card required – completely free for small or public projects.

Johannes has been contributing to the FreeBSD community in various forms since 2012, runs a software development consultancy and is currently developing a FreeBSD-based All-In-One solution for software development startups. His time away from keyboard is spent on a beach, 60ft under water, behind a wheel, on slopes in the Alps and making friends around the globe.

Start Using Harrow Today!

Harrow is flexible, powerful and can make your team much more efficient.

Start free trial