Categoria: EN

Ansible: an easy way to start with config management

Many, many years back, I developed a script to help me run commands on many unix machines or network equipment easyly. I called it tabatool (yeah, weird name, don’t ask). It used Tcl/expect, it had a config file with host names, groups, usernames for login and sudo, etc. I eventually stopped working on it and settled on using a mix of parallel-ssh, clusterssh, fabric, and more specific expect scripts when needed. And them later got to config management with Puppet.

But I really miss a good push mode for Puppet sometimes. Some way of saying “take this code I have right here live in front of me, and apply it to that (or these) machines, NOW” and bam, it goes and do it. Not always, but sometimes. Like adding more powerful config management to fabric or to my old script.

Ansible is like that. It is all that I ever wanted my old script to be, and much much more. It is very flexible, powerful, and it tries very hard to KISS as much as possible. It runs over SSH and it needs no agent on the remote machine. Having played with it a little bit already, I believe Ansible is about the easiest way for a sysadmin to start at configuration management. With the plus of having a very good chance of not needing/wanting to leave it for other tools as the journey progresses.

If you want to follow the examples bellow with “hands on”, let’s get installation out of the way: see the docs here. They publish packages for Ubuntu, CentOS/RHEL, and it’s also pretty common running from a source checkout from Github. On Debian and Ubuntu, a straight apt-get install ansible will also work, but the versions on the official repos might be quite old.

Basic start: ad-hoc usage

Say you want to make sure NTP is installed on all your Debian/Ubuntu machines. List their names on a file (it will be your ansible inventory) and run the command below. I’ll call my file production.

ansible -i production all -u morales --ask-pass --sudo \
  --ask-sudo-pass -m apt -a "name=ntp update-cache=yes"

Depending on your machines setup you might not need any or all of the user and password options above. You can set them as default in a configuration file as well. Ansible recommends you use authorized SSH keys to login, but is fine with passwords, sudo, su and all that. The all soon after -i means all hosts of that inventory, -m apt means the ansible module for apt, and you pass parameters to that module with -a.

You can also run an arbitrary command in a similar way:

ansible -i production all -u morales --ask-pass --sudo \
  --ask-sudo-pass -a "rm -f /tmp/sad_unwanted_file"

This uses the command module, which happens to be the default module, so you don’t need to say -m command in this case. There are many many other modules available. Want just to upload a script and run it? Fine, use the script module. Bonus: with the optional creates= parameter it only runs the script if the said file does not exist.

ansible -i production all -u morales --ask-pass --sudo \
  --ask-sudo-pass -m script -a "/tmp/ \ 
  --some-arguments 1234 creates=/etc/blah/file.conf"

Enter configuration management

Ok, that’s cool, you can already dump parallel-ssh in favor of ansible. But the real deal comes next. We can organize several of those module calls together in a file or set of files, called playbooks. Very very shortly, a playbook is an ordered set of plays, and a play is an ordered set of tasks applied to a set of hosts. Playbooks and plays are written in YAML, Yet Another Markup Language, one that’s pretty easy for humans to work with. The tasks are performed by modules like apt and script and so on.

Let’s see that ad-hoc ntp install, in a playbook called ntp.yml:

- hosts: all
  - name: Install NTP
    apt: name=ntp update-cache=yes

You now run it using the ansible-playbook executable instead of ansible:

ansible-playbook -i production -u morales --ask-pass --sudo \
--ask-sudo-pass ntp.yml

Too much work for just installing a lame package. So let’s also set the configuration file and make sure NTP is restarted whenever ansible changes that file:

- name: Install and Configure NTP
 hosts: all
   ntp_server: [,]
 - name: Install NTP
   apt: name=ntp update_cache=yes
 - name: Copy the ntp.conf template file
   template: src=ntp.conf.j2 dest=/etc/ntp.conf
     - restart ntp
 - name: restart ntp
   service: name=ntp state=restarted

Now the play has two tasks and calls a handler to restart NTP. It also sets variables in the beggining of the play (but they could be set on other files), and one of the tasks fills a template of the ntp.conf file, using those variables. The templating language is Jinja 2. Here’s a possible ntp.conf.j2 for use above:

driftfile /var/lib/ntp/drift

{% for i in ntp_server %}
server {{ i }}
{% endfor %}

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict ::1

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

Let’s finish showing some ansible’s output for the code above (assuming the machines didn’t have ntp installed):

 $ ansible-playbook -i production -u morales --ask-pass --sudo \
     --ask-sudo-pass ntp.yml
 PLAY [Install and Configure NTP] ***********************************
 GATHERING FACTS ****************************************************
 ok: [box1]
 ok: [box2]
 TASK: [Install NTP] ************************************************
 changed: [box2]
 changed: [box1]
 TASK: [Copy the ntp.conf template file] ****************************
 changed: [box1]
 changed: [box2]
 NOTIFIED: [restart ntp] ********************************************
 changed: [box2]
 changed: [box1]
 PLAY RECAP *********************************************************
 box1 : ok=4 changed=3 unreachable=0 failed=0 
 box2 : ok=4 changed=3 unreachable=0 failed=0

If you decide to change the ntp_server list, for example, and run it again, here’s what yout get:

 $ ansible-playbook -i production -u morales --ask-pass --sudo \
     --ask-sudo-pass ntp.yml

 PLAY [Install and Configure NTP] ***********************************
 GATHERING FACTS ****************************************************
 ok: [box2]
 ok: [box1]
 TASK: [Install NTP] ************************************************
 ok: [box2]
 ok: [box1]
 TASK: [Copy the ntp.conf template file] ****************************
 changed: [box2]
 changed: [box1]
 NOTIFIED: [restart ntp] ********************************************
 changed: [box1]
 changed: [box2]
 PLAY RECAP *********************************************************
 box1 : ok=4 changed=2 unreachable=0 failed=0 
 box2 : ok=4 changed=2 unreachable=0 failed=0 

It figures out ntp is already installed, but changes the file and restarts the service. And if you run it again without changing anything, it will just report ok=4 changed=0 at the end: nothing needs doing, so nothing is done.

And there you have it. A small piece of your infrastructure codified in a text file, which can be versioned, reviewed, debugged, shared, and… run. It’s also executable documentation, the only kind of documentation that has a real chance of not getting outdated. New guy arrives in the team, and needs to understand how we do X? Read the code.

Beautiful. Welcome to configuration management.

That was just a very tiny scratch on the surface of Ansible

Really tiny. Let me list some others things Ansible can do:

  • The inventory can be organized in groups. You can define variables based on those groups and use them inside plays (think stuff like “this group of servers uses NTP/DNS servers A and B, that other one uses NTP/DNS servers B and C).
  • There are some automatically available variables with facts about the system: things like hostname, FQDN, os version, and much more.
  • You have loops and conditionals.
  • You can “package” your tasks in reusable roles, and them apply them to hosts in several different plays/playbooks.
  • There are many community-made roles on the Ansible Galaxy (like the Forge for Puppet).
  • You can do a dry run before doing it for real.
  • Sensitive vars/data may be encrypted using ansible-vault, so you don’t need to store it in clear text inside your code.
  • You can run commands on your local control machine, having them in same playbook together with other regular remote tasks. You can also prompt yourself for information.
  • You can even wait for reboots.
  • Windows support already exists, modules for it are starting to appear.
  • You can do everything from the command line, but you can pay for support and access to the web interface called Ansible Tower.

The Ansible documentation could maybe be richer, but it’s also pretty easy to follow.

Final Remarks

I guess the main thing I would like to transmit here can be summed up like this: ANY sysadmin (at least *nix ones) can find an immediate use for Ansible, and can start using it without really studying it any more than reading this post, and then move on to automating more and more stuff with it, little by little, in an easy learning/practice curve.

There are other alternatives. I have used a lot of Puppet, and intend to take Salt for a spin next. But most of all I find Ansible KISS attitude really compelling, specially for people starting with this class of tools.

Setting a backup default route for quagga in linux

Sometimes you don’t need to automate something, you just need to find a better setup with your existing tools.

I got some experience with Quagga for the last couple years, and it’s great. It’s seems to be a very very rare event, but I’ve been bitten by a crash in quagga once or twice. And that’s specially bad if that machine receives its default route through quagga and it ends up without any default route.

So I like to set a static backup default route, in case the dynamic one goes away. That’s usually good practice anyway. Just set a static one in linux with larger metric, someone said. But no. Not with Quagga. Quagga does not install a route if there’s already an existing off-quagga route for that destination, even with a different metric. Seems silly, but it’s true.

Workaround: use linux’s multiple routing tables feature. Check the link if you never heard of it. By default, you already have three tables, setup like this:

# ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Quagga usually installs routes in the main table, which is the regular one. The ip rules above tell the kernel that if it does not find a route in main, it should look in the default table.  So all you need is:

ip route add default via x.x.x.x table default

Quagga will now install the dynamic default route in table main, because he didn’t find any default route there. But if that route somehow disappears, the one in the default table will kick in to save the day.

How to change a file using Puppet

One of the first things I was curious about Puppet (or configuration management in general) was “how can I change a file?” (besides the pretty obvious alternatives of copying a static one and im-sure-they-do-support templates).

So, leaving aside solutions that apply to very specific types of files and more esoteric options like those involving manually exec’ing sed and things like that, there are five ways to do it:

File resource, take one

To the simplest problem, the simplest solution. If you want the file to have a static predefined content, you just copy one using the File resource:

file { '/tmp/file01':
  ensure => file,
  owner  => 'root',
  group  => 'root',
  mode   => '0644',
  source => 'puppet:///modules/mymodule/file01',

You also use the file resource to create symlinks and directories, and you can copy dirs and its files recursively too. Check the reference for more info. In the example above, the source attribute references a file inside a puppet module, which is the most common way of doing it. That works out-of-the box when you use modules. But puppet has a internal file server that you can tweak with for more advanced use cases.

File resource, take two

You can also use File to define the file’s content instead of the source:

file { '/tmp/file02':
  ensure  => file,
  content => 'Yeah, I am file02, so what?',

But you usually use that with templates:

$say_hello_to = 'dude'
$myname = 'file03'

file { "/tmp/$myname":
  ensure  => file,
  content => template('mymodule/polite-file.erb'),

That template, stored inside the module mymodule, looks like this:

$ cat mymodule/templates/polite-file.erb 
<% if @say_hello_to -%>
Hello <%= @say_hello_to %>,
<% end -%>

I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.

After applying that manifest, we have a /tmp/file03 with this content:

Hello dude,

I'm file03, on a Ubuntu system, nice to meet you.

Now, where that @operatingsystem variable (replaced here by “Ubuntu”) came from? Well, besides variables defined within your puppet manifest, templates can use values provided by Facter.

File_line resource

If you wanna change just one simple line on an existent file, File_line is your friend:

file_line { 'someline':
  path  => '/etc/hosts',
  line  => ' somehost',
  match => '^192\.168\.0\.1.*',

It adds only that line if it’s not already there. If you supply the optional match parameter, it will replace the line that matches the regex, instead of adding a new one. File_line is not in puppet core tough, but in the widely used puppetlabs/stdlib module.


Ok, let’s start to complicate things a little bit. The ripienaar/concat module does … well … file concatenation:

concat { '/tmp/file04':
  mode  => '0644',
  owner => 'root',
  group => 'root',

concat::fragment { 'file04_fa':
  target  => '/tmp/file04',
  content => "another piece a\n",

concat::fragment { 'file04_fb':
  target  => '/tmp/file04',
  content => "another piece b\n",

concat::fragment { 'file04_f0':
  target  => '/tmp/file04',
  content => "this is the first piece\n\n",
  order   => 01,

The Concat resource is pretty much like a File resource. The other fragments link to it by their target parameter and may also define the order. How is that complicated? Well, it isn’t in fact. Not the code. But puppet’s report log, oh boy, goes wild:

Notice: /Stage[main]/Concat::Setup/File[/var/lib/puppet/concat]/ensure: created
Notice: /Stage[main]/Concat::Setup/File[/var/lib/puppet/concat/bin]/ensure: created
Notice: /Stage[main]/Concat::Setup/File[/var/lib/puppet/concat/bin/]/ensure: defined content as '{md5}256169ee61115a6b717b2844d2ea3128'
Notice: /Stage[main]//Concat[/tmp/file04]/File[/var/lib/puppet/concat/_tmp_file04]/ensure: created
Notice: /Stage[main]//Concat[/tmp/file04]/File[/var/lib/puppet/concat/_tmp_file04/fragments]/ensure: created
Notice: /Stage[main]//Concat[/tmp/file04]/File[/var/lib/puppet/concat/_tmp_file04/fragments.concat]/ensure: created
Notice: /Stage[main]//Concat::Fragment[file04_fa]/File[/var/lib/puppet/concat/_tmp_file04/fragments/10_file04_fa]/ensure: created
Notice: /Stage[main]//Concat[/tmp/file04]/File[/var/lib/puppet/concat/_tmp_file04/fragments.concat.out]/ensure: created
Notice: /Stage[main]//Concat::Fragment[file04_fb]/File[/var/lib/puppet/concat/_tmp_file04/fragments/10_file04_fb]/ensure: created
Notice: /Stage[main]//Concat::Fragment[file04_f0]/File[/var/lib/puppet/concat/_tmp_file04/fragments/01_file04_f0]/ensure: created
Notice: /Stage[main]//Concat[/tmp/file04]/Exec[concat_/tmp/file04]/returns: executed successfully
Notice: /Stage[main]//Concat[/tmp/file04]/Exec[concat_/tmp/file04]: Triggered 'refresh' from 5 events
Notice: /Stage[main]//Concat[/tmp/file04]/File[/tmp/file04]/ensure: defined content as '{md5}7586b5db4d26995af5e7d6eb23f7eb7e'
Notice: Finished catalog run in 0.78 seconds

Concat is great, very handy. But it is also very verbose, makes the report log harder to read. Besides that, it needs to generate the file sections locally on the destination machine in runtime. Doing a –noop run will now throw errors at you because those file sections do not get created when running noop, and everything that depends on the final resulting file will fail as consequence.

When you have concat in the recipe, troubleshooting and testing it can get quite annoying. I use it frequently, but only when I need it.

Bring the bazooka: Augeas

Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files. –

Augeas is way cool, and it’s not exclusive to puppet, not at all. There is a cli interface (augtool) you can use from anywhere. There is also a puppet resource type for it. Here’s an example of using it to define a setting of your fstab:

augeas { 'setfsoptions':
  context => "/files/etc/fstab",
  changes => [
    "set *[spec = '/dev/sda5']/opt[2] noauto",

UPDATE: Aron Roberts pointed out in a comment a new puppet+augeas documentation available here. Nice!

I’ve just defined that the filesystem in /dev/sda5 shouldn’t be automatically mounted. It will do just that. The rest of the fstab line for that file system is unaffected, keeping it unmanaged by puppet. You can’t do that with File_line or anything else show above.

Install augtool and run augtool print /files/etc/fstab to understand how augeas parses that file. Augeas understands the syntax of files by what it calls lenses, and it comes with several of them ready for dealing with a range of common unix configuration files. If you are brave (and need it) enough, you can create the lens you miss by yourself.

But getting the augeas commands right might be tricky, particularly if you have to deal with numbered entries.  I usually test the commands using augtool before getting them into puppet, so I can have a faster debugging cycle.

The sixth way: use ready-for-use modules and don’t care

As I finished this I realized there is a sixth way that was worth mentioning: grab a ready for use module from the Forge that takes your parameters and does the job for you, changing the file you need to change. No need to know how it does that!

Need to change an ini-style file? See puppetlabs/inifile.  Like augeas power? domcleal/augeasproviders uses it to implement several higher level puppet resource types. Windows work? puppetlabs/registry may help.

And much more. At the Forge you find modules to help manage all kinds of stuff: apache, mysql, tomcat, sudo, dhcp, resolv.conf, apt, … and the list goes on and on. Check it out.

service blog start

Well, it’s about time to kick this blog alive.

So this is a blog about automation in the context of system administration. Automation is such a broad term. For example, monitoring is for me a little bit automation. It automates away the need to regularly check things by hand, which is paramount, by the way.

Shell scripting, ssh multiplexers, libraries and tools for automating non-interactive deployments or both text and visual interactive applications are all great tools to have under any sysadmin’s belt, and I’ve been using then for many years. They are still very useful, but recently I added the ultimate automation tool to my belt: a configuration management software. And that’s a role new game, called Infrastructure as Code.

Infrastructure as code is not just about making you type less, tough. It’s about consistency, repeatability, executable documentation, versioning, peer reviewing, testing. Automation is cool, but realizing the benefits from all these things together is what really excites me.

There are quite a few config management tools out there, and I decided to go with Puppet. So far I’m very happy with it, and it will certainly be a frequent topic here.

Update: my friend foscarini pointed out another quite interesting config management alternative.