unfoldthat.com

Chef scripting quick start

This post is about Chef¹. Chef is a big odd but very powerful piece of software. You’ll probably find it extremely useful for your deploy tasks - as soon as you can understand it.

There is something that can extremely help us with learning Chef: Vagrant. Please consult my tutorial on how to install and initialize a Vagrant virtual machine before (or while) proceeding.

What Chef does?

In the core of Chef are recipes. Recipe is a script (in Ruby with a good DSL), and they are mostly used for deploy. For example, you can find already written scripts for installing and setting up nginx, postgres, etc., and write your own for your specific things like cloning your git repository or running python manage.py syncdb. These scripts are better than just shell-scripts²:

1. They are platfrom-independent (on the one hand, you probably use only one platworm anyway, on the other hand, you probably not planning to rewrite shell-scripts if it’s needed).
2. They have a good DSL: to install nginx, you write: package "nginx", not some strange command with switches you’ve never heard about: sudo apt-get install nginx -y -q or something like that.
3. They are able to understand which things you already installed/configured so you won’t rerun commands and restart daemons once again.
4. They can a lot: not only running shell commands, but creating files from templates, working with version control systems, restarting services, making HTTP requests, etc.

As soon as you have your collection of recipes, you can run them on any server. To run it, you should tell Chef a run-list. A run-list consists of several runnable entities: for example, recipes (also roles, which have their own run-lists).

You can run Chef as “solo”, that means you just install Chef on your server, tell it where your recipes are (like an already downloaded package or a package in the web somewhere), and run them. If you want more power, you can use “chef-server”, which is able to deploy servers from scratch into ec2, rackspace and other hosting providers, and manage already deployed servers (they are called nodes).

The simplest thing

You can imagine all the power and happines Chef is going to bring in your life, but let’s start with the simplest task. Let’s assume you and other devs are working on a project and you want them to just use Vagrant and have everything installed by Chef. As soon as you are able to do it, you’re going be able to deploy the same server somewhere in the internet, but it’s a bit more complicated, so let’s start with your first recipe.

So, install vagrant, initialize it (but don’t run it yet) and clone a chef-repo. You want to write your first recipe to do several things:

1. Install needed stuff
2. Write configuration file
3. Run preparation stuff (like creation of a database)
4. Do sanity check on your code (like running a test suite or something)

The code needed won’t be cloned from a git repository, but it will be shared with your virtual machine from your local computer (we’ll add cloning later when we’ll be dealing with deployment).

The first recipe

I’ve created a simple project for this tutorial, you can use it. All it needs is django, as its requirements.txt states. But we’ll need pip for it, and we’ll need setuptools to install pip. So it’s not too simple, let’s write a recipe. We’ll need a cookbook for our project, let’s name it cheftutorial (cookbook is a folder in the chef-repo/cookbooks/ folder). We also need to create a recipes/ directory inside of it (for recipes, obviously).

mkdir chef-repo/cookbooks/cheftutorial/
mkdir chef-repo/cookbooks/cheftutorial/recipes/

Now, a recipe! Create a file chef-repo/cookbooks/cheftutorial/recipes/default.rb (default.rb is something like __init__.py for chef - a file which name you can omit if you want to reference a recipe). We need to write our recipe now. First of all, let’s take a quick look at our possibilies: Resources are things we can do in a recipe. As ToC suggest, there are lots of them. We’ll need to install a package, create a file with specific contents, run a shell command as root and run shell commands as a user. There seems to be resources for all these tasks, namely package, file and execute. Now we’re confident, let’s write it already:

# install pip
# (the least strange way, imho)

package "python-setuptools"

execute "install_pip" do
    command "easy_install pip"
    user "root"
end

# install requirements

execute "install_requirements" do
    cwd "/home/vagrant/"
    user "root"
    command "pip install -r /home/vagrant/cheftutorial/src/requirements.txt"
end

# write a config

file "/home/vagrant/cheftutorial/src/config.py" do
    content "halo = True"
    owner "vagrant"
    group "vagrant"
end

# run preparation

execute "prepare_things" do
    cwd "/home/vagrant/cheftutorial/src/"
    user "vagrant"
    command "python prepare.py"
end

# test

execute "coolproject_hopeitruns" do
    cwd "/home/vagrant/cheftutorial/src/"
    user "vagrant"
    command "python coolproject.py"
end

Let’s add this recipe to Vagrantfile’s runlist and configure a shared folder:

# just sharing a current folder
config.vm.share_folder "cheftutorial", "~/cheftutorial", "."

config.vm.provision :chef_solo do |chef|
  chef.cookbooks_path = "chef-repo/cookbooks"
  chef.add_recipe "cheftutorial"
#   chef.add_role "web"
#
#   # You may also specify custom JSON attributes:
#   chef.json.merge!({ :mysql_password => "foo" })
end

Now we can try running vagrant up. If its run ends with this habba dabba, everything is not fine:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

cd /tmp/vagrant-chef
chef-solo -c solo.rb -j dna.json

The output of the command prior to failing is outputted below:

[no output]

Let’s try to find the beginning of a big stack trace:

[default] [Wed, 11 May 2011 22:46:42 -0700] INFO: Ran execute[prepare_things] successfully
: stdout
[default] /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/bin/../lib/chef/mixin/command.rb:184:in `handle_command_failures': stderr
[default] : : stderr
[default] python coolproject.py returned 1, expected 0: stderr
[default]  (: stderr
[default] Chef::Exceptions::Exec: stderr
[default] )
: stderr

That seems to be the error: Chef reported that our “coolproject.py” returned 1 - that means, exited with an exception. Let’s take a look, why:

vagrant ssh

# we're in the virtual machine!

vagrant@vagrantup:~$ cd 
.cache/       cheftutorial/ .ssh/         
vagrant@vagrantup:~$ cd cheftutorial/src/
vagrant@vagrantup:~/cheftutorial/src$ python coolproject.py 
  File "coolproject.py", line 3
    import .config as config
           ^
SyntaxError: invalid syntax

Oh, that seems to be my fault - I assumed there will be a newer version of Python. Let’s just fix it in code: remove the line and just add import config. Now we can re-provision our server with vagrant provision. That’s what is said:

[default] [Wed, 11 May 2011 22:54:04 -0700] INFO: Ran execute[coolproject_hopeitruns] successfully
[Wed, 11 May 2011 22:54:04 -0700] INFO: Chef Run complete in 1.106473 seconds
[Wed, 11 May 2011 22:54:04 -0700] INFO: cleaning the checksum cache
[Wed, 11 May 2011 22:54:04 -0700] INFO: Running report handlers
[Wed, 11 May 2011 22:54:04 -0700] INFO: Report handlers complete
: stdout
[default] : stdout

Oh, that seems cool enough: everything is fine. As you can see from the speed of running Chef, it didn’t reinstall pip and setuptools from scratch.

Enhancement

Did you notice these ugly hard-coded vagrant specific paths in our recipes? I did, too. That’s a common problem: every server has its own unique (or pretending to be unique) parameters. Chef can handle it: it has node attributes. These are attributes associated with a node (a server on which we deploy things using chef). For Vagrant server, we can define them right in a Vagrant file. Let’s define app_user, home_folder and source_folder:

# add into "config.vm.provision" block:

chef.json.merge!({ :app_user => "vagrant",
                   :home_folder => "/home/vagrant/",
                   :source_folder => "/home/vagrant/cheftutorial/" })

We can access a node attributes dictionary right in our recipe using node variable (by the way, these :things are used for identification. You can think of them as very immutable strings, at least for now):

# something like that everywhere its needed

execute "install_requirements" do
    cwd node[:home_folder]
    user "root"
    command "pip install -r #{node[:source_folder]}src/requirements.txt"
end

We can run just vagrant provision to redeploy, but we want to be really sure that it will work from scratch, so let’s try vagrant destroy and vagrant up instead.

Okay

Thank you. That’s all you need to know to write simple Chef scripts. Use their “awesome” documentation if you need more (or less?) clarity. I’ll write about using chef server a bit later.

All the code from this post is avaiable as a github repository. I reproduced all steps needed, so you can try to read history.

12th of May, 2011

Please send your comments, ideas, rants and job offers at v.golev@gmail.com.

Made with Nginx, Jekyll, Git, EC2 and Emacs.