Set up your remote Rails environment quickly with Chef

The purpose of this article is to show how to prepare a Chef repository in order to automate remote server configuration.
Chef is a tool that has been chosen by many companies (e.g., Amazon or Facebook) because it is very flexible and easy to use. Although, preparing the right configuration (experimenting mostly) is quite time consuming, it really pays off later. I am a Ruby on Rails developer and I hate preparing production environment and because I don’t do it very often I always forget some how to do certain things, for example how to set up Postgres users (it always frustrates me), or looking for a command to run RVM installation. But the worst thing is how time consuming it is. You need to actually sit in front of the computer and participate in the whole configuration process — wait for one action to finish in order to start the next one. With Chef you can provision the remote server automatically. Chef greatly improves efficiency — just enter few lines into the console, sit back and enjoy. After couple of minutes everything should be ready for you to use.

Preparing local environment

Before we start creation first Chef repository that will automate the whole process of server configuration you should prepare your local machine.

Prerequisites:

Ruby, Ruby Version Manager and Gemset

I suppose you already have some Ruby Version Manager (RVM or rbenv most likely) and Ruby so there is no point in explaining how to install it.
If you don’t have it just visit http://rvm.io to get the instructions on how to install RVM or http://rbenv.org/ to get instructions for rbenv. I am using RVM so this will be my ruby version manager of choice for this article.
The Ruby version I am using is Ruby 2.1.1 (the latest at the time of writing this article). It would be a very good idea to create a separate gemset for our Chef configuration. In RVM you can do this by typing $: rvm gemset create chef_article.
To use newly created gemset type $: rvm gemset use chef_article.
This will separate our Chef gems from other gems, which can save us a lot of troubles later.
If everything went right you should have RVM, Ruby and new gemset on your local machine. If you type $: rvm gemset list you should see your new gemset on the list. It will be marked with an asterisk (*) if it is currently used.
One last thing we will need is a bundler gem.
You can easily install it by typing $: gem install bundler.

Knife, Chef and Berkshelf gems

In the previous section we prepared our workbench. Before we start creating our Chef repo, however, we will need some tools.

First you will need to create the Gemfile, where the Ruby version, gemset that we want to use (so when we enter the directory with chef repo it is automatically switched to our Chef gemset) and all the needed gems will be specified.
To do so let’s first create directory for the repo. You can name it anyway you want.

$: mkdir chef_repo && cd $_

Next with the help of the bundler gem initialize a Gemfile.

$: bundle init

Open the Gemfile with you favorite editor and fill it with:

#ruby=ruby-2.1.1
#ruby-gemset=chef_article
source 'https://rubygems.org'  

The first two lines will set the proper Ruby version and the gemset for our gems. The last line was already in the Gemfile and it specifies the address where the bundler should look for the gems.

Now it is time to add some gems for Chef that will help you in the repository creation and in provisioning the server. Go to the Gemfile again and add this lines at the bottom of the file:

gem 'knife-solo', '0.3.0'  
gem "chef", "~> 11.10.0"  
gem 'chef-zero', '1.7.2'  
gem "berkshelf", "~> 2.0.14"  

After that in your console run:

$: bundle install

and wait until all the gems are installed. Check with

$: gem list

to see if all the gems have been properly installed.

ssh-key

In order to make it easier to connect with the server let’s create an ssh-key. In your console type:

$: ssh-keygen -t rsa -C "your_email@example.com"

and press Enter and accept all the default options (press Enter). If the ssh key was successfully created you will need to add it to the ssh agent. In your console type:

$: ssh-add ~/.ssh/id_rsa

Now you will have to add your public key to your server so that the server can recognize your private key and establish the connection. The easiest way to do this is to use a tool named ssh-id-copy. If you are linux user you probably have it installed already. If you are on a Mac then you will have to install it manually. You can do this by copying this line of code into your console:

$: curl https://raw.github.com/beautifulcode/ssh-copy-id-for-OSX/master/ssh-copy-id.sh -o /usr/local/bin/ssh-copy-id

and then

$: chmod +x /usr/local/bin/ssh-copy-id

Or if you have bower you can install it with bower.
To copy your key to the server machine type:

$: ssh-copy-id [-i [identity_file]] [user@]machine

for example

$: ssh-copy-id -i ~/.ssh/id_rsa.pub root@0.0.0.0

That’s it. Try connecting to your server by typing

$: ssh root@your_server_ip

If you weren’t asked for the password it means that the key was successfully copied to your server.

Setting up Chef repository

If you didn't spot any errors in the previous steps, all the gems have installed properly and you managed to set the ssh-key let's go and create the first Chef repository.

Chef terminology

Chef uses the kitchen terminology to describe its behavior and structure. Let's quickly go through most important terms.

Recipe

Recipe is the place where the simple component like Ruby, postgresql or nginx is defined.

Cookbook

Cookbook is a collection of recipes. For example you can create a postgresql cookbook. Inside it you might define recipes for postgresql server and postgresql client.

Role

Role is a set of recipes. For example you might define a role with a name Database. This role will combine for example postgres recipe and redis recipe.

Node

This is a single server definition where you will install all the components you need.

Databag

This is a collection of JSON files where you can define objects used by your recipes. For example this might be the list of the users which should be added to your server. Or maybe list of sites added to your nginx configuration.

Creating a Chef Repo

First you need to initialize a Chef repo. Go to your console, make sure that you are in the chefrepo_ directory and execute the following command:

$: knife solo init .

Your console will output these lines while creating the repo structure:

WARNING: No knife configuration file found  
Creating kitchen...  
Creating knife.rb in kitchen...  
Creating cupboards...  
Setting up Berkshelf...  

Let’s see how the Chef repo structure looks like.
Your chefrepo_ should have the following content:

chef_repo:  
    .
    ├── Berksfile
    ├── cookbooks
    ├── data_bags
    ├── environments
    ├── nodes
    ├── roles
    └── site-cookbooks

Chef repo structure

Berksfile

Here you can add cookbooks that you would like to have in your chef repository. This works pretty much like Gemfile in ruby projects. Just add the name and optionally git repository of a cookbook.

cookbooks

This folder will contain all the downloaded cookbooks. Cookbooks in this folder will be overwritten by the downloaded cookbooks each time you provision to your server thus Making changes in cookbooks contained by this folder won’t bring you any good.

data_bags

Here we will add JSON files with definitions for example for users.

nodes

This directory is be the list of your servers containing JSON files with configuration for particular server.

roles

This is where the definition of the different roles that will provision your server are placed.

site-cookbooks

This directory works just like the cookbooks directory with one difference. Cookbooks in this directory aren’t overwritten, so you can change them and configure accordingly to your needs.

Adding cookbooks

To add some cookbooks open Berksfile. It should be populate with

site :opscode  

This is the place from where the cookbooks will be downloaded.
Lets start populating the Berksfile with cookbook names and repositories where these cookbooks can be found. Add the following lines just below site :opscode:

> cookbook 'users', git: 'https://github.com/opscode-cookbooks/users.git'
> cookbook 'sudo', git: 'https://github.com/opscode-cookbooks/sudo.git'
> cookbook 'rvm', git: 'https://github.com/fnichol/chef-rvm.git'
> cookbook 'nodejs', git: 'https://github.com/mdxp/nodejs-cookbook.git'
> cookbook 'postgresql', git: 'https://github.com/hw-cookbooks/postgresql.git'
> cookbook 'redisio', git: 'https://github.com/brianbianco/redisio.git'
> cookbook 'nginx', git: 'https://github.com/opscode-cookbooks/nginx.git'
> cookbook 'imagemagick', git: 'https://github.com/someara/imagemagick.git'

This set will be sufficient enough to set up the Rails environment.
With the cookbooks in place it is time to set up some roles.

Adding roles

Users

Users cookbook adds users to the server, granting them appropriate privileges and so on. We will use it together with Sudo cookbook which will grant sudo privileges to the user.
Users should be defined in the databags_ directory.
Let’s add the first user and set up the first role that will be used to provision our server.

User definition

First create users directory in the databags_ directory.
In the users directory create a file deploy.json and open it in your editor. Paste the following lines into the file:

{
  "id": "deploy",
  "ssh_keys": [""],
  "groups": [ "sysadmin"],
  "shell": "\/bin\/bash",
  "password": "",
  "system_user": true
}

Let’s quickly go through the most important options:
id: is the name of the user sshkeys: here you can pass an array of strings. Each string should represent one ssh key. password: encrypted password. This shouldn’t be stored in plain text. Instead go to your console and type $: openssl passwd -1 "yourpasswordinplaintext". This will generate an encrypted password which can safely be inserted into your recipe. Copy the output of the above command and paste it between the quotes in the _password option.

Users role

Go to the roles directory and create empty file. Name it users.json. This is the file where the users role will be specified.
Each role is defined in a form of a JSON object.
Copy and paste the below definition into your users.json file.

{
    "name": "users",
    "description": "Setting up users",
    "default_attributes": {
        "authorization": {
            "sudo": {
                "groups": ["sysadmin"],
                "users": ["deploy"],
                "passwordless": "false"
            }
        }
    },
    "json_class": "Chef::Role",
    "run_list": [
        "openssl",
        "build-essential",
        "chef-solo-search",
        "users::sysadmins",
        "sudo"
    ],
    "chef_type": "role"
}

We should spend a few minutes to talk about what all these options mean:

name — this will be the name of the role, used later in our node definition.

description — you can shortly describe here what the role does.

defaultattributes — this is where you set values for the attributes of the recipe. This values will be used when there is no value for this attribute in the node definition. In this case we set up _sudo attribute in authorization module, specifying which users and groups are allowed to use sudo, as well as if the sudo command can be used without the password.

jsonclass and cheftype — these options specify the type of the definition. This is important to set in order to distinguish between roles and recipes definitions especially when the names of the recipe and role doubles.

run_list — this option contains an array of strings — names of the recipes and other roles. If there is no name confusion between roles and recipes it is enough if you just write "openssl" or "build-essentials". However, if your repository contains also "openssl" role it would be a good idea to distinguish them by writing "recipe[openssl]" or "role[openssl]”. If you would like to use other than the default cookbook recipe you should specify it by typing the name of the cookbook and the name of the recipe after the double semicolon, for example "users::sysadmins" or "recipe[users::sysadmins]."

That’s it! The first role has been specified. Let’s go and set up another.

rvm, ruby, rails and few other gems

This one is easy. We already added the cookbook for rvm to the Berksfile, and this is enough to install rvm and ruby version of our choice, create a gemset and install Rails and other useful gems.
Again go to the roles directory. While being inside create a new file and name it rvm-ruby.json. Copy and paste below lines

{
    "name": "rvm-ruby",
    "description": "installation of rvm and ruby",
    "json_class": "Chef::Role",
    "run_list": ["rvm::system"],
    "chef_type": "role",
    "default_attributes": {
        "default_ruby": "ruby-2.1.1",
        "global_gems": [
            {"name": "bundler"},
            {"name": "rails"},
            {"name": "pg"},
            {"name": "redis"},
            {"name": "redis_object"},
            {"name": "compass"}
        ]
    }
}

This one is rather self explanatory but let's quickly go through the options.

name, description, cheftype and jsonclass don't need any explanation I presume.

run_list — the only recipe that is used here is "rvm::system" which will install rvm system wide.

defaultattributes — the _defaultruby_ option tells the recipe to install and set 2.1.1 as default Ruby version (it will be automatically installed if it is not already there). It is also possible to create a gemset by adding @ after the Ruby version; e.g. 2.1.1@myrailsapp. The globalgems_ option value is an array of key, value objects. Here we can specify which gems are to be installed. There are many more options you can specify, but this configurations seems to be enough for the purpose of this article.

With this we will have rvm, Ruby version of our choice, Rails and few other gems installed.

postgresql and redis

Time for the next role, i.e. database.
Go to the roles directory and create a new file. Name it database.json and paste the following lines into it:

{
    "name": "database",
    "description": "database installation",
     "json_class": "Chef::Role",
    "run_list": [
        "recipe[postgresql::server]",
        "recipe[postgresql::client]", 
        "recipe[redisio::install]", 
        "recipe[redisio::enable]"
    ],
    "chef_type": "role",
    "default_attributes": {
        "postgresql": {
            "password": {"postgres": ""},
            "config": {"listen_addresses": "localhost"}
        }
    }
}

This role is also pretty easy to understand. There is only one option you have to change.
runlist — This role will use two recipes from the _postgresql cookbook: server recipe and client recipe. We will also need two recipes from the redisio (installs Redis) cookbook; namely install recipe and enable recipe.

defaultattributes — Inside _postgresql object you will find password key. You should generate it with the command you used to generate password for user i.e.$: openssl passwd -1 "yourpasswordinplaintext" and paste the output between the quotes next to the password key. There are other options you can set. To learn more go to the readme of the postgresql cookbook.

nodejs

To install nodejs go to the roles directory and create a new file. Name it nodes.json. Paste the below lines into it.

{
    "name": "nodejs",
    "description": "node installation",
    "json_class": "Chef::Role",
    "run_list": ["nodejs"],
    "chef_type": "role"
}

Nothing fancy here, and I assume that at this point everything should be understandable.
Let's move to the next one.

nginx

Similarly to other roles, start by creating an empty file in the roles directory, and name it nginx.json. Paste the following lines into the file:

{
    "name": "nginx",
    "description": "nginx installation",
    "json_class": "Chef::Role",
    "run_list": [
        "recipe[nginx]", 
        "recipe[nginx::authorized_ips]"
    ],
    "chef_type": "role",
    "default_attributes": {
        "nginx" :{
            "default_root": "/home/deploy/apps",
            "authorized_ips": ["127.0.0.1:3000"]
        }
    }
}

Let's quickly look at this definition.
runlist — Two recipes will be run here. First _nginx recipe will install nginx http server. Then authorizedips_ recipe will add some authorized IPs to the nginx configuration. defaultattribues — _defaultroot_ option will set the path of the directory were the website files should be found. We also added Rails app IP to the authorizedips_ configuration option. Now it is time to go to the last recipe, which will be very simple as well.

imagemagick

The Imagemagick role, similarly to the nodejs role, is very simple. Just go to the roles directory and create a new file. Name it imagemagick.json and paste in the following lines:

{
    "name": "imagemagick",
    "description": "installs imagemagick",
    "json_class": "Chef::Role",
    "run_list": ["recipe[imagemagick]"],
    "chef_type": "role"
}

I think that there is nothing here that needs an explanation. Let's go to the next point.

Remote machine

We have everything we need on our local machine. We have installed rvm, ruby, knife and chef gems as the tools to provision our server. We also added berkshelf as a cookbook manager. Later we created ssh-key and added it to our server. After that we initialized a Chef repository, added few cookbooks, a user in a data bag and in the end we defined few roles that will help us install all the things we need to start our development work.
Now it is time to install it all on the remote machine.

Preparing remote server with knife solo prepare

First we will use the knife gem which is not only able to initialize a Chef repository but also prepare the remote server.
Go to your console, make sure that you are in the chefrepo_ directory and run this command:

$: knife solo prepare root@remote_machine_ip

This will do a couple of things for us.
First, it will log in to you remote server and install Chef. It is smart enough to recognize the remote server OS and adapt the installation process accordingly.
Second, it will create a new file named yourremotemachineip.json_ in the nodes directory on in the chef repository on your local machine. This is where we need to specify which roles should be used to provision the server.
Open the newly created file and paste these lines into it:

{
    "run_list":
    [
        "role[database]", 
        "role[rvm-ruby]", 
        "role[nodejs]", 
        "role[nginx]", 
        "role[users]", 
        "role[imagemagick]"
    ]
}

This will tell the Chef which roles it should use for the server provisioning.

Installing cookbooks on a remote machine with knife solo cook

Now it's time to run the installation. Go to your console, make sure that you are in the chefrepo_ directory and run this command:

$: knife solo cook root@your_remote_machine_ip

That's it! This will run the scripts and install everything on the server. After it finishes try to ssh using newly created user deploy to log in to your server. If everything went OK you will log in to the server and you will have all the modules installed.

I hope I managed to learn you how to work with Chef, and how to set up a simple Chef repository. Chef can do a lot more, there are many great resources on the Internet, and there are many more cookbooks for you to try out. You have a good base with which you can experiment and improve. Have fun!

#chef #rails #automation
Wojciech Krysiak
CTO and co-founder of RST-IT. Technology enthusiast with a passion for coding. Wojtek is a charismatic Team Leader, who never allows anyone in the team to lose energy