Continuous integration of webapps with Buildbot

Monday, May 27, 2013

Buildbot is a powerful and flexible tool used by many sophisticated products like Python, Firefox, Webkit, Google Chrome, Twisted, Node.js and many more.

Buildbot Buildbot, the Continuous Integration Framework Buildbot is primarily used to build software for different architectures covering the process from code compilation to packaging, but it can also be used to do continuous integration of less sophisticated projects where tools and services like Jenkins and Travis-CI are more popular but less flexible.

In this story I introduce Buildbot to run test suites for two products, a sample web application and a sample dependent web project. The web app is a fully functional open source example of a Django pluggable application. While the project is an implementation of the Django tutorial plus a few simple test cases.

Source code for the app and the project as long as the configuration files for Buildbot are available in GitHub.

1. The scenario

The configuration will face the following final scenario:

  1. Test suites for both the app and the project will run under several versions of Django and Python in order to be sure that the app is usable for every supported Django release and that the project might be easily migrated from one release to the next one.

  2. The web application is hosted in a public repository in GitHub while the project is in a private repository in an in-house server. Doing so I cover two ways to send notifications to Buildbot. The webapp will notify Buildbot through a web hook, while the project will do the same through a post-receive hook created in the self hosted repository.

  3. I want Buildbot to build the web project when changes hit project's repository. And given that the project depends on the app, I will tell Buildbot to build the project when the app's repository gets updated too.

  4. Both, the app and the project are Django artifacts but they can be replaced with RoR/RVM and required gems or any other software stack.

2. Concepts and components

Buildbot is a Twisted based distributed service customizable through a Python script. The service is divided into several components that have to be put together to run the desired configuration.

Let's take a quick look at the components and their relationship before addressing the setup.

2.1 Master and Slaves

Buildbot first class citizens are masters and slaves. You can have several of both types, but the typical configuration uses one master and several slaves. In this example I use only one master and three slaves, all them running under the same OS instance.

Think on the master as the service that runs the web, gets notifications from repositories, orchestrates operations with slaves and sends notifications to users.

Think on slaves as environments. When Buildbot is used to build software for different hardware architectures slaves live on appliances powered by such architectures. When it's about testing Django code, slaves may well be virtualenvs running each a different combination of Python/Django.

2.2 Other components

Before the slaves start running the builds Buildbot needs to know what to build and when. Something you do by declaring the following components:

  • A ChangeSource object
  • Filters and Schedulers
  • Build Steps and Build Factories
  • Builders

These components are the core of Buildbot's flexibility. Let's see their functionality and how they are related to each other.

The ChangeSource object is responsible of getting notifications on source code changes from repositories. Source code changes are the usual starting point of the whole process. They lead Buildbot to wake up the next component: Schedulers.

A Scheduler will run a list of Builders. You can associate a Filter to each Scheduler to decide whether the Builder has to run or not. Only those schedulers that pass their filter will trigger their list of Builders.

A Builder will run the sequence of Build Steps that make up the build. A Build Step might be a shell command, a SCM command, a trigger for a scheduler or any other step you might want to run as part of the build process. A set of Build Steps are grouped together under a Build Factory.

Finally, each Builder object is created with a Build Factory and a Slave. The Builder runs in the given Slave by executing the sequence of Build Steps represented by the Build Factory.

There are three pictures in the following section that help to put these concepts in clear.

3. The configuration

To make an easier introduction I will approach the configuration in three steps:

  1. First setup only the web project.
  2. Then add the setup for the web app.
  3. Add setup changes to make Buildbot build the project after successfully building the app.

I first detail the configuration regarding the core components of the master. Read the complete story before doing any actual configuration. The complete Buildbot installation and setup are covered later, including master.cfg files for each step.

3.1 Setup only the web project

Buildbot Configuration Layout, 1/3 Buildbot configuration layout to build a Django web project. First Buildbot will build the web project. The steps to build the project are going to be the same in each slave. The goal is to prove that the project builds successfully under the supported releases of Python and Django.

The configuration uses the following components:

  • One ChangeSource object to receive notifications from project's repository.
  • One Scheduler that triggers three Builders.
  • Three Builders that run all the same BuildFactory (sequence of BuildSteps) in three different Slaves.

Look at the image to see how these components are connected.

3.2 Add the setup for the web app

By adding the web app to the configuration Buildbot will have to distinguish when to build the project and when the app. A situation easily handled by a couple of filters.

Buildbot Configuration Layout, 2/3 Buildbot configuration layout to build a Django web project and a Django web app. The new setup adds:

  • One Filter to check whether code changes come from the project repository.
  • One Filter to check whether code changes come from the app repository.
  • One Scheduler that uses the app Filter and triggers app Builders.
  • Three Builders that run all the same BuildFactory (sequence of BuildSteps) to build the app in three different Slaves.
  • Monitor configuration for the change_hook_dialects parameter in order to receive notifications from GitHub.

The important additions here are the two filters and the change_hook_dialect parameter in the status object.

This extra status parameter tells Buildbot to allow incoming notifications from GitHub in the web path change_hook/github.

3.3 Build the project after building the app

So far, with the current configuration, a source code change will make Buildbot either build the project or the app but not both. However, given that the project depends on the app it seems reasonable to check whether a source code change in the app would eventually break the project.

Buildbot Configuration Layout, 3/3 Buildbot configuration layout to build a Django web project and a Django web app. Apps' Builders trigger Projects' Builders through Trigger build steps and Triggerable Schedulers (blue lines). In other words, each of the project's builders could run right after Buildbot runs each of the app's builders.

Say the app builds successfully under Python 3.2 and Django 1.5. Buildbot could then build the project within the same slave to verify whether the project also builds under the same Python/Django conditions without errors.

Buildbot has a special build step called Trigger. Such a build step triggers a special type of scheduler called Triggerable, that in turn runs the list of builders associated with the scheduler.

For the case of this story the Trigger step can be used as the last step of each of the app builders. The trigger step will run a triggerable scheduler that in turn will run the project builder for that very same slave.

4. Doing continuous integration

The app has a function called do_something that is being used by the function do_otherthing in the project. These two functions play the role of the dependency that the project has on the app.

Both functions are covered with test cases:

  • In the app, the test case for do_something checks whether the function returns the integer value 10.
  • In the project, the test case for do_otherthing verifies that the function returns the integer value 11.

Change 'do_something' and its corresponding test case to make the function return a string and to make the test validate that result.

Such a change comes to represent one of the many changes we could face during the development cycle of any application. When pushing the changes to the repository the project won't build.

The Buildbot configuration for this example uses 'master' branches for both the app and the project, but to run a full continuous integration environment you might want to run builds on development branches too, as to be certain that new code and patches don't break anything before it lands in production.

5. The solution

All the files and steps-to-reproduce the complete Buildbot configuration are available here.

You can adapt the configuration to cover other cases, extend the build steps and/or builders to do additional tasks like code checking, build documentation, packaging and deployment. Buildbot is flexible enough as to cover any corner case.

Checkout Buildbot's official documentation or get direct support by asking in the mailing list or in the #buildbot channel in the IRC.


Your comment

Required for comment verification