Sunday, March 24, 2013

Notes on Continuous Delivery - Continuous Integration


This is a continuation from my previous post and overall the series on Continuous Delivery. If you haven't been following, that's okay, every topic is self-contained, so read along. The contents of these posts are taken from the book titled "Continuous Delivery" by Humble and Farley (resources below) overlaid with my own professional experience. In this article, we talk a bit more about Continuous Integration (CI) and some practices developers should follow to ensure a smooth environment.

CI requires that an entire application's build process executes on every commit. This includes compilation, running automated test cases, code minification, packaging, static analysis tools, etc. In general, everything that you do when deploying code to production, should also run at every build. If the build breaks, development stops whatever they were working on and focus on getting the build back to a stable state.

Automation is very important. Devote time to build automation into your CI process from the beginning, you will reap the benefits later. The more automated tests you have and code coverage you have, the better. You should write tests of different types, in order of priority:
  1. Unit tests: fast do not require any external dependencies (eg. database). Use an XUnit tool such as JUnit, PHPUnit, etc.
  2. Integration tests (component tests): slower, require database setup
  3. Acceptance tests: slowest, run some kind of acceptance test software (eg. Fitnesse)

Having a comprehensive test suite will give you peace of mind that your application is behaving the way you expect it to at every commit; think about how important this is when you are refactoring code. I really believe in Refactoring, it's essential in order to keep the code base clean and relevant, yet hard to sell it to your business owner. If you break the application as a result of refactoring, then you can kiss away any other opportunity to do so. A good resource for this is the book: Refactoring to Patterns by Joshua Kerievsky.

Continuous Integration is very important in this process because you can discover issues early. Lots of unit tests can uncover bugs, lots of integration tests can also uncover issues with external or third party dependencies, such as your database or Web Services.

On a personal note: do not refrain from using command line tools or using the command line version of any tool in your CI environment, they are by far the most stable. IDEs and different tools have gotten more sophisticated and user friendly over the years, but all in all, nothing beats the CLI.

Theoretically, you don't require a tool or application to perform CI, it is a process, you can home grow your own. However, it is recommended you have one in place. I have had really good experience with the open source application called Jenkins. It integrates nicely with Maven using the Jenkins Maven plugin. There are others such as  CruiseControl and commercial ones like Atlassian Bamboo, to name a few. The way most Continuous Integration software works is by continuously polling your version control server for changes. When a change is detected, the software runs the commands you have specified in the build job. 

Essential Practices

Here are some simple tips to ensure a smooth CI environment:

1. Don't check in on a broken build

Fix it quick. The check-in following the failure should be that which addresses the problem. Beware of this is a slippery slope.

2. Pause before commit

Before committing, update your code to the latest release and run all of your unit tests. Commits should be painless, do not try to do too much by using commit triggers, such as running static analysis, updating change logs, etc. This will slow down your pace tremendously; promote the habit of checking in frequently.

3. Keep a stable build at all times

Take active responsibility for your actions. Breaking the build is a normal part of  development. A broken build once in a while is good because it validates that the process is actually doing something right. If the builds never break, then you should be suspicious. Sometimes, a defect might catch you by surprise on a Friday afternoon at 5:30 pm as your are getting ready to head out. At this point, you can either: 1) Accept it, take a deep breath, stay and fix it (recommended); or 2) Revert your change. In any case, you don't want to hinder your co-workers. Specially, if you are part of a distant distributed team as they will be trying to work while you are sleeping. Fixing it on the spot is always preferable while the code is fresh in your mind.

The point to get across here is that by getting in the habit of checking in frequently, your CI process will do the heavy lifting for you and proof your code. Checking in frequently will build small incremental changes into your system. If 5:00 pm rolls around, and you are not certain about your code, don't commit your change.  This shouldn't be very risky if you have just small changes. This leads me straight into the fourth point...

4. Anticipate being able to revert your changes

I think this is super important and could potentially be very challenging. The reason this is extremely important is because you will want your production environment to be able to be rolled back at any point in time. Meaning, strive to have working code at every check in. In my experience, this can be very difficult when you are dealing with something like a schema change or a data change. Older code might not work with new schema changes. For instance, for every ALTER, DELETE, INSERT, UPDATE script, you should have the corresponding undo of that change. Establish a naming convention for your scripts; it should be very clear and obvious that as part of reverting your change, you should find and execute a rollback script.

4. Don't comment out tests

This is obviously a bad practice and should be done as an extremely last resort. If there are other developers waiting on a particular piece of code to be there, and they cannot move forward because of this, you may decide to comment out a test and commit your code to get them going. However, you MUST jump back to this as soon as possible. If it was an existing test that used to pass, you have just checked in a bug into the system.

5. TDD

It seems no agile development book is complete without some mention to Test Driven Development. To be honest, I am not really big on it. I don't promote it on any team that I lead or am part of. If you do it, awesome! Keep it up. However, I think developers should be very diligent writing tests and strive for high code coverage. And this should apply to both back-end as well as front-end code. Running tests should be part of every commit and are crucial of your are to do any code Refactoring.

I don't want to slip into a tangent here. In theory TDD, sounds like a good idea, but in practice extremely hard to do. Many people claim they do it, and I think they confuse this with writing a unit test for every method. I don't to aggravate the TDD community, so that's all I will say on this topic...  for now.

5. Distributed Version Control

We have already talked about how important this is for CD. Currently, I use Subversion but I am starting to explore Git more and more (Mercurial is another option). I believe that Git contains everything SVN does plus a lot more. With DVCS the ability to work offline and to cooperate across remote teams becomes so much easier. I would NOT dare write a single line without some sort of version control.

Alternatively, you can use the mainline model of version control with Git as though you are using a centralized versioning control system such as Subversion: simply create a master branch, point your CI environment to it, and have every developer push changes directly to it. The contrary is not true, however. When you think about how an open source project works (whether that's your situation or not), you would understand you can only do this with a DVCS such as Git.


The main selling point for Continuous Integration is that you are exercising your process at every commit, a production push becomes just another iteration of this process. This is extremely valuable.

If you are waiting on a build for some reason, use your this spare time wisely: walk around, start a quick chat, check your email, stretch your muscles, etc; the benefits of stepping away from code for a few minutes are wonderful. 

Expect to break builds once in a while, this is normal. You might even break other code that was not directly related to your task. It is still your responsibility to fix it --take active responsibility for the build!

Stay tuned for the next post as we will take a look at implementing a testing strategy!


  1. Humble, Jez and Farley David. Continuous Delivery: A Reliable Software Releases through Build, Test, and Deployment Automation. Addison Wesley. 2011
  2. Extreme Programming Rules -

No comments:

Post a Comment