• Back to all articles

Build and Release Management

So, you want to write software? Don't forget that you'll need to build or package it, test it, fix some stuff, test it again, and ultimately release it... somehow. The "somehow" is the art and science of Build and Release Management.

Let's get some terms straight.

Building software, as any 15-year-old geek knows, is the generic term applied to the compilation or aggregation of sources into a usable utility or application. Sources needn't be compiled (built) in the classic sense, which relies upon a compiler linking files together; modern systems may consist entirely of interpreted source files (such as those for Perl or PHP). Still, putting everything together in a usable format may be considered "building".

Releasing software refers to the process of providing some named (or otherwise uniquely identified) files to others for use. The others may be your department at work, your classmates, or The World.

Managing the release means you know:

• What went into it,
• Where it went,
• Why it went there,
• And how to deal with it when bugs are reported. (They will be.)

Why is any of this important? I mean, if it compiles, SHIP IT! Right?

Wrong.

The larger and more complex the software project becomes, the greater the need for an extremely well-managed Build and Release function.

What characterizes a well-managed Build and Release function?

Several features are present, some quite basic. You may even be doing some of these things on your project. If you're only doing one -- or none at all -- walk away from the keyboard now before you hurt yourself.

Overnight builds

This sounds so basic, but I've seen many really big shops that weren't doing this. Building the software overnight accomplishes two primary objectives:

1. It establishes the code-wise integration of the project.
2. It immediately alerts you to code-wise disintegration.

These may sound like the same thing, but they aren't. You expect to see code-wise integration as the project matures; the overnight build confirms you have it. You do NOT expect to see code-wise disintegration (that old "it built yesterday, but not today" problem), and thus want to know immediately when it is encountered. Engineers should be the only ones who see legitimate code-wise disintegration. They should catch it during their unit tests and fix it prior to committing to the project. By running overnight builds, you will never have more than 24 hours of source code changes to review and/or backout if the build breaks.

Overnight builds also give your QA department something to throw test cases at, and to target for specific bug regressions. If you don't establish overnight builds very early on in your project, your QA team will suffer schedule slippage late in the project. This is almost guaranteed.

Competent Build Numbering

If you're going to build the project every night (and more often, some days), you will need to establish a build numbering system before you do much of anything else, along with some rules for build publication. Build publication rules refer to what builds get published for QA consumption and where. This usually just means a special directory structure on a specific machine. You should determine this up front, start using it, and not change it. This way, throughout the entire project cycle, any interested party can quickly look at any build. Use a sequential numbering scheme that is flexible enough to provide specific, built-in meaning in the number itself. For example:

freepository20110

This build "number" allows for visible identification of its contents. While the exact numbering scheme I use for Freepository isn't important, the fact that I can look at this number and know the build contains:

• Freepository
• Version 2 source
• Internal build # 110

is very useful. Ideally, come up with a numbering scheme that is easily incremented via automated build scripts. The last thing you want to do is remember what build number to use, and then screw things up by accidentally reusing a number or skipping a number. (But hey, I've never done that...)

Automate, automate, automate

Human error suchs. I mean sucjs. Oh shti. See? Human error is an acceptable risk on trivial, performed-once operations such as writing an email message. Human error is NOT an acceptable risk when building and releasing your software. If you can type the steps into the keyboard, you can automate the process. Trust me, it is worth the trouble.

I automate virtually everything. In fact, an extremely complex Perl script is writing this article for me. I'm really asleep...

Which brings me to Perl. I began using it in 1996 and am still learning some neat tricks. This is really the Magic Belt. Using Perl, I have automated tasks that at first seemed completely unapproachable. If you don't like Perl, try Python or even one of the older scripting languages, like sed or awk. Heck, some of my earliest automation (1985) was accomplished using DOS batch files. Just use something.

Auto-Build Numbers

Got a build number embedded in your source? This is pretty common. Open up virtually any application and click "About". You'll see something like "Version 12.3 Build 1234". Those strings came from the source code, and someone probably had to hand-edit them prior to that build, commit them, tag the source, and then re-edit the strings to free the build number. Yuch! Automate that whole nasty process. Write a little script that reads your master build number file, increments and caches it, then opens the source file with the build and version string in it, greps or searches for the string in the file, does a replace, closes the file, commits it, and tags it.

Scheduling

I have encountered numerous problems over the years with cranky AT and WinAT schedulers, so I gave up and wrote my own. I use Perl; use whatever you like, but remember that Perl is extremely portable. It's very likely that the problem you resolve in Perl today will be a problem you'll face later on another project. By doing all your work in Perl, it is likely that you'll be able to reuse your work.

Reuse

Modularize your automation so it can be used over and over and over. Perl modules are an obvious Perl-centric solution. Create and export all your globals in a few well-thought-out modules. I have one module (now about 4 years old) that does NOTHING but figure out what today is and what yesterday was. Sound worthless? Not if you have to automatically roll back a release that begins to exhibit problems once in production. The concept of a build system-wide "Yesterday" is very useful.

Notification

I have my build system email me when it starts and finishes. I log everything, and, at the end, cull out the important stuff and format it into a message. This gets sent to the project team, and the text has a high-level "Build OK" message. There's a link for the details, but keeping the eye-catching message brief and to the point is extremely useful.

Since I know how long the build takes (to within a couple of minutes), I can be reasonably sure something is wrong if I don't receive the "completion" email on time. This allows me to proactively investigate and deal with whatever may have happened. This reduces schedule impact and, ultimately, project costs.

HTML

Web servers are everywhere. Use one as your presentation manager for builds and releases. All of my logs have the capacity to be displayed in HTML. We're not talking about flaming logos here, but rather the newest lowest common denominator: the browser. There's nothing fancy in my build logs, just lots of data that I must present in the most meaningful, cost-effective fashion possible.

Database storage

Once you discover how much data a build generates, you'll begin to consider all the really cool things you can do with that data, or, more properly, metadata. Examples of build metadata that I capture and key on the build number and date are:

• Build number
• Build date
• Build version
• Branch-tag used
• Overnight build (Y/N)
• QA tested (Y/N)
• QA test results (Pass/Fail)
• Location of full logs

I use PostgreSQL and MySQL. Each is freely available and can be quickly set up on a Linux box, and you can import your existing logs, after you massage them a bit. There's a practical limit to the amount of metadata you store. For example, even though you could capture and store things like "Number of binary files included in the build", you have to ask yourself whether the data is real. Just because a piece of information is available, it doesn't automatically become data.

Implementing one or more of the features mentioned above could have an immediate positive impact on your project. Start with the biggest ones, like the overnight build, and move on from there. Getting the framework in place isn't easy, but it only has to be done once.

So get to it.