Rational Unified Process (RUP), Extreme Programming (XP), and Agile methodologies trumpet the benefits of incremental development: software, like everything else, is easier to build in small steps than one big bang. Building an application daily is an iterative development best practice that helps expose bugs more quickly, raises clients' confidence and paces developers. But when daily build is put in place, a structured approach is required to promote the build and make it available to testers and end users. If not, the daily build quickly becomes unmanageable.
This paper describes a build promotion process that we in the Capgemini Technology Consulting group have successfully implemented in a number of large J2EE and .NET software developments. It shows how we have structured our daily build process and structured the build promotion from development to production servers. It also addresses the issues that arise when building daily, and how we managed them.
This paper is organized as follows. The daily build is first introduced, then a discussion of its positive effects on developers, testers and users is followed by a examination of the issues related to promoting and managing daily builds. Finally, the paper shows how difficult it can be to fix bugs and build databases when building daily.
The Daily Build
A build refers to the required steps to create a fully functional and executable version of a software application from its basic components (code, database, configuration files, etc.). A good build process must:
Be automated and repeatable i.e. never require human intervention to complete its work. Automation is essential to reap the benefits of the daily build.
Be portable to different computers and environments
Report the results of its operation to humans that validate the results. Typically this takes the form of e-mails sent to the "build master". Other communication mediums such as Short Message Service (SMS) can also be leveraged.
Perform multiple operations including but not restricted to:
Extracting/tagging most recent copy of the code from source control
Generate a change log file (useful for identifying who introduced the latest defect)
Generate version file identifying date of build and incorporate it in the executable
Building the application and its peripherals (documentation, etc.)
Packaging the application for deployment
Deploying and installing the application to target environments
Running regression tests
Reporting build results, including success/failure but also build metrics
Breaking the build is a capital offense. Developers who break the build must be identified and, at a minimum, fix the build. Authors have devised different incentives to keep the build running: have the developer who broke the build wake up at 5AM daily to smoke test the build, have them wear a funny hat, or have them contribute money to the office pool. Use one that works for your team but never tolerate broken builds.
The reference section provides additional references on the daily build process.
The Positive Impacts of Building Daily
Building your application daily has numerous positive side effects.
Reduced integration work: developers are encouraged to commit their code as soon as it has been tested. This way their code does not get out of sync with the main development branch. Developers are also encouraged to get a fresh copy of the source code tree on a daily basis, or at least before starting each new feature.
Quick spotting of defects: new commits in the code repository are included in the next daily build - which occurs at 5AM every morning - and then smoke tested. Defects are spotted before developers forget about the code changes they made. Furthermore, since the build process also generates a changelog file, it is relatively easy to identify which commit introduced the defect and who is to blame.
Application quality improvement: since the whole software is tested on a daily basis, daily builds reduce the risk that new defects will go unnoticed. Defects are spotted and fixed on a daily basis. This prevents defects from accumulating in the software to the point where a significant effort would have to be expanded in order to fix them all.
Progress visibility enhancements: since the software is built daily, it is easy for the project manager to see which features are there and which ones are not. Clients have visibility into the process and be reassured that milestones will be met.
Team morale booster: seeing the software working provides a good morale boost to the troops. This benefit in itself is enough to justify the adoption of this best practice.
High impact, low maintenance: the set-up of an automated daily build process takes time. Once automated, few minutes are needed for the configuration manager to read the e-mail sent by the build script and verify that the build is good. This process is very low maintenance and does not eat away valuable time from the team's busy schedule.
The Build Promotion Process
As we have seen, building daily goes beyond providing a sanity check on coding software. It brings new features or bug fixes that users and testers need to do their work. Once you start to build daily, requests will flow from all around to get the daily build into their server. Expect this behavior not only from testers but more generally from anyone using your application!
However, it is clear that you cannot push a build every day onto any server. Testers, end users, and support personnel need to have "predictable" behavior from their application, which the daily build cannot provide. At the same time, they would like to have the latest bug fixes and new features that the daily build process exposes. Having a daily build requires you to manage the releases very carefully.
The diagram below summarizes the build promotion process we have put in place.
Figure 1 - The build promotion process
Build promotion refers to the act of transferring a build from one server to another. Promoting a build means copying it to the target server; the build must not be rebuilt but rather simply copied, thus the need to identify and archive builds as they are created. We have defined roles for each server as well as rules to promote a build one server to the next. Table 1 - Build promotion rules -summarizes the build promotion process.
Daily Build Tips, Tricks, and Traps
Continuous builds For large projects or projects where building takes a long time (hours), it is important to build continuously throughout the day. Typically this is done on a dedicated server that reports results and archives builds as it runs. This allows developers to quickly validate if their changes build properly without having to wait for the next daily build or consume their workstation CPU cycles for a long period of time.
Development Team Synchronization When using the daily build process, it is essential to make sure the development team members are synchronized to avoid breaking the build. If developers are not synchronized, it is possible for one developer to check in code that builds properly on his workstation but still breaks the build if someone else checked in incompatible code.
To avoid this problem, you must define during the day certain periods:
Safe check in: during that period anyone can check in code into the version control system. This is the default period during the day
Safe check out: during this period, developers must update their code to match the version control. This period typically occurs just after the daily build has been smoke tested and shown not to be broken and lasts 1-2 hours. No developer is allowed to check in code during that period.
No check in or check out: during this period no check in or check out is allowed. This period begins when the daily build occurs and lasts until the build is smoke tested and shown not to be broken.
Managing change: creating patches We recommend that builds be promoted at regular intervals, thus continuing the rhythm put in place by the daily build. The build promotion process table suggests daily, weekly, monthly and quarterly promotions for daily, QA, staging, and production servers.
Sometimes however, the promotion process cannot be as straightforward as the table suggests. For instance, builds that have made it through the unit, smoke, and light integration tests do break during heavy functional testing. Sometimes the failure is major enough that it keeps testers from going any further until a new build fixes the problem. When that happens, the daily build can often fix the problem but it cannot be deployed because it introduces new features that are not stable enough for QA, staging, or production servers.
The solution is to patch the build in question (branching the code in the version control database from the point where the build in question was tagged), fix the problem, then promote the patch to the server where the issue exists. For example, testers find that a new account opening feature does not work on the QA server. That bug prevents them from exercising all test cases after account opening. The daily build already has that bug fixed but it cannot be promoted to QA server because other features are partially implemented. The account opening feature can be patched rapidly. A new, patched QA build is created and deployed, so as to provide testers with an updated build that lets them continue their tests while the more thorough fix finds its way in the normal build promotion process.
Building Databases Building Java or .NET applications is relatively simple. Code is stored in a central repository and everything is built together at once. Simple and (relatively) easy to automate. Databases, however, are much more complex to build automatically because the build must not only create the schema but also create data. To make matters worse, often data must be migrated from a previous schema, and this is difficult, sometimes impossible, to automate.
Thus, the following must be done to properly promote databases during the build process:
Re-create the database schema from scratch using SQL or other scripts extracted from the source code control system. This implies that all database schema changes are scripted, which is often not the case. But it can be done with a bit of rigor and constraints, especially in environments where database schema can only be modified by a few database administrators.
Recreate the base data from scratch using SQL or other scripts. Again, this is seldom done but can be and should be the default best practice with all teams.
Convert the data in the database from the previous schema. This is rarely ever done as it implies that scripted changes to the database schema also be accompanied with data conversion scripts.
As you can see, it is possible to automate the database creation and make it part of the automated daily build process. However, more often than not, you will find that databases are not built daily and are managed in an ad-hoc fashion.
However, Extreme Programming "Lessons Learned" [1] presents a great solution to DB build issues. It basically says that:
Three databases are created: gold, silver, and bronze.
Database schema changes are promoted from bronze, to silver, to gold then finally to production database (which we refer to as the "master database")
Database data is migrated from production to gold, then to silver, and then bronze databases
All changes to database schema must be scripted. The sum of all scripts, when applied in sequence, allows database migration to be possible
The migration and promotion processes must be automated to ensure that they can be performed often, rapidly, and in a uniform manner.
They too, however, agree that managing promotion/migration scripts is a difficult yet essential tasks that must be followed closely by all involved.
Conclusion
Adopting a daily build process is essential to any software development project. It benefits all team members, from developers to end users. The build script must be customized for each project and fine tuned to map the application's structure and dependencies. Even when available and fully automated in an unbreakable, religious, daily routine, the build process raises issues with end users wishing they could have access as fast as possible to bug fixes and new features. A build release process must be put in place to ensure that appropriate levels of testing and predictability can be ensured for all users. Finally, we have seen how the daily build process creates needs for new developer's procedures, bug fixing shortcuts and non-trivial scripting and controls to ensure database are built along with the code they support.
Special thanks go to Pascal Forget, and Adnaan Sikandar for their review and comments of this paper.
About the Author
Farid Mheir is a Senior Technologist in the Capgemini Technology Consulting group. He has more than 20 years of experience in software development and engineering. He has significant experience in software application design and implementation, with a focus on large web-based solutions, leveraging Rational Unified Process and Unified Modeling Language to translate business requirements into technical specifications.
Mr. Mheir has integrated systems with existing applications and processes, recently using Web services to link remote Microsoft-Unix systems. His experience with the development and operations of 24/7 e-commerce applications has given him an understanding of performance and maintenance requirements. He is also experienced in the delivery of multiple applications via a distributed delivery mode, with teams based in North America and Europe. He can be reached at farid@mheir.com.
When errors occur on a Web site, they should be handled in a way that helps the user to get back on track. Unfortunately, setting up customized error pages in IIS usually requires something many Web developers lack -- access to and familiarity with the Web server's administrative interface. With CustomError for IIS, developers can add error pages, coded by hand or created in their favorite editor, by simply uploading them to a designated directory. No administrator intervention is required.
Building an application can be more than pressing F5. With an increasing
number of quality packages being released, developers for the .NET platform now have options to create a very sophisticated build process. Aaron Junod describes a sample build environment and shows how a number of tools can work together to make reliable, predictable, and value-added builds. [Read This Article][Top]
This short article provides source code for a classic ASP online database functions testing application and shows how to configure and use the tool for either SQL Server or Oracle. [Read This Article][Top]
One of many improvements ASP.NET brings to the development table is in error handling. Adam Tuliper whips up a simple ASP.NET solution for handling those pesky and unexpected post-production errors. [Read This Article][Top]
Mansoor Ahmed Siddiqui explains debugging and tracing and shows how to create custom
trace listeners to help ensure hassle-free development. [Read This Article][Top]
Firing events on a Web server is an easy task. However most of the easy solutions require you to have your own dedicated IIS or SQL Server on the Internet to play with, a privilege not shared by many. In this article, Matthew Muller shows you how to get the same functionality in a shared hosting environment. [Read This Article][Top]
Unlike programming inside a complete VB system, when using ByRef with ASP and COM, a complication arises because ASP's VBScript is not typed, but the component's VB is typed. This article will briefly explain how ByRef can be used with ASP and COM. [Read This Article][Top]
The script in Mark Newlands' article this week handles how errors are displayed and logged. It can capture all values in use at the time (e.g. form, querystring, session,and application level) and records them if you set a Boolean value to do so - displays custom HTML if required. Sends email, logs to database, and/or text file. [Read This Article][Top]
Transact-SQL provides developers with several database error-handling methods. Use these functions to efficiently handle database errors and add an extra level of data validation. This article discusses the @@ERROR, SP_ADDMESSAGE, and RAISERROR functions and provides examples on how to implement them. [Read This Article][Top]
Read what advice members of the 15Seconds Discussion list had to offer on forcing pages to refesh, even when the user hits the back button. [Read This Article][Top]
Mailing List Want to receive email when the next article is published? Just Click Here to sign up.
