Launch Plans – Your Ticket to Excellence

McDonald’s has a checklist for making a Big Mac, but surgeons also use checklists so they don’t cut off the wrong limb of a patient.  Checklists sound mundane, but they ensure quality.  In the case of a Big Mac, the checklist makes for a consistent low cost product. In the case of a surgical operation, checklists help to reduce infection rates and eliminate human error.

Memory is transient, especially the biological kind. For software launches checklists make things go smoother at that critical moment when all the new features go live. Consistently getting the small details right separates the professionals from the amatures. My philosophy is to have two written plans when it comes to a software release:

1) Deploy Plan

2) Back Out Plan

 

In a past life on a small team we kept our plans in a release notes folder – eg /release_instructions/v4.2/. Checklists kept the team in sync and made deploys go smoothly. This article covers what types of notes we had in the launch plan for a LAMP based SaaS product.

Deploy Plan:

A deploy plan proves the team knows what it is doing. It is basically a set of steps. It could be a text file, word document, set of issues in Jira, whatever… The deploy plan gives you a chance to rehearse on QA or staging. Have the least familiar team member run through the plan on their own to spot confusing or incomplete sections.

A typical deploy plan has several steps. For example, there might be a database schema change associated with an ETL script that needs to be executed.  A verification step could be included. Along with the code refresh, it is time to apply security patches to the server. The plan looks like this:

************************************************
A) Pre Deploy Steps.
************************************************

1) Make sure tag for 4.2 gold release is in place.
2) Backup database before proceeding!
3) Copy database backup to another server and share where it is with team.
4) Set application into maintenance mode.

************************************************
B) Run system level OS updates on servers.
************************************************
1) Take snapshot of servers in hypervisor layer.
2) On web server and DB server:

$ sudo apt-get update
$ sudo apt-get upgrade

************************************************
C) Deployment instructions for the various database scripts in the 4.2 release.
************************************************

DDL is in the /documentation/databasemap/4.2/ folder.

Perform steps in the following order:

1) Run DDL and SQL updates as root:  

	a) create_4.2_tables.sql
	b) basic_4.2_data_seeding.sql

2) Run data conversion:

	a) Edit /utility/config to point to the right DB.
	b) Run script /utility/4.2/populate_section_tables
	c) Run script /utility/4.2/verify_new_tables
           (should get zero errors)

************************************************
D) Deploy the new code.
************************************************

1) Checkout 4.2 gold release tag.
2) Run unit tests. If they pass, continue.
3) Continue with ./deployment_instructions.txt, eg:
	$ app_deploy_script...

 

The exact steps and what tools are used will vary greatly depending on how the system is setup. This article is not about deployment tools – there are many to choose from. This just gives you an idea of what could be organized before a launch. Make sure to rehearse it at least once to shake out the bugs.

Back Out Plan:

The back out plan is there in case something goes wrong.   What if a last minute bug causes the deploy to fail? What if a library is different on the live server? Well you could wrestle with it and miss your dinner, or you could roll back safely and deal with it in the morning.

Sample back out plan:

1) Restore backed up database:

   a) Drop db
   b) Create new blank db
   c) Restore backup db
   {provide sample commands}

2) Revert to last checkout and re-run deployment script which replaces symbolic links and restarts necessary services:
   $ cd ../{last deployed tag}
   $ ./{run deploy script}

 

This is super simplified and will vary based on your situation. You might be able to revert a snapshot and reset the system clock. You might not have the luxury of dropping and restoring the database either. The point is to make backing out a plan, so in case you need to you can. No sense in working late and making even more mistakes, introducing last minute bugs, or accidentally causing data loss while making a cowboy tweak to the live server.

Taking this further:

Deploy and back out plans will vary considerably depending on the type of software, platform, hosting environment, etc. The example above was based on a pretty simple web application. It was running on just two servers. A command line driven deploy script was used to setup configuration files and remap symbolic links to make the application ‘live’.

A cool trick in the cloud is to spin up new instances, deploy onto those, test as if they are staging servers, then when ready do a DNS cut over. At that point, just spin down the old instances, or leave them hanging around until you are ready to terminate them. Data synchronization is something to pay attention to with this one.

Another trick with virtualization is to take snapshots of the servers so you have an extra safety net. Do that before applying OS updates just in case one goes south.

Some applications and even a few modern frameworks have a built in maintenance mode that can be enabled by an administrator which tells visitors that the site is undergoing maintenance.

When it comes to doing the build and generating packages / build artifacts I recommend Jenkins – http://jenkins-ci.org/ and  Bamboo by Atlassian.  In a past job we had a ‘One Click Deploy to QA’ process setup that any team member could trigger.  Automating deploys is an investment and you’ll have to decide where to strike the balance economically. Once you try it though, you’ll never go back.  An automated build becomes part of the deploy process and removes the element of human error present in the basic approach given above.

 

This entry was posted in Application Development, For New Developers and tagged , . Bookmark the permalink.

Comments are closed.