How did we get here?

It is 2am on a Saturday morning, your dinner is cold because you have been on a bridge call with the Operations and Engineering teams (now jointly called DevOps) since 7pm trying to roll back the changes applied in the latest update. There are a number of items that need to be rolled back to get to the previously working state but because there is no clear understanding of all the changes that occurred, no one is comfortable rolling back and now you are waiting on a new build to roll forward.

This could all have been avoided if your initial suggestion about using a Canary release strategy were adopted, but no one was willing to commit the resources needed to write the Jenkins code to manage a Canary release. So as you try to listen to a circularly mind-numbing conversation on the bridge your mind drifts in to visions of Canaries spinning above your head…

Canaries on the brain