Releases: The Last Frontier of Standardization

As software engineers, we strive to better our craft and leave a lasting mark on the organizations we work for. Throughout our careers, we balance two types of knowledge: the combination of business domain and technical stack is our bread and butter. Sometimes we might value the business domain over the technical stack and vice versa as we expand our toolkit. For example, we might work at a financial institution (let’s call it Acme Investment Bank) for a broad domain, but where the tech stack is not bleeding edge. On the other hand, we might roll the dice and work for a firm that has a bleeding-edge tech stack but where the domain is very specific, like a company that sells wine to your pets. 

No matter if you work for a bank or an app that is revolutionizing wine delivery for pets, as an engineer you tend to get better at developing features. Design patterns and approaches learned on one project can transfer into others, while the new challenges add to your skill set. Ironically what does not transfer easily between projects is the process of deploying and releasing the software that you work so hard to build. For most organizations, deployments and releases are team-centric since applications are unique. 

Let’s say you have been leveraging Scala for several years, are a ninja at the Scala build tool and SBT, and can write a Dockerfile in your sleep. The trifecta of language, build, and packaging represent core software development skills.  At some point, you might achieve a self-proclaimed 10x developer status and mumble something like Dinesh on HBO’s Silicon Valley around “sleek and performant Scala code.” Your skills are now very valuable to both Wine for Pets and Acme Investment Bank. 

You take a job at Wine for Pets, which has embraced the “move fast and break things” motto. As a developer, you have direct access to production and no safety net of process/approvals, which would catch breaking changes or give guidance on items that are not your specialty. You will have to learn through trial by fire, and learn multiple domains in which you might not be an expert. This is the full lifecycle developer at its fullest.  

On the flip side, let's say at Acme Investment Bank there is a lot of process/approvals. That safety net is there, but it really hinders you from getting your features out in the wild. Having to navigate “one more JIRA” to line up your changes without the confidence of actually deploying. Telling someone the steps to execute a task when you haven’t done it yourself. You will start to get conflicting feedback from different engineers who are trying to safeguard their own functional areas and new to your application/steps, which adds to the fog. 

Even though you are highly skilled in Scala, both organizations take drastically different approaches. You are effectively thrown to the wolves in both; one being systemic with Wine for Pets and one being subjective, passing all sorts of human-centric approvals at Acme Investment Bank. This is the release learning curve. 

In both cases, even though the development skills stayed the same, the release and deployment skills were totally different. When I talk to developers at different firms, I get N+1 responses on how a release occurs. There might be similarities in the code base around code quality and accepted design patterns, but everyone releases differently. 

The brunt of effort for software engineers embracing design patterns and helping industry standards flourish in the code base seems to fall short during a release cycle. Finding the minimum acceptable path for the risk tolerance of the organization is sometimes required to meet deadlines. As engineers we are under immense pressure to deliver, and the number of items “shifting left” in the name of engineering efficiency is increasing the burden. The learning curve has been growing ever sharper, but there are ways for us to soften it. 

For most organizations, when you start a project you are between deployments/releases. There is a delay in learning the process because deployments are usually not a daily activity. In academia you will learn design patterns but it wouldn’t be until you enter a professional setting that you learn what deploying/releasing entails. Most organizations deploy differently so it takes time to hone in on different approaches. 

One acronym that DevOps organizations try to embody is CAMS from John Willis: Culture, Automation, Measurement, and Sharing. Puppet’s State of DevOps Report 2019 shows we still struggle with the sharing. Objectively trying to measure sharing or disseminating information, even at the report’s highest level of maturity firms, is still a challenge. 

Standardization starts with sharing. This could mean sharing details about deployments with candidates or people thinking about joining the team. During almost every development project that I have been on, the deployment process isn’t discussed during the interview. This shows the importance of the team documenting their approach and creating internal/external documentation on what works and what does not. 

Moving towards platforms that are more ubiquitous is one approach organizations are taking. Kubernetes is seen as the great equalizer in platform development since, organization-to-organization, Kubernetes as a platform is fairly similar. Even Kubernetes, though, has lots of ways to deploy, e.g Helm, Kustomize, KPT, Skaffold, Tekton, etc. Typically for Kubernetes-based roles, you can reverse engineer the deployment approach with specific tools in the job or project description. Lots of the tooling and methodology are moving towards pipeline based development. 

One way of systematically recording what you need to do to get your features out is by using a pipeline. Pipelines come in many shapes and forms. Some might be highly automated, while others complete and have to wait for a human to kick off another one. 

Back to sharing, you don’t just share your release pipelines for security reasons. Even scrubbing for passwords and secrets, you can still give an attacker a decent amount of topology information. With infrastructure moving toward more ubiquitous platforms, e.g. Kubernetes or public cloud infrastructure, sharing at least your approach in your pipeline is not a security concern. You could also share your pipelines with other teams in the organization. Collective learning is important, and automating the lowest common denominator is a goal of Continuous Delivery. 
Continuous Delivery is the science and methodology of eliminating the release learning curve by automating your pipelines. Similar to design patterns, there are patterns that Continuous Delivery helps to enable such as a Canary Deployments. A big goal of Continuous Delivery is to systematically lay out all of the confidence-building steps needed for a successful deployment or roll-back. All of the automation goals you put into your pipelines are goals of engineering efficiency. 

Back to the Scala trifecta of the language, a build tool SBT, and packaging with a Dockerfile; even with all that knowledge, getting your ideas into the hands of users is sometimes out of your control. The goal of engineering efficiency is to reduce the toil it takes to get your ideas into the hands of users.

 
Engineering efficiency pushes the DevOps trifecta of people, processes, and technology to examine how to organizationally reduce the learning curve. It’s used not only for building confidence in the process but also building the engineer’s confidence that he or she can successfully estimate a process outside of their control. 

Technology platforms come and go, and this is why there are so many approaches to releases. Most of the time you have to work with multiple platforms that require multiple approaches. Sharing approaches upfront and not delaying or avoiding the learning curve with someone new can move us toward standardization. Common approaches such as rolling, blue/green, and canary releases are matriculating into many technology stacks. 

Better understanding and estimation are beneficial to both the business and engineering organizations. Focusing on features that earn value instead of fighting to pay off technical debt with something you did not create e.g the deployment bureaucracy machine is where the industry is headed.