Recognizing When You’ve Got a Problem

Regardless of the market you’re serving, when you’re in the software business, the ultimate mark of success is shipping software that your customers can use. The act of shipping should be a controlled process, ideally invisible to the customer, and (hopefully) surprise-free.

As relatively fresh Paradigm employees, our January 21st 0.64 release demonstrated that the existing release process was no longer adequate to serve the company’s ambitious goals. The process stretched out over two weeks after the completion of development as failures in communication and a clear lack of direction allowed things to spin out of control. Perhaps most frustrating was a feedback loop of tweaks being pushed into the release branch with inadequate notice, requiring the restarting of regression tests for that area of the release, which took long enough for another tweak to be pushed in elsewhere, perpetuating the cycle.

While 0.64 was successful and delivered features we at Paradigm are quite proud of, the effort was stressful and exhausting for all involved - customers, management, developers, and QA. We knew we had to - and could - do better.

Looking down the road, it wasn’t difficult to see that the process would become unsustainable if it wasn’t unsustainable already. Knowing Paradigm’s goals around growth and customer-centricity, it wasn’t hard to see that someday in the near future, there’d be a release that this process would be utterly incapable of delivering.

Free to Solve Problems

Once that was apparent - those of us dissatisfied with the situation realized one of the best things about being a Paradigm employee: We are empowered to solve problems.

To break that down: a number of employees – from different arms of the company, through open and frank communication – came to a shared conclusion that the process was broken and that it needed fixing. They identified the issues with the release process, defined where it should be, and self-organized a team to start the work of getting things to where they needed to be.

And no one got on our cases and told us that we couldn’t. No one complained that we were going outside the boundaries of our departments or job descriptions. Instead, we received encouragement to press ahead with the labor, and all the frank and honest feedback we could need as we developed our work.

‍

Mapping a Path

We knew where we were - and we weren’t proud of it. We had a good idea of what our ideal release process would look like. We also knew that we couldn’t move immediately to that new process. So we began with the implementation of three core elements:

1. Defined roles and responsibilities
2. Duty rotations for the roles
3. Checklists to follow

Defined Roles and Responsibilities

Originally, releases were driven organically by the engineers with a stake in the features being released. While this approach worked when Paradigm was a smaller company, the complexities of our first microservice release illustrated the growth limitations of a loosely driven, self-organizing process. It wasn’t hard to see that the company’s move to divide the development of its products into multiple business units would multiply that complexity.

To solve this, we introduced structure to the process by defining specific duties required to facilitate the release process:

• The Facilitator to coordinate the team delivering the release.
• The Manager to coordinate the evaluation of the release candidate.
• Business Unit representatives to serve as subject matter experts on new features being released.

These roles are filled by individuals drawn from all arms of the company, and so the company’s business units continue owning their feature contributions, and we own the delivery of each release to the customer together, as a team.

Duty Rotations

Duty rotations were both a solution to the need to keep all engineers trained on the release process and a recognition that releasing software is significant work.  With every release, each role is filled by a fresh volunteer, and the team from the previous release rotates out to a buddy role to pass on important lessons learned from the previous release, or to back up the team should anything come up. The buddies from the previous release rotate out to focus on other work so that they are rested for their next rotation. Knowledge is not allowed to stagnate within one mind, and we all push one another to improve knowledge.

Checklists

When the processes that drive a release are passed through word of mouth and taught with hands-on experience, process consistency and repeatability are difficult to achieve. As checklists are a commonly recognized way to establish a repeatable set of steps to achieve a goal, we wrote template checklists for the roles driving the release process. Now, rather than relying on our memories to ensure every task of the release process is done, there is an understandable list that describes those steps. If anything changes, or improvements need to be made, then everyone involved is empowered to update the checklist. We hope that in the future, we can then automate portions of these checklists to increase efficiency and consistency.

‍

Iteration and Improvement

These simple techniques have already improved our processes phenomenally. Our most recent releases have been marked with better communication, less stress, and more consistent deliveries. Where we needed two weeks after feature completion to vet 0.64 and get it into customer’s hands, our 0.68 and 0.69 have been able to be vetted and delivered in 3-5 days. In those cases where we’ve found problems that have slipped through into recent releases – our newfound organization has enabled us to get hotfixes to solve our problems in hours rather than days.

And that is something to be proud of.

As exciting as that success is, it is just the first step of a greater journey of process improvement. The challenges ahead of us are just as exciting. Through the process of iteration, we’ll continue honing this process until we reach our goals, and along the way, everyone involved will hone their skills.

If you’re someone who enjoys a challenge – and the opportunity to make your mark upon a growing organization by taking ownership and improving the world around you – you may find Paradigm to be a good fit for you. So check out our careers page and see if there’s a role that strikes your interest.

Article by

Wayne Mansell

Software Development Engineer in Test

‍