Obtaining an always green deployment pipeline gets more and more difficult as you scale your development organization. One common approach is to hide changes behind feature branches – where changes can be checked before reaching the mainline. Merge Requests is a normal way to obtain this workflow and we will use the terms interchangeably.

This works, most of the time, but it comes with the risk of creating the wrong culture, one where people are afraid of breaking the deployment pipeline by merging into the mainline. This fear leads to larger, less frequent merges, and longer integration cycles and is one of the main arguments against feature branches in the first place [4] [5].

If we are not careful, then we, implicitly, create a culture where we optimize for Mean Time Between Failure (MTBF) – State of Devops [1] and subsequently DORA [2] teaches us that focusing on improving Mean Time to Recover (MTTR) is a better way to reach high performance.

The key metrics for measuring throughput and quality are for production environments but we can apply the same analogue thinking to the deployment pipeline as this is just a shorter feedback loop, for the commit-stage, consider tracking

• Integration Frequency (i.e. how often we push or merge and as a result run the commit stage)
• Integration Lead Time (Time since the commit or Merge Request (MR) was created until it landed on the mainline)
• Mainline Recovery Time (How long it takes from a failed commit-stage job to be resolved and succeed again)
• Build Failure Rate (The rate between succeeding and failed commit-stage jobs)

If we want to reach these goals in production, it naturally follows we need to do it at least just as fast in our development environment.

Ironically, code changes are not the only source of failures – and the developer making the breaking change might be unable to do something about it on their own, which only increases the fear of making changes. According to Continuous Delivery [3] there are 5 likely reasons why a pipeline might fail

• There is a bug in the application code
• There is a bug or invalid expectation in a test or test code
• There is a problem with application configuration
• There is a problem with the deployment process
• There is a problem with the environment

Only some might be directly influenced or caused by the developer, generally flaky tests and/or infrastructure will add onto the fear – the DevOps organization must work towards breaking these environmental issues. If a developer cannot tell “I broke it” from “infrastructure broke it”, fear persists.

Normalize and Standardize Reversals

Mean Time to Recover is greatly influenced by the ability to simply undo the commit - either by removing it from the mainline entirely or by reverting the commit, i.e. adding a commit that undoes the change.

To normalize this behavior, make sure it is well documented as part of the normal development flow. Standardization means more people throughout the organization can help recover, and makes it easier to debug when the standard procedure fails.

Lower the Blast Radius of a Failed Deployment Pipeline

If you try to eliminate all failures before the mainline, you must run increasingly expensive validation: full pipeline runs, shadow environments, comprehensive test suites. This will quickly become an expensive (and likely error prone) affair as it scales horizontally with the amount of Work in Progress.

The real problem emerges when a failed build blocks other developers from integrating.

Recovery typically takes at least twice the pipeline cycle time. One cycle for getting a failure result and another for verifying the pipeline now succeeds. For a three-minute pipeline, that is six minutes and one can barely make it back from a coffee run. For a fifteen minute pipeline, this adds up to thirty minutes – that is several lunch breaks throughout the day.

Splitting the deployment pipeline into individual jobs where you can get feedback earlier will help you to quickly answer if you are blocking other people’s work. In Continuous Delivery this distinction is in fact made, where the fast feedback is provided by the “commit stage” whereas the slower running “acceptance stage” then figures out whether this is something that can actually be released.

The key is to distinguish between “fast vs slow” running tests - only check the things that are fast and help the next developer out such as

• Verifying the code compiles
• Verifying system invariants e.g via static code analysis checks such as linting
• Verifying application correctness via unit tests

This check should be bounded to match the team’s attention span, typically between 3-10 minutes. This could be an important metric to measure as well separate from integration lead time.

All the other quality checks come after as they do not provide value at this stage, deployments and deploy-script validation typically runs later as failed deployments do not block other developers - remember that a merged MR or feature branch does not mean the feature is necessarily done.

To protect later stages from failing prematurely we can make use of feature flags until the code is finally ready. We are thus de-coupling integration from deployments and we can safely collaborate with teams without jeopardizing the production release.

Empower and Educate

People are in general inclined to “do the right thing” – at least as long as it is easy. Continuous Delivery practices and flow optimization in general are not always intuitive. What benefits the broader (development) organization may impose constraints on individual teams. Education is a continuous necessity as there will inevitably be turnover (if not, you have other problems of stagnation to solve), it is not enough to say “we do " here. The processes and rules must be restated in the organization's own language explaining what feature branching or Trunk Based Development means here, and what the expected behavior is. Never assume people read the book or even bothered googling it. A process document where the last change was a few typos three years ago does not count. Any process that does not change repeatedly is a dead process.

Naturally, to motivate the right behavior you also need to make people able to perform whatever operation is required, opening a ticket is not resolution, it is escalation. One side of the coin is the fast feedback (in a readable format) that can tell the developer “this was your doing” - with a clear indicator of what inadvertently broke. The other side of the coin is that they must also be given permissions and knowledge to be able to resolve it on her own accord.

Measure and Improve

Nothing speaks like a visual dashboard (sometimes called an information radiator) with clear-cut data showing the organization where we are (and where we want to be) - one of the impediments, and probably hazards is that it forces us to define with mathematical precision, and come to terms with reality, how we measure productivity and how effectively we actually work.

Using the adapted DORA metrics, as defined earlier in the post, my personal recommendation would be to target

Integration Frequency

If the deployment pipeline is triggered on changes to the mainline we can simply increase a counter every time the build starts and select the period we wish to get the rate for e.g. daily. For periodic based triggering mechanism we need to find another mechanism such as closed MRs.

Continuous integration practices advocated for integration work at least once a day, where more modern day practices advocate for even more frequent integration. This is an opportune time for the organization to discuss and agree on the target rate.

Integration Lead Time

In isolation this metric is easy to game. But together with integration frequency, it becomes harder to fake - we can choose to get the MR creation time from the code repository service or, perhaps more universal, the time of each created commit if possible. Integration Lead Time should be calculated from the earliest commit timestamp in the MR/feature branch to the timestamp when that change is merged into the mainline (i.e., when the merge commit is recorded).

There are many reasons this value might get skewed, normally I see work paused because of priority changes (and this is one thing we want to combat) - other reasons might be vacation or sickness. In general this is a signal that people may not be checking in frequently enough. Long lead times often indicate batching or delayed integration.

Mainline Recovery Time

We need to track not only every build, but also the final result of the job paired with timestamps. Tracking this as a time series we can filter out and compute the mean time difference between the first red job until the next green one. We can also choose more direct approaches if MTTR is the only value we are interested in but we need these values to also calculate the build failure rate.

The target MTTR is a conversation that needs to be agreed on between relevant stakeholders, as well as when it is appropriate to roll back other people’s changes.

Build Failure Rate

While DORA’s CFR measures production failures, we track commit-stage failures as an earlier signal to catch issues before deployment.

Now that we track the result of each individual build we can calculate the proportion of builds that fail.

At this stage we are only aiming for “good enough” – fast feedback is often more valuable than exhaustive feedback at this stage.

Accept an initial high failure rate, say 20-30%. As the system stabilizes and flakiness in the environments or test-suites are addressed the targeted failure rate can be lowered.

[1] https://dora.dev/research/2016/2016-state-of-devops-report.pdf [2] https://dora.dev/guides/dora-metrics/ [3] Continuous Delivery - Dave Farley, Jez Humble [4] https://youtu.be/v4Ijkq6Myfc [5] https://a4al6a.substack.com/p/stop-using-pull-requests

Andrea Laforgia gives a much better discussion on how this strategy can be used as a stepping stone for pure TBD, providing excellent reference material along the way.

In short, the approach categorise changes into three scenarios

ship - Small changes, side effect free, push changes directly to trunk. Perfect for fixes or otherwise side-effect free changes.

show - Create a merge request and merge it when it passed checks. Perfect for feature development where you don’t need work to get stuck in the “waiting for feedback” pit.

ask - Request and discuss feedback before going any further. Perfect for inter-team changes, newcomers to the code-base and re-designs

My contribution is the flowchart below combining the whole thing into a single decision tree from the perspective of an individual contributor.

Core concept definitions and caveats to be aware of when defining a strategy for verifying system behavior post-deployment.

I hope this text will be used for defining a good strategy, bringing everyone on a team - from junior to senior together into a discussion of setting a common baseline and understanding of distinctions and subtleties. The content is a likely candidate for a skills.md file perhaps.

I am purposefully leaving out any tool recommendations in order to keep the text high-level and to keep it clear of bias towards a particular stack.

Slice and Dice

Component Recursion

These definitions work for systems, and can be broken down to aggregate components and again into covering subcomponents.

A system can be made up of architectural quanta - i.e. groups of separately deployable units or such as a set of microservices. Such quanta make up a set of subsystems.

It does not make sense to talk about readiness or health without specifying what falls under our implied “what”. A component can be healthy while the overall system is not. The overall system can also be healthy whilst there are unhealthy or failing components (within some pre-defined threshold).

Additionally, components are often structured in layers, adding another dimension to their design which we will explore next.

Systems are like Onions

Today, most services run in some form of virtualised environment. As a result there are at least two operating systems under the service you want to monitor (host OS + guest OS). Each layer makes up the platform of the next one - at some point, containerization (like docker) or virtualization (like a Virtual Machine) will make the platform isolated for the component - but other subcomponents will share an identical layer. Similarly, these layers will share strengths and weaknesses, and might be monitored in the same manner.

Readiness Checks

Performed once at initiation and answers the question: Is the service ready to serve requests in a meaningful way yet?

The check is very useful for blocking downstream processes that require the component to be functional before proceeding.

Focus is on whether the component is able to serve main functionality, such as answering http requests, the component might present itself as ready but with degraded performance (a system might be ready to perform work but slower whilst warming up a cache writing things to disk).

The check ought to be time-bound, and must eventually result in failure to prevent failures propagating down-stream where they are more difficult to identify.

On a system/deployment quantum level it makes sense that readiness takes on the value of the readiness state of the component with least ready state but that depends on the architecture. If you spawn multiple workers horizontally, you do not necessarily need all of them up.

The most simple “readiness” declaration is simply writing a value into a designated file as the last operation performed.

stateDiagram-v2
    [*] --> Starting
    
    Starting --> Ready
    Starting --> Failed
    
    Ready --> [*]

    Failed --> [*]

Healthchecks

Ongoing check of the component that takes place throughout the lifetime of the system.

Can be performed as a self-check (i.e. from inside the component), from an external agent, or both.

Focus is on the factors that sum up to whether a component is healthy or not. e.g

• Are all internal services running
• Are resource consumption within thresholds.
• Did the readiness check complete

It is necessary to decide if/when readiness affects health. Do components getting re-deployed, and thus not ready, affect system health? Perhaps only for critical services or for specific times of the day.

As described in “Systems are like Onions” it is not uncommon that we want to split health of a component into the layers that make up the component. E.g. we might wish to monitor container or virtual machine “health” separately from the health of the webservice that runs on top of it.

As a good starting point, monitor the health of the layers that make up the platform externally, and expose the results of a self-check factoring in component specific behavior.

The right approach needs to take into consideration instrumentation strategies and tooling.

Now that these distinctions are in place, we are ready to dive into how these tie together when it comes to smoke testing.

Smoke Testing

When deployment is complete and readiness checks report success, then it is time to do the final smoke test verifying system behavior.

If the readiness check or health check performs operations similar to what a user would do (e.g. hit the service from the external domain address) - then the minimal smoke test suite is merely conducting a verification that the state is as expected by issuing a GET request against that target domain adress.

If the checks up until now are based on self-reporting, then a successful smoke test suite should access and verify the system the same way a user would. You are basically writing an external auditor. This is also where it might be confusing, and indeed code-reuse from readiness checks is to some extent practical.

The more advanced suite will conduct a series of automated fake operations similar to acceptance tests to verify the system also works with the different components when put to actual work. These will mimic transaction or code-paths through the deployed stack. A simple login operation will not only check the HTTP endpoint, but possibly backend API calls and the underlying DBMS.

A final note is that most systems are not deployed just once. Throughout the build process they are usually deployed multiple times over in testing or staging environments. The scope, and perhaps difficulty, of smoke tests is to identify functionality that deals with verifying the actually deployment, i.e. to apply the same set of test equally throughout different deployments as they mature, such that potential errors are detected early in the process and not post-release.

Resources and Recommended Reading

The content is based on hard earned experience, lessons from seniors I have worked with, guided by these works by giants that stand out in memory

• Release It, Michael T Nygaard - Architecture and Site Reliability Engineering before there was such a team
• Continuous Delivery, Dave Farley and Jez Humble - How to deliver quality software, fast. I think that is even the tagline of this book.
• Practical Monitoring, Mike Julian - More concrete definitions and low-level strategies for operations including process building.
• Building Evolutionary Architectures, Neal Ford, Rebecca Parsons, Patrick Kua, Pramod Sadalage - Designing and building software systems in an agile way, guided by instrumentation, data driven observations and user needs/business value to feed the cycle.

Wrapping Up

Even small teams witness terminology confusion and overloading of terms. As the number of actors grow, so does this problem.

What we name things is not important, what we mean by those names is.

Verifying the system after deployment is an important step for any product and from this post it should easy to define the validation strategy from a common baseline using smoke tests and on-going health checks.

The most challenging part was reconciling my frustration towards colleagues and decision makers who did not seem to care about something I felt was of great importance – even though I worked in a team where code quality was very highly regarded and prioritized, I wanted even more I wanted everyone to strive for the same perfection as me.

The fact is, it is never going to be enough quality or tidiness and you will always land back on a satisfactory level of 7 (or a 3?) – not unless you can find a good metric.

There are many great lessons that experience, and philosophy, has taught me to harness emotions. Two of these I will expand upon in this post

1. The path is the goal
2. Love your fate

The Path is the Goal

My unwavering focus on code quality and technical debt led me to explore systems design, build systems, and beyond. It has led to hands-on experience with the inner workings of tools to generate documentation and visualizations based on reflections of the code, abstract syntax trees for unconventional linting rules, and a myriad of templating and interpolation techniques.

This pursuit of quality has made me the engineer that I am today. I like that engineer.

I have also learned that the quality I sought was a golden calf. I have had the humbling experience of working on the same codebase long enough to witness the unintended consequences of my own work. One experience that stands out to me was a refactoring initiative I took on for the purpose of improving code quality – years later, working with the same code I found my earlier assumptions to be false, and my “improvements” had made the code worse. This lesson, among others, has forced me to rethink my definition of code quality.

Quality is not about a uniform and tidy codebase, it is about adaptability. I have found that it is better to leave working code unchanged unless you absolutely have to, regardless of how it looks.

Coding guidelines and static analysis are invaluable tools for preventing code rot from setting in. I would not like to work on a project without them, at least at the integration stage. But be wary, as these things can quickly devolve into bikeshedding, with teams or individuals obsessing over the wrong bottlenecks. The road to hell is indeed paved with good intentions.

Love Your Fate

This stoic phrase (or Zen-Buddhist, take your pick) encourages us to accept things as they are. Only then are you free to act.

Code is the way it is because work has been done – work that provided real business value. After all, that is our motivation for code quality, isn’t it? It is okay; it is good code because it is what brought us here. The design, the foundation, the toolstack too. It held up the codebase, it kept the business – well, in business. Perhaps instead of calling it “debt”, we should view it as battle scars.

Whether it is debt or not does not matter much; what matters is the next step you take. Make those steps towards internal acceptance, and practice identifying smaller increments to make. To identify the real bottlenecks, I have learned to ask myself where the real problems are, what obstacles prevent people from performing their work, and what difficulties new personnel must face. What traps do they consistently fall into?

Conclusion

I have grown to a place where I think there is no such thing as “technical debt”, I prefer the term “unmanaged complexity” which I believe was coined by Dave Farley.

I have realized that management or decision makers are not to blame or be held accountable for poor complexity management – and refactor is not the solution, that is a cop out. This is an engineering problem, a group problem and the two most effective things I can do to battle it is

1. Write documentation, and then write some more. Specs, tickets, commit messages–everything matters. As a senior, I know and understand the context and systems better than most. I understand the “whys” and the theory behind. That needs to be communicated – or else everyone makes up their own explanations.
2. Spend all the time needed to help, show, discuss, review the work of junior staff. Sometimes it is skills or experience lacking. More often than not, they lack context. Even with documentation, gaining a true understanding of the system takes time.

I find that most often the poor complexity management comes from well meaning people who don’t know better because of lack of context and/or experience – most often it is misunderstood attempts at making improvements that backfires into more complexity – it is a socio-technical issue and I have transgressed into treating it as such.

By reframing technical debt as unmanaged complexity and embracing these lessons, I have found not just peace, but also a more rewarding inner work life.

Defects refer to anything that functions unintentionally, while operational defects typically require manual intervention or tracking through automated observations. This post introduces a structured workflow for teams to collaboratively detect, track, and resolve such issues. It emphasizes structure and communication across functions and responsibilities.

I will argue how the data collected can be used by the team and decision-makers to prioritise fireproofing. It is not an all encompassing terminology-worthy methodology, just a starting point for any team to start working with these issues in a structured manner.

Imagine a case where a rogue process forgets cleaning up after itself after analyzing a file upload i.e. many files are written to the temporary directory, eventually exhausting the disk. There are load-balancing in place, for other reasons, and this particular thing happens once in a while on one out of 4 replicas. The machines are normally restarted for other reasons (it is the default first step resolution method) and is usually handled when half the fleet is down.

Issue Tracking

Tracking issues by registering them is a tedious task and requires the kind of overview you don’t really have. Tickets are the backbone of almost any task management workflow so there is an almost certainty this will allow you to implement the workflow with existing and known tools.

We need two types of tickets. I will call them “bug” and “task” but their type is not important - how they function within the team’s workflow is.

The bug-ticket is a collection item and must be able to aggregate task-tickets into a child/parent relationship. The bug-ticket potentially lives for a long time in the backlog.

The task-ticket follows normal board workflow. This is the sort of task the “expedited” lane or “unplanned work” is actually for.

With the overview of the structure in place, let us look at how to handle observations.

Handling Observations

A new task-ticket is created on each observation.

Naming

Use some convention of title e.g Handle: "<error-message> "on "<instance>"

1. The static prefix “Handle” becomes recognizable from glancing over summaries what class of tickets this belongs to
2. error-message is that nice juicy explanation of what actually happened. e.g “Full disk usage” or “Out of Memory” Most often you will find that sentence in a log or from the monitoring dashboard. This will most likely carry in some form of the bug-ticket title as well.
3. instance is the identifier of what machine or workload this was observed on. If you have multiple replicas you will want to know which one. e.g. “worker_fa4cd67e” or “service-db-2”.

A naming convention ensures overview and readability for the human eye as the backlog grows, and helps include a minimum of relevant information.

The handler (or handlers, preferably) should make a best effort attempt at evaluating whether the observation is a known defect, or a new one. This classification helps determine what the right course of action is. It is also a great opportunity to communicate with your team and leverage collective knowledge “Hey we’re seeing this – has anyone observed it before?” In the team, you should agree on a distinct course of action when no precedence can be found that ensures it can be properly found and triaged at a later point in time.

The team registers two tickets – one for the bug and one for the observation, linking them together. (No space left on device and Handle: "No space left on device" on replica-2 respectively).

Description

The primary objective is to collect evidence. Attach stack-traces, log outputs. Images of the monitoring visuals. You are the criminal investigator of a crime-scene - you don’t know what details are relevant so collecting comprehensive evidence is key.

Most important is:

• Making a record of the actions taken.
• Recording what state you can of a system before doing any destructive operations (i.e. if you clear a folder due to full disk utilization, what is actually on there filling it up will explain something).

The week after, replica-1 goes down. Same error message No space left on device. The team remembers the bug ticket from last week and knows it is the same thing. Running ls -lah on / reveals /tmp to be the culprit, containing a large number of files with varying size. The /tmp folder is wiped, restoring the service. The actions and output is added to a new ticketHandle: No space left on device on replica-1" which is linked back to its parent.

Classification

The bug-ticket represents the continued learning about; the bug in the machinery.

Focus is on iteratively filling in the description of the bug as we get wiser, allowing understanding to build up over time whilst keeping cognitive load low. For a first iteration, a copy/paste of the first observation ticket description will do. When we classify and triage early, we immediately start homing in on the pattern and cause of the defect.

From the initial task we likely know “how to identify/remedy” right from the start. As more observances are made, we might get smarter - i.e. instead of restarting the whole machine, we just have to restart this service. Start recording “next steps for investigation” (this is why you triage when handling, there might be clues as what to record or what actions to take next time around) - e.g If the disk fills up recurringly, we might want to start investigating what files or workloads are present when it errors.

This structured approach will slowly reveal the root cause or appropriate mitigation of the defect to you and your team. Not only do you know the cause - you managed to track observances which mean you can extract the data of

• How often does this happen?
• What is the impact?
• (When) will this happen again?

These are questions you need to answer to make a case for working on a defect a priority.

The following week, disaster strikes again this time hitting replica 2 and 4. The week after next replica 3. Every time /tmp is filled up with some weirdly named files of varying size. The team can now show that there is something going on and it seems to hit the machines one by one. They manage to make the case to set aside time to investigate on the next sprint. One member of the team believes to have found the coding error causing this issue. It is all written down into a ticket linking back to the issue.

With the root cause and patterns of the defect identified, the next step is to prevent it from recurring again.

Fireproofing and Remediation

You might think you managed to get to the root cause of an issue and fix it just to experience the same errors come popping up.

Addressing a defect is distinct from tracking and observing it in the wild, and separate tickets should track that progress. Luckily it goes well within this workflow and structure to track other work done relating to the bug-ticket – we can attach all of those work items alongside any observation.

The issue is found to indeed be a temporary file not deleted after processing an upload. This strikes a debate on handling of temporary files and one team member finds a good article upskilling everyone.

In some cases you need to delegate work to other teams - in which case the context we are creating with this workflow becomes even more valuable. Make sure to properly fill out those tickets with relevance, business value and expectations. Also remember to respect their cadence and workflow.

A similar issue with temporary files elsewhere in the codebase is highlighted - this time it is not in the team`s own code but in another service - another ticket with a description of the potential issues and a description of the remediation is made, linking back to the bug. The ticket is then assigned to the colleague team. It makes it onto their next sprint without any back-and-forth.

Wrapping Up

There you have it, a structure to work with that provides all the data you need with the tools you probably already have at your disposal and use every day. From here, it is up to you, and your team, to make use of this workflow and resulting structure for analysis, data-driven decision making, and remediation, to tackle big or recurring problems incrementally, instead of repeatedly, together.

What might be new, is my audacious attempt to provide a suggested priority or order in which to apply. I tried crystalizing the effort into concrete actionables that, in my humble opinion, yields the highest reward.

My intended audience is anyone I just so happen to think might benefit from reading these thoughts on the subject by linking them directly, shamelessly promoting my own thoughts - and, of course, any random developer passing by searching to hone his or her skillset.

Tip #1: Rebase Early, Rebase Often

This tip placed at the top as it exposes you to any troubles as early as possible - merge conflicts are hell, but the larger the diff gets, the less likely you are able to resolve it and the more likely you will have wasted your time trying or applied a poor remedy.

Rebase your current work at least a couple of times a day, e.g. when you sit down at the start of your day and immediately after lunch. Personally I normally rebase on main prior to pushing anything remotely.

My favorite command (aliased I call it refresh) is git pull --rebase --autosquash --autostash this makes rebasing non-intrusive on my dirty worktree.

Tip #2: Keep Commits (and Merge Requests) Small

Iterate functionality in the smallest way possible - stay on-point in your Merge Request (MR) and prefer to make more MRs. (I know, if review process is slow this is a pain, but then that pain needs to be addressed as a cultural/practical problem).

In general, you shouls consider that the more lines of code changed makes you more susceptible to merge conflicts (and your risk conflicting with the works of others).

• Keep style and cleanup changes not directly related to your changes i.e. separate from feature work.
• Keep the number of changes and files in a commit down.
• Keep the number of commits in an MR down.
• Keep commits atomical (i.e. a particular change in the MR should be revertable without affecting other code)
• Write your changes so they can be integrated and work in isolation without being “activated” - i.e. add your library function and tests, then in isolation you can add the call site code.
• Use -p when adding changes, this will allow you to review and partially add changes in a file. (note that you cannot add untracked files with this option).

Kent Beck’s Tidy First is highly recommended reading, I have spent over a decade practicing good commit structure, still this book took me to a whole new level of thinking about when and how to structure my efforts.

Tip #3: Use Descriptive Commit Messages

In the ideal world we complete a task, switch context and then complete another one. In the real world, we switch contexts all the time - sometimes it is other work taking precedence forcing us to switch, sometimes it is time off due to weekends or vacation.

Pay it forward to your future self and practice being explicit in words giving a summary and your current reasoning for making a particular change.

Even more important is to write down what is currently not working or missing (so that you do not come back after an extended period of time, thinking work is complete and merge it to find out you entirely missed everything but the golden path).

I love pointing to https://commit.style/ and https://dhwthompson.com/2019/my-favourite-git-commit as good short examples where https://cbea.ms/git-commit covers the good commit message in depth. I also recommend adopting something akin to https://www.conventionalcommits.org/en/v1.0.0/ - not for the properties of parsing and automating generation of changelogs but for the purpose of ensuring all important pieces of information for posteritys sake are somewhat added the commit message.

If you think this sound like a lot of work, then then at first you maybe right - this brings me to tip #4.

Tip #4: Learn to Rewrite Branch History

Consider commits on a branch as “staged” changes - they are up for revision until they hit the main branch, in terms of content, description, and order for that matter - learn to quickly do history re-writes, the more you do it, the more second nature it becomes. Here is a selection of such re-writes I use on a daily to weekly basis.

See also https://about.gitlab.com/blog/keeping-git-commit-history-clean/ for some more in-depth explanation of my presented scenarios. As an aside https://ohshitgit.com/ explains a number of ways to get you out of any pickle you are bound to get yourself into trying out these things.

Add a missing file/change to the last commit

git commit --amend --reuse-message=HEAD will commit any changes currently staged. I alias it git fix

Add a missing file/change to a previous commit.

Assuming the changed is staged, use git log or similar to find the subject commit sha. Then git commit --fixup <commit-sha>. This will add a commit with a special message to your history and you don’t have to think of a commit message. Use git rebase main --autosquash to patch the target commit (or git rebase -i main --autosquash if you want to inspect the order first)

I need this bugfix to continue my work but it is unrelated

Commit the change here and now to feature-branch, then checkout another branch from main and use git cherry-pick feature-branch (as long as the commit is the one on top cherry pick will do this) - push this fix and create an MR - continue working on feature branch and when the fix ME is merged, the bugfix change commit will automatically be removed from feature-branch upon rebase as it is empty.

I Have Multiple Commits That When Squashed Finally Works

Using git rebase -i main - I prefer the mechanism where squashes are made by fixup actions on subsequent commits (usually the first commit contains the description I wanted in the first place). Sometimes I reword the first one.

Sometimes this gets hairy, especially if you placed other changes in between - I then first go for multiple iterations i.e. using the rebase command over and over selecting only a subset of the changes to fix/squash each time.

If that doesn’t help I take on more drastic measures not covered here.

This change was placed in the wrong commit

Assuming there are no subsequent changes causing merge conflicts, git rebase -i main and select edit for the commit.

Rebase will now halt git restore -p --staged <filename> until the staged changes look like the commit you want. The remainding changes are left unstaged now.

Then git rebase --continue. Now the accidentally committed changes can be added to the appropriate commits.

Pro tip: Rebasing is just an operation that replays a set of commits onto a new base branch - you can even make new commits while performing this operation.

Tip #5: Learn the Internals of Git

You don’t have to like working in the command line or even be able to get yourself out of any particular jam - but you need to invest time understanding the basics and the internals of Git. Most important is knowing the concepts and terminology just for the purpose of googling and prompting to get yourself out of a rot. Pro Git is available for free https://git-scm.com/book/en/v2 and is recommended reading for anyone working profesionally with software development.

In the end, nothing replaces learning the commands and their flags little by little with the help of the command line output and a manual.

Wrapping Up

You don’t get good without greasing the groove. It might feel like a lot of work up front, but down the line this knowledge will save you headaches and redoing work. You are likely stuck with git as the defacto version control system for any foreseable future - and when the time comes, these lessons apply to the new tool as well. Any time invested upskilling in version control will be sure to pay itself back over and over during the course of the next decade(s) you don’t need to go the full mile instantly, just use this to consider where your next 1% improvement might be.

The purpose is to explore a method that allows for ease of experimentation working with the tools at hand to get started quickly with DORA as opposed to designing the perfect generic production setup.

If you need a primer on DORA metrics consider reading https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

The TL;DR: Using gits built-in annotated tags to register when a release has been made, it is possible to use the “taggerdate” value og the reference. This can be used to compute the timedelta between each release to give the deployment frequency. Subsequently to obtain all commits in each release and compute the time between when the commit was made to the release time giving the lead time to release metric.

Applications and limitations

For the example code the tag-system is expected to be used for releases only. It will probably take some customization and hardeking to get it working in a concrete scenario.The method will probably work in a simple pipeline structure as well as a fan-in pipeline structure. However the more sources you add and the more frequent changes are made, there is provably better ways to compute these values. And of course, if you rewrite/squash history as part og merging etc, you will lose important pieces of data. Keep development in the trunk!

Creating Tags

When a new release is made, it is necessary to get the CI pipeline or a manual tool to post an annotated tag back to the repository.

$ git tag -a release-123456789 && git push origin release-123456789

Annotated tags are used as the timestamp of the tag creation time can then be used as opposed to lightweight tags that will only record the timestamp of the referenced commit. That might be ok for your use case.

A comment on security

Beware that you do not inadvertently add a code injection vulnerability into your pipeline. Any acces from the automated pipeline to the repository should be with access to push tags only 

Deployment Frequency

Deployment frequency is the time between deployments to the production system. I like to adopt the terminology of continuous delivery pipelines and use the term “release” instead of “deployment”. Given a deployment can be an internal deployment to a certain cluster etc of a release candidate. This mixed terminology has got me confused more than once. In order to calculate the metric, we simply need to know the timestamps of each individual release and can then compute the mean of all the timedeltas between the releases. Notice that when to record this timestamp differs whether you are doing continuous deployments or working with release candidates.

def get_releases():
    cmd = "git for-each-ref --shell --sort=taggerdate --format='%(if:equals=tag)%(objecttype)%(then){\"ref\":\"%(refname:short)\", \"dt\":%(taggerdate:format:%s)}%(end)' \"refs/tags/*\" | grep -v \"''\" | sed -e \"s/^'//\" -e \"s/'$//\""
    output = subprocess.check_output([cmd], shell=True)
    
    releases = []
    for line in output.decode('utf-8').splitlines():
        releases.append(json.loads(line))

    return releases

def calculate_deployment_frequency(releases):
    deltas = []
    for i in range(1, len(releases)):
         deltas.append(int(releases[i]['dt']) - int(releases[i-1]['dt']))
    return statistics.mean(deltas)

Lead Time for Change

The mean time to deploy is a bit more tricky to compute and requires the known timestamps of each and any change made as well as which release it is part of. We want to calculate the distance from all of these changes to the release timestamp.  This will give us the metric for one release, so we need to compute this for each changesets to each release.

def calculate_lead_time_to_release(releases):
    start = None
    deltas = []
    for release in releases:
        release['commits'] = []
        end = release['ref']

        r = f'"refs/tags/{start}".."refs/tags/{end}"'
        if start is None:
            r = f'"refs/tags/{end}"'
        
        subcmd = 'git log --pretty=\'{"ref": "%H", "ts": "%ct"}\' ' + r
        subout = subprocess.check_output([subcmd], shell=True)

        for line in subout.decode('utf-8').splitlines():
            cmt = json.loads(line)
            delta = int(release['dt']) - int(cmt['ts'])
            # Maybe bucket the commits into the release?
            # release['commits'].append(delta)
            deltas.append(delta)

        start = end
    return statistics.mean(deltas)

Conclusion

I have shown a viable simplistic method for computing throughput metrics without a complicated setup making use of event handling and databases to get started using these metrics. The important part to me is how to obtain the relevant datapoints as they can be sliced and bucketed to any liking from here.

Complete example script

#!/usr/bin/env python3
# Author: Ronni Elken Lindsgaard <ronni.lindsgaard@gmail.com>
# License: MIT

import subprocess
import json
import statistics
import argparse

units_of_time = {
    'days': 60*60*24,
    'hours': 60*60,
    'minutes': 60,
    'seconds': 1
}

def main(args):
    releases = get_releases()
    divisor = units_of_time[args.unit]
    print(f"deployment frequency: {calculate_deployment_frequency(releases)/divisor}")
    print(f"mttr: {calculate_lead_time_to_release(releases)/divisor}")

def get_releases():
    cmd = "git for-each-ref --shell --sort=taggerdate --format='%(if:equals=tag)%(objecttype)%(then){\"ref\":\"%(refname:short)\", \"dt\":%(taggerdate:format:%s)}%(end)' \"refs/tags/*\" | grep -v \"''\" | sed -e \"s/^'//\" -e \"s/'$//\""
    output = subprocess.check_output([cmd], shell=True)
    
    releases = []
    for line in output.decode('utf-8').splitlines():
        releases.append(json.loads(line))

    return releases

def calculate_deployment_frequency(releases):
    deltas = []
    for i in range(1, len(releases)):
         deltas.append(int(releases[i]['dt']) - int(releases[i-1]['dt']))
    return statistics.mean(deltas)

def calculate_lead_time_to_release(releases):
    start = None
    deltas = []
    for release in releases:
        release['commits'] = []
        end = release['ref']

        r = f'"refs/tags/{start}".."refs/tags/{end}"'
        if start is None:
            r = f'"refs/tags/{end}"'
        
        subcmd = 'git log --pretty=\'{"ref": "%H", "ts": "%ct"}\' ' + r
        subout = subprocess.check_output([subcmd], shell=True)

        for line in subout.decode('utf-8').splitlines():
            cmt = json.loads(line)
            delta = int(release['dt']) - int(cmt['ts'])
            # Maybe bucket the commits into the release?
            # release['commits'].append(delta)
            deltas.append(delta)

        start = end
    return statistics.mean(deltas)
    

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Calculate DORA throughput metrics')
    parser.add_argument(
        '-u', '--unit', help='unit of time', default='days',
        choices=['days', 'hours', 'minutes', 'seconds']
    )
    main(parser.parse_args())

This post will not be about any principle in particular, but instead about how I apply those principle in practice, and more importantly when in the process of development I apply those practices.

Growing Your Software

As of late, I have been reading and watching much material that draws similarities between software engineering and gardening, more than architecting and building buildings. I subscribe wholeheartedly to that analogy as that very much describes how my process looks like. Enough of that analogy, I only ventured there to make a point:

  Running code is alive, or need to be. It needs to grow into being
  and it needs care. Once it has matured, it can live mostly by
  itself, but when we start writing the code we know only little
  of how it will look like in the end.

I for one have been fooled many times over by designing systems up front in order for priorities to be shifted mid-way. To mitigate this, I rigorously apply tracer bullets and the extreme programming practice of quickly punching a hole. This makes me stay agile, in the sense that changing directions is fairly easy.

Getting feature complete is a matter of iterative refactoring, which brings me to

Another Analogy

When working on a greenfield project, we do not have much to work with. It is a clean slate, but it is also a time of many insecurities. Usually, we have some sort of idea of what we are about to get ourselves into, we know enough to sketch a design or architecture.

I sometimes draw, I think I am actually quite good at it using guides and everything. That makes me draw up (get it?) this analogy.

The initial sketch is good, we have a general idea of the main components that need to interact, where they will be placed - or at least a suitable place. So, stop analyzing and get to business, don’t worry about the principles just yet (wait, remember KISS and YAGNI here). Crank up the volume on your headset or whatever gets you going and get writing some code (also don’t forget your TDD practices, they’re going to be very important going forward.)

Oh, and NO DRYing. Everything should be WET/copy-paste at this point.

Welcome to round 2

Ok, you nailed that first user story! You punched a hole and you got a sunshine scenario working. It may be running on stilts, but the tests pass. (and by the way, if you did not write a complete end-to-end component integration test, now is the time to get back to your test-suite) - because we are now ready to refactor for great good – and refactoring means testing for regressions, constantly. Component integration tests ensures that you do not forget to update the interfaces between the already moving parts.

Waaait a second. Interaction between moving parts maybe calls for some design by contract?

But first, get those changes committed, reviewed and deployed.

Now, you are ready for your, first no-brainer refactoring. Get started on an interface that formalizes the structure between those component. You got yourself some tight coupling there so that part should maybe put into a shared library between the two components.

Don’t worry about having to deal with tests, you already wrote those. You maybe need to change the in and out on your components’ unit tests but your integration test should be steady as a rock - you’re just checking for regressions. If you find that you are starting to meddle with the component integration test, you’re overstepping your boundaries just now (except maybe for some renaming of imports)

There is a possibility that other smells accumulated. Personally, up to this point I have usually been somewhat inconsistent with naming (it takes me some working with code and terminology before getting it, almost, right… I never get naming right). In practice, I start removing abbreviations in variable names and make sure all methods are verb-phrases, all the code smells I can think of really.

With experience, you will know yourself well enough to know what you usually let lying around. Might as well take care of it immediately.

Also, DO NOT DRY yet.

Round 3

Time passed, you had a breather because the system needed to get running on the double, and then something else caught your POs attention… And you needed to fix this bug you left lying around in another system, and also your cat ate your commit, and…

Aaanyhow. At least you probably had a weekend or a chance to sit back and think about the mess you made thus far. And your PM calls you up saying “I got some new exiting task for you, and PO wants it yesterday for this customer demo who will cancel anyhow”.

Great, you got another task, and new requirements to draw some more lines – and maybe also start using an eraser. You can begin to see where the code is heading (ok, you probably already knew because the backlog accumulated like a waterfall thanks to PO/PM). Do yourself a favour and ignore it, they have just as little idea of what comes next. Because it is most likely going to be something that they haven’t thought up yet.

But this is good. Now we start getting a glimpse of the future of this code. Until now, we really just did some pre-germination or some sketching. Now it’s time to take a new pencil and start drawing some actual lines. (this analogy is really working for me!) You can start making plans for your code. And you have a very real idea of what you focus needs to be. It is the immediate surrounding of the lines (of code) or components that are making it into the changeset.

A word of caution, remember not to go full retard here. I have been caught here too many times myself thinking “is this SOLID? Ugh, which way does the composition arrow go? what do I need to do to get it just right?” Just don’t.

It is a waste of your time. Or, it is not a complete waste, but your time is better spent elsewhere so it is a waste of your employers time and your wild-ass-guesstimate is closing in on its due date. The important part is, your code is maturing, and it is solidifying (and maybe a little bit of SOLIDifying too?) which means you also start being able to make better judgments about the immediate future of the code, based on the lines you are actually changing.

If you are anything like me, you will also start identifying the pieces of the main code that really ought to be dragged out into subroutines, split up or rename components, ordering function arguments etc.

You may start DRYing now. But consider if things should not still be WET.

Rinse Repeat

This is it, it is this pattern iteration applied continuously to use ones best judgment and applying the boyscout rule ad-libitum.

Personally, I expect 2-3 major refactorings before I know where code is supposed to be loaded and placed. Then I start working on the details. Again, this is like when I draw, after drawing the guides, I work on the “main” lines and features. Draw those over and over to figure out exactly how they should be placed between one another. And when I got this sorted, I start working playing with the details.

For the code, it means that now it begins to make sense to start running down the checklist of your favorite principles and patterns to apply. One by one.

Stay agile

As a senior developer I have witnessed refactoring is something developers needs to be encouraged to do (especially if they did not write the code they are changing in the first place and they have no ownership). I regularly get “but it has already been reviewed” or “Let us do it some other time” when suggesting refactoring. I have a feeling the resistance is really due to it being seen as an admittance that “If I refactor, it must mean I was wrong the first time around and I don’t like admitting that”.

If that sounds like a familiar I hope it helps to remember that a garden is never perfect either, it is filled with living things requiring attention and it needs to be taken care of constantly. Sometimes, in order to grow, plants also need a little bit of time, or moved into or out of the shade. Sometimes they need to be trimmed, pruned, or nurtured. Some plants just die, some gets replaced for other reasons. In any case, the garden needs a caretaker or it will either overgrow or rot. Likewise will code.

I hope this inspired you. I know design principles can be hard to follow, not just understand, but also apply. This is written with the intention of giving some general pointers as to how a senior engineer goes about obtaining quality code. I am sure I have not managed to write down this process in full, and ask me in a year, and I would probably lay it out differently.

If you aim for perfect, you never get done. So just take one small step in the right direction, with each and every commit.

Summary

Design principles are great. I use them all of the time. But most importantly, I also know when in the process of development I should apply them.

• First iteration of a project or feature, only do the ones you got down habitually and simply “know” from experience is going to work. Focus on the business value, not the code.
• Second and third iteration, start identifying where the code is headed, where the “moving” parts supposedly are and get them isolated. Start clearing out smells as you get more intimate with the problem domain and its terminology.
• Ad-libitum, take care and clean the camp-ground - you will know when something starts to mis-align, and this is where you take a step back and refactor.

Recommended Reading

Litterature that makes the basis for my train of thought, or at least the teaching I recollect came from

• Clean Code, Robert C Martin
• The Clean Coder, Robert C Martin
• Test Drive Development, Kent Beck
• Extreme Programming, Kent Beck
• The Pragmatic Programmer, David Thomas & Andrew Hunt
• Refactoring, Martin Fowler

To me, utility functions are small bits of code that are DRYed as they are repetitive and does not relate directly to what the application I am writing does. It is more of a short-hand than anything else and I would otherwise have to write the exact same lines of code over and over.

If it relates directly to the application, I don’t consider it a utility function but an application of the clean code “functions should do just one thing”

def isoformat(dt: datetime):
   return dt.strftime('%Y-%m-%dT%H:%M:%S')

An example of a utility function. This is so generic that it is now part of the standard library

Utility functions usually “stand alone”, as they are not directly part of the application logic, it makes sense to keep them boxed in a util library or similar.

Even though they are defined for a particular part of the code, given the entire project works with the same problem domain, there is a considerable possibility it is re-usable elsewhere.

Testing

Utility functions are a very good place to practice TDD - you put something in and something comes out. Both ought to be simple and clearly defined. Utility functions, in my book, should rarely come with side-effects.

No doubt the function will be tested as part of an integration test for the application logic (right?).

However, from a maintenance point of view there is a possibility that the application logic will change or be removed from making use of that function. An observant developer would remove the utility function together with when making that change, but since it is added as a dependency. But, if there is no tooling to detect dead code, the likelihood of that utility function haunting the code-base after its utility has stopped is non-negligible. This in itself adds entropy, however when removing the application code, as a side-effect you also remove test-coverage of the utility function. Without separate tests, you get double up on code rot – in fact you get triple, because the knowledge of its utility is also removed.

Therefore, a utility function must always be tested separately from the application logic which is the main cause for its existence.

Altogether, when I add a utility method, I ensure to always include a separate unit test for that function.

def test_strftime_isoformat():
    dt = datetime(1989, 11, 09)
    assert strftime(dt) == '1989-11-09T00:00:00'

This conveys intent of the function to any other developer who might come by and think about changing anything. If not discouraging change, at least the expected interface is clear for the existing utilization of the utility function.

One might imagine a change

def isoformat(dt: datetime):
    return strftime(dt)

def strftime(dt: datetime, fmt='%Y-%m-%dT%H:%M:%S'):
    return dt.strftime(fmt)

Notice how neither application logic or tests are changed.

Documentation

The only users of a utility library is other developers on the team. The best place to put documentation is in the source code.

This is the most likely place another developer would look for what a specific utility method does anyhow, is it not?

Finally, it will allow you to populate project documentation automatically.

def isoformat(dt: datetime):
    """Turn timestamp into an iso8601 string."""

    return strftime(dt)

def strftime(dt: datetime, fmt='%Y-%m-%dT%H:%M:%S'):
    """Use `fmt` to format `dt` into a string.

    Equivalent to calling `dt.strftime(fmt)`.
    """
    return dt.strftime(fmt)

These utility functions are so simple that when looking at the code it does not bring any value. However, when generating documentation for the utility library you see some benefit.

(without a docstring, by default sphinx will not show you the function in the first place)

And of course, there is more to it.

Tests are Documentation

Nothing shows or documents the code better than tests, and python has a neat way of finding and running tests from within the function docstring. Meet doctest

You get to check that your code is correct and show how it works at the same time. After all, who remembers if ISO8601 has the T in it or not? And is Z required?

You also do your future self, and any other developer coming by, a one up by keeping the test right where the implementation is – we all have enough files open as it is, don’t we?

def isoformat(dt: datetime):
    """Turn timestamp into an iso8601 string.

    >>> isoformat(datetime(1991, 12, 25, 16, 35))
    1991-12-25T16:35:00

    """
    return dt.strftime('%Y-%m-%dT%H:%M:%S')

It is way more descriptive and readable than :rtype: str is it not? And you get some neat formatting in your documentation too

Summary

• Use utility functions for code of generic character. Detach them from the application code.

• Be sure to treat the utility library and functions as stand-alone components.

• The code in itself is of a generic character and may not show how it is intended to be used. Make sure to leave clues.

• You can can code and test in one place, giving you documentation for free.

Appendix

Try out the module below with

python3 -m doctest -v util.py

from datetime import datetime

def isoformat(dt: datetime):
    """Turn timestamp into an iso8601 string.

    >>> isoformat(datetime(1991, 12, 25, 16, 35))
    '1991-12-25T16:35:00'

    """
    return dt.strftime('%Y-%m-%dT%H:%M:%S')