Blog 14.10.2021

Devops 101 – Part 3-2: Failure is always an option

Good to see you here! We have no doubt this post has good information, but please keep in mind that it is over 3 years old.

Though this does fall under psychological safety, I considered this to be a big enough subject (especially for myself) to handle in a separate article. Please enjoy.

Fear of failure: my story

Again, I start with myself as an example. As a kid and a teen, I was good at school. I was not the perfect ten student (for those outside of Finland, our grading in school is generally from 4-10, where 4 means failure), but I was pretty good. I was also musical, and first played keyboards, then moved on to saxophone, and can say that I was pretty good at that too. Sounds great, right? The problem is that because I was good, then a lot was also expected of me. Especially by my father, who was also quite musical, and at one time played in a band. So, my daily routine was first school, then home and food, small break, homework, and then practice music. At that point, it was already evening, and I had some time for myself, though even then I was criticized for spending too much time on the computer and not doing something constructive. I lived in a small village, and all my “friends” (through the grades 7-9 I didn’t really have them) were living about 15 kilometers away, so not much socializing.

At some point, my music grade was 7, quite mediocre. But to my father, that was almost a personal insult. He was not happy, and he let that show, and I feared him. Another good case was when I got 6 from one math course in high school, and even though it was possible to improve it (and I did raise it to 8 later), it did not matter, my father was furious. Every time, when I had band practice, my father was there, and you can be certain that if something did not please him, he let me know about it later.

Why all this talk about my father, and me growing up? It is because it is the source of my fear of failure. If it was not enough having the peer pressure and other kids sniggering if someone made mistakes on the chalkboard, there was the demon at home, that made sure that I knew I had not done something up to his standards. I have always been emotionally sensitive, so I began to fear mistakes, fear failure.

I successfully got to the university and into the program I wanted. When the studies there began, I was shocked – I had expected it to be harder, but not that much. No need to say that my studies did not start well, and I made one mistake after another (e.g. tried to do too much at one time). And at some point, my fear of mistakes made me avoid situations where I could fail, causing me to skip tests, for example.

Needless to say, I managed to get things in order, switched to study software development (believe it or not, I was studying to become a teacher and a scientist), and found my passion in something I am good at. I cannot say that I am completely over my fear of failure, but I am working on it and trying to turn it into a strength.

To err is human

Mistakes happen and not everything goes as planned or expected. What I had to learn the hard way, what I was not taught at home, is what to do when something does not go according to plan. It took a long time for me to learn that, and I want to share that with others who may battle with the same problems.

Nobody is perfect. We all make mistakes all the time, and after spending some time in the industry, I sometimes feel that everything works just by bugs canceling out each other. How we treat ourselves and each other is what is most important when something goes wrong. Most of the time, the worst thing that can happen is that money is lost. While not desirable, it is far from causing injuries or death. Of course, there are cases where lives are at stake, but even there are other people involved and processes designed to avoid life-threatening mistakes.

Just before I was starting to write the blog post, I saw a video by Louis Rossman, where he talks about what he sees as a recent big mistake. I recognize myself in what he is talking about, there are times where I go back and think to myself “why didn’t I do this differently”. It is always easy to be wise in hindsight when we often have more knowledge on matters compared to when they were topical. At those moments, it is always good to ask oneself: considering all that I knew then and the life situation I was in, could I have made some other decision. Maybe there were not enough facts, and a decision had to be made there and then. Maybe my life situation was so different, that it was not possible to do anything else.

Psychological safety and errors

In my previous blog post, I took a quick look at psychological safety, which is an important feature to have in order to get rid of the fear of failure. I did not have that safety as a child, but I do have it now. I have learned that the world does not end when one makes mistakes as we all make them all the time. When the surrounding environment allows one to make mistakes and even fail at times while giving support for fixing situations, a completely different view on those things can then form.

I love making pop-culture references and there are two great quotes for this topic. The first one is in the topic, popularized by the Mythbusters. “Failure is always an option”, is what the show teaches. Multitude of unexpected things may happen, builds and experiments can fail, but what matters is what was learned from the failure and what we do next. Culture of experimentation and learning is also in a crucial role in devops: try, experiment, learn and try again if not successful.

The second quote is from Star Trek The Next Generation: “It is possible to commit no mistakes and still lose. That is not a weakness. That is life.” (Jean-Luc Picard). In the scene, captain Picard was consoling Data, who as an android expected to be superior to humanoids in strategic games, but had still lost a game, coming to a conclusion that he was not working properly and could not be trusted anymore. Often there are more variables involved than we expect, and we can do everything by the book, but still fail. Later, like Data or Louis Rossman, we can keep on thinking all the things we could have done differently.

Mr. Rossman also talked about how someone else could have done better job and not made the same mistakes, which reminds me of the devops anti-patterns of single root cause, human error and blame culture (for more information on these, see Effective Devops by Jennifer Davis, Ryn Daniels). Often there is no single reason for some mistake or failure, but rather there are multiple reasons or a sequence or convergence of situations that made it possible to happen and caused it. Human error is an easy way out of root cause analysis, but does not provide any useful information for the future and blames humans instead. When done properly, the analysis would look into what was known at the time of the event and what was the overall situation, including the internal and external factors such as long hours causing tiredness or too tight a deadline. Writing the event off as a human error also implies that someone could have done a better job, which is a sign of blame culture.

From fear to strength

I admit it, stupid and silly mistakes happen to me all the time (admit it, these happen to us all). But at work, I try to use them as a strength instead by thinking to myself “If I make this kind of mistake, then someone else can do it too. How do I prevent myself and others from doing it again”. It guides my thinking and helps me to provide a safe environment for others as well: if I try not to punish myself for them and instead learn from them, I should not punish others either, but rather teach them.

I love when a plan comes together eventually

I hope that this blog post gave you something to think about, whether it is you who are afraid of failures, or maybe it is your friend or a colleague, or perhaps none of those. As of last thoughts I want to share a couple of approaches I use all the time:

  • “The Mythbusters methodology”: always have multiple backup plans ready if and when something does not work. It does not need to be a detailed plan, often an idea is enough, just as long as it is quick to change direction when you notice that something is hitting the wall.
  • My “Sudoku methodology”: go for the solution that leaves the most paths open if the future is not very clear. This way, while the implementation does what is needed, it does not assume too much of the future and the direction is easier to change requiring a smaller amount of work.

For those who are afraid

I wanted to share my story as a conversation opener, and perhaps some of you can relate to how the fear has come to be or to just recognize the fear from yourselves. I truly hope that you have a safe environment where you can discuss it, share your feelings, and figure out how to reduce the fear (hopefully get rid of it someday). Help each other, listen to your peers, face these issues together. You are not alone!

DevOps

Aki Mäkinen

Aki is a versatile software engineer and a devops philosopher. He is passionate about bringing a better and deeper understanding of devops to clients and colleagues.

Back to top