Though this does fall under psychological safety, I considered this to be a big enough subject (especially for myself) to handle in a separate article. Please enjoy.
Fear of failure: my story
Again, I start with myself as an example. As a kid and a teen, I was good at school. I was not the perfect ten student (for those outside of Finland, our grading in school is generally from 4-10, where 4 means failure), but I was pretty good. I was also musical, and first played keyboards, then moved on to saxophone, and can say that I was pretty good at that too. Sounds great, right? The problem is that because I was good, then a lot was also expected of me. Especially by my father, who was also quite musical, and at one time played in a band. So, my daily routine was first school, then home and food, small break, homework, and then practice music. At that point, it was already evening, and I had some time for myself, though even then I was criticized for spending too much time on the computer and not doing something constructive. I lived in a small village, and all my “friends” (through the grades 7-9 I didn’t really have them) were living about 15 kilometers away, so not much socializing.
At some point, my music grade was 7, quite mediocre. But to my father, that was almost a personal insult. He was not happy, and he let that show, and I feared him. Another good case was when I got 6 from one math course in high school, and even though it was possible to improve it (and I did raise it to 8 later), it did not matter, my father was furious. Every time, when I had band practice, my father was there, and you can be certain that if something did not please him, he let me know about it later.
Why all this talk about my father, and me growing up? It is because it is the source of my fear of failure. If it was not enough having the peer pressure and other kids sniggering if someone made mistakes on the chalkboard, there was the demon at home, that made sure that I knew I had not done something up to his standards. I have always been emotionally sensitive, so I began to fear mistakes, fear failure.
I successfully got to the university and into the program I wanted. When the studies there began, I was shocked – I had expected it to be harder, but not that much. No need to say that my studies did not start well, and I made one mistake after another (e.g. tried to do too much at one time). And at some point, my fear of mistakes made me avoid situations where I could fail, causing me to skip tests, for example.
Needless to say, I managed to get things in order, switched to study software development (believe it or not, I was studying to become a teacher and a scientist), and found my passion in something I am good at. I cannot say that I am completely over my fear of failure, but I am working on it and trying to turn it into a strength.
To err is human
Mistakes happen and not everything goes as planned or expected. What I had to learn the hard way, what I was not taught at home, is what to do when something does not go according to plan. It took a long time for me to learn that, and I want to share that with others who may battle with the same problems.
Nobody is perfect. We all make mistakes all the time, and after spending some time in the industry, I sometimes feel that everything works just by bugs canceling out each other. How we treat ourselves and each other is what is most important when something goes wrong. Most of the time, the worst thing that can happen is that money is lost. While not desirable, it is far from causing injuries or death. Of course, there are cases where lives are at stake, but even there are other people involved and processes designed to avoid life-threatening mistakes.
Just before I was starting to write the blog post, I saw a video by Louis Rossman, where he talks about what he sees as a recent big mistake. I recognize myself in what he is talking about, there are times where I go back and think to myself “why didn’t I do this differently”. It is always easy to be wise in hindsight when we often have more knowledge on matters compared to when they were topical. At those moments, it is always good to ask oneself: considering all that I knew then and the life situation I was in, could I have made some other decision. Maybe there were not enough facts, and a decision had to be made there and then. Maybe my life situation was so different, that it was not possible to do anything else.
Psychological safety and errors
In my previous blog post, I took a quick look at psychological safety, which is an important feature to have in order to get rid of the fear of failure. I did not have that safety as a child, but I do have it now. I have learned that the world does not end when one makes mistakes as we all make them all the time. When the surrounding environment allows one to make mistakes and even fail at times while giving support for fixing situations, a completely different view on those things can then form.
I love making pop-culture references and there are two great quotes for this topic. The first one is in the topic, popularized by the Mythbusters. “Failure is always an option”, is what the show teaches. Multitude of unexpected things may happen, builds and experiments can fail, but what matters is what was learned from the failure and what we do next. Culture of experimentation and learning is also in a crucial role in devops: try, experiment, learn and try again if not successful.
The second quote is from Star Trek The Next Generation: “It is possible to commit no mistakes and still lose. That is not a weakness. That is life.” (Jean-Luc Picard). In the scene, captain Picard was consoling Data, who as an android expected to be superior to humanoids in strategic games, but had still lost a game, coming to a conclusion that he was not working properly and could not be trusted anymore. Often there are more variables involved than we expect, and we can do everything by the book, but still fail. Later, like Data or Louis Rossman, we can keep on thinking all the things we could have done differently.
Mr. Rossman also talked about how someone else could have done better job and not made the same mistakes, which reminds me of the devops anti-patterns of single root cause, human error and blame culture (for more information on these, see Effective Devops by , ). Often there is no single reason for some mistake or failure, but rather there are multiple reasons or a sequence or convergence of situations that made it possible to happen and caused it. Human error is an easy way out of root cause analysis, but does not provide any useful information for the future and blames humans instead. When done properly, the analysis would look into what was known at the time of the event and what was the overall situation, including the internal and external factors such as long hours causing tiredness or too tight a deadline. Writing the event off as a human error also implies that someone could have done a better job, which is a sign of blame culture.
From fear to strength
I admit it, stupid and silly mistakes happen to me all the time (admit it, these happen to us all). But at work, I try to use them as a strength instead by thinking to myself “If I make this kind of mistake, then someone else can do it too. How do I prevent myself and others from doing it again”. It guides my thinking and helps me to provide a safe environment for others as well: if I try not to punish myself for them and instead learn from them, I should not punish others either, but rather teach them.
I love when a plan comes together eventually
I hope that this blog post gave you something to think about, whether it is you who are afraid of failures, or maybe it is your friend or a colleague, or perhaps none of those. As of last thoughts I want to share a couple of approaches I use all the time:
- ”The Mythbusters methodology”: always have multiple backup plans ready if and when something does not work. It does not need to be a detailed plan, often an idea is enough, just as long as it is quick to change direction when you notice that something is hitting the wall.
- My ”Sudoku methodology”: go for the solution that leaves the most paths open if the future is not very clear. This way, while the implementation does what is needed, it does not assume too much of the future and the direction is easier to change requiring a smaller amount of work.
For those who are afraid
I wanted to share my story as a conversation opener, and perhaps some of you can relate to how the fear has come to be or to just recognize the fear from yourselves. I truly hope that you have a safe environment where you can discuss it, share your feelings, and figure out how to reduce the fear (hopefully get rid of it someday). Help each other, listen to your peers, face these issues together. You are not alone!
DevOps officer’s (b)log, stardate 202105.13
It has been truly challenging. Many months have passed since my last blog post Devops 101 – pt. 1: Journey to enlightenment. Writing the (b)log entry has proven to be much more challenging than expected, and this is the third time I write this almost from scratch. My current project has also taken most of my time, so writing this down had to be delayed again and again.
What could I say about silos without sinking into the bottomless swamp? I could state it shortly and just mention, that it is all about communication and transparency, or really go into the details and possibly bore my readers. Where is the neutral zone?
Looking at the past years, what I have seen, heard, and read, and thinking about what creates silos, there are multiple reasons. It might be a business decision. If the project is a multi-vendor project, each of them producing one component or maybe working as part of the team, business decisions may lead to knowledge being held back from the external members. On some level, this is natural in the world of corporations, but at the same time, it actively reduces transparency and can cause trust issues. “If we are holding back information X from the team Y, what are they hiding from us?”, one may think. I can understand withholding information on some level as a business decision, but at the same time, it does not help keep up the sense of trust to others.
The physical manifestation of a silo (though this one might be for wheat)
Behind hiding, knowledge can be fear of losing status, job, money, power, or having to be responsible for something. It may be abuse as well. If business partners hide relevant information and work behind each other’s backs, they may be abusing the relation to gain something from the other by not telling everything. If this kind of thing comes up someday to the other party, it can be very damaging to the relationship. “If you used our relationship and goodwill for your benefit while hiding information from us, why should we trust you?” The answer often is, “It is just business”, but it never is “just” business. It is never two faceless corporations cooperating, it is the employees, whether ordinary Joes or business managers, but it is always human beings working together and for them, trust is important. But more on trust some other time.
Something, something, something, separation of concerns. Something, something, something, devops.
While I fully support the idea of separation of concerns in software development, and in fact, it is my prime directive, implementing it over-zealously everywhere can lead to silos as well. If the team structure or organization structure is based completely on it, it may lead to having a separate team take care of security, releases, operation, development and so on. Code is just passed from one to another and there is very little communication and transparency between them. Concerns indeed have been then separated, but now there are silos.
Wisdom is to know how to use knowledge. It is a good thing to have leads for different subject matters since no one can know everything, but the work must be made visible between team members and different teams, there must be transparency so others are informed what is happening, what blockers there may be and what new features are being designed and implemented. That allows efficient cooperation early on when specialists can give feedback each other and possibly help remove the blockers.
There are other reasons as well for silos, and sadly, sometimes getting fully rid of at least some of them might not be possible. One of these situations is laws and regulations, which might require certain type of process or separation of responsibilities. While I do not have personal experience on these matters, they are something that might pop up when dealing with things like national security, military and finances. A dear colleague of mine also shared a story that budgeting could create silos, when different teams are not allowed to help each other because of budget reasons, even briefly.
The Silo Crusades
But even though silos are something to get rid of usually, hunting them down religiously can also lead to trouble. It is easy to see spies and enemies everywhere when one really tries to look for them. I have seen it resulting in attempts to force people out of their comfort zones, thinking that those zones are silos, but there is a key difference between being in a silo and being in a comfort zone, and that is the walls.
If there is a strict wall between people, then it is a silo, but if there is transparency, openness, a constant stream of information, and the possibility to move from one place to another, then it is a comfort zone. We all have them, some might be more specialized in back-end development, some are purely front-end developers, and so on. But if it is allowed and made possible for us to step outside of those roles, and participate in doing other things as well (and of course knowledge is spread by communicating), can we really talk about creating silos? From my point of view, no.
The Man, The Myth, The Legend, The Lord of Darkness
How could I then as a devops officer help the crew remove the silos? I already have mentioned communication and transparency as key features to this but there are so many moving parts. Size of the teams, amount of the teams, how people communicate and work, how people phrase things, cultures (e.g., organizational, team, national), the level of psychological safety, tools – oh, there are so many factors involved in this and there are things that cannot be forced on people.
I cannot force others to change their culture, but I can encourage them to communicate more and ask when they do not know something. I can organize demo sessions where everyone can voluntarily demonstrate what they have done so far and how something works. I can drive for faster and much more open communication methods such as Slack channels instead of email discussions.
Having one source of truth is not something to forget either. Multiple wikis or places for documentation just leads to confusion and increases memory load. Phrasing also matters. It can be very discouraging if the other side in a conversation uses sarcasm, or weaponized form of advocatus diaboli argument. Devil’s advocate is easy to weaponize into something that does not really feel like something intended to drive the discussion further. It may be intentional or unintentional, making it dangerous to use and in my books an anti-pattern to be avoided.
The Devil, (Public domain)
If we consider the following discussion:
*Developer has suggested improvements in the architecture and payback of technical debt to make the development work more fluent and safer in terms of deployment*
Manager/PO/TPO: “If I play devil’s advocate for a while, and take the role of the end-customer. Why should I as the end-customer care if the development work is easier or more ‘fun’ to the developers, what do I get out of it? What is the return-on-investment?”, says the product owner.
This argument feels confrontational, doesn’t it? There is the difference of power, one being in a decision-making role and the other being an “ordinary worker”. The argument dismisses the initial reasoning done by the developer and instead suggests that only the return-on-investment matters, which can be difficult or even impossible to calculate or estimate in certain cases. It may well be a valid point, that to someone the ROI is the only thing that matters, but this does show that the arguer (in this case the product owner) does not have the skill or will to cooperate and help combine the initial argument to what is needed to convince the other parties that ultimately pay the bills, moving all the responsibility to the developer. This corrodes the sense of psychological safety, the feeling that developers matter, reduces conversation, and instead increases frustration leading to cynicism. All of this increases other risks such as burnout and losing the employee to some other company that cares about people’s wellbeing making it anti-devops.
So, in the end, the enemies of silos are constructive discussions, a good amount of quality communication, and transparency. With the help of them, we can create better quality software and faster, without burning out ourselves. Of course, there are other factors that must be taken into account, but at least now we have a good basis to build upon.
Computer, end devops officer’s (b)log and save.
Työkalu- ja prosessilähtöisyyden sijaan jatkossa tulisi keskittyä yhteisen kulttuurin luomiseen ja ihmisten tarpeiden huomiointiin.
DevOps on menetelmä, jonka tarkoitus on mahdollistaa tehokas yhteistyö ja poistaa siiloja ohjelmistokehityksessä ja digitaalisten ratkaisujen operoinnissa. Menetelmän käyttö on yleistynyt viime vuosina kiihtyvällä tahdilla.
Goforella DevOps-arkkitehtina toimiva Jani Haapala haluaa kuitenkin nostaa esille DevOpsin käyttönottoon liittyvän yleisen haasteen etenemisjärjestyksessä.
– Usein DevOpsia lähestytään työkalut ja prosessi edellä. Me näemme, että DevOps ennen kaikkea kulttuuri ja etenemisen tulisi tapahtua ihmiset huomioiden. Kun kulttuuri on määritelty ja yrityksessä jaetaan yhteiset arvot, voidaan prosessit luoda ja työkalut valita tukemaan näitä, hän kuvailee.
Haapalan mukaan tällä tavalla saadaan luotua aito ja kestävä digitaalinen evoluutio sekä ympäristö, jossa jokaisella on turvallista työskennellä.
Investoinnit kulttuuripääomaan osaksi toimintaa
DevOpsin suosiota selittää osaltaan se, että se liittyy vahvasti moniin digitaalisten alojen suurimpien menestystarinoihin, kuten Appleen tai Googleen.
– Näissä yrityksissä kulttuurin rooli on hyvin vahva, lähes uskonto. Sen avulla kaikki organisaatiossa voivat kulkea samaan suuntaan.
Haapala uskoo, että kun kulttuuri on toimiva ja yhteisesti ymmärretty, saadaan DevOpsista irti kaivattu hyöty.
– Lopputuloksena sekä nopeus että innovaatioiden määrä kasvavat.
Haapala toivookin, että kulttuuriin ymmärrettäisiin investoida siinä missä työkaluihin. Se on erityisen tärkeää tässä ajassa, jossa yhdessä toimimisen mallit ovat muuttuneet täysin.
– Olen itse nähnyt, että niissä yrityksissä, joissa kulttuurinen yhteistyön tekemisen malli on olemassa, ei etätyö ole muuttanut mitään. Sen sijaan joissain yrityksissä kotitoimistolle siirtyminen on saattanut katkaista kommunikaation täysin, hän kuvailee.
Haapala muistuttaa myös DevOpsin perusperiaatteista, joihin kuuluvat vapauden ja vastuun yhdistelmä.
– Menetelmää ei pidä kontrolloida ylhäältäpäin, vaan sitä käyttäville tulee antaa täysi vapaus ja toisaalta vastuu täyttää asiakkaan toiveet, hän vetää yhteen.
Alkuperäinen artikkeli on julkaistu 30.4.2021 osana Suomen Kuvalehden Tulevaisuuden Suomi -julkaisua.
Gofore järjestää Aitoa DevOpsia käsitteleviä webinaareja. Juontajana niissä toimii artikkelin asiantuntija, Jani Haapala.
Recently an increasing number of new variants are emerging around DevOps. This is of course a testament to the gravity of DevOps as movement. Each variant claims to be something new and shiny. DevQaOps, DevBizOps, DevApiOps and DevSecOps, to mention a few. It almost seems like there is a new variant coming each week which claims to solve all of your problems. Have you ever wondered why that is or what is all of this fuzz about?
When DevOps was first coined, the Tech world was not the same as it is now. The Tech domain is a rapidly evolving landscape filled with disruption and fads. The required skills & competences of ten years ago often no longer apply today. The advancements in e.g., cloud computing and microservices have created a demand for new skills and expertise. The team that applied DevOps ten years ago, shouldn’t look the same today from the capabilities standpoint. That is why one of the core pillars of DevOps is continuous learning, you’re never finished learning. As DevOps gained popularity, more and more organizations tried to jump on board the hype train. This meant that organizations with different levels of maturity started applying DevOps and make it fit to their reality. Like many other paradigm shifts before, applying a tried and tested framework and making it ”fit-for-purpose” can lead to all sorts of conundrums.
Applying a best practice and making it ”fit-for-purpose” without a deep understanding of it, is a recipe for disaster. Especially if your take of ”fit-for-purpose” is to twist and turn tried and tested practices to fit your paradigm. An excellent satire piece was written about this by Ron Jeffries – We tried baseball and it didn’t work. It is often referenced in books and pieces of Agile and DevOps. DevOps includes all the required capabilities for software production. The DevOps cycle is simplified visual illustration for introductory purposes. It does not contain all the necessary information for adopting DevOps. Just because the cycle traditionally does not display aspects such as UX Design, InfoSec or Quality Assurance, it doesn’t mean that they are excluded from it.
One major factor leading up to the DevOps variants can be attributed to the ever-increasing shortage of tech experts. The digital disruption has created a reinforcing loop of higher demand than the supply. More and more often DevOps teams consist of junior level programmers and/or system admins. They have not had the opportunity to perfect their craft and are oblivious to the holism that is software production. One cannot address things they are not aware of. Unintentionally excluding critical practices from software production has led to the need to retroactively address them. There are no shortcuts to mastery.
The emerging variants are partly due to ’Fear of Missing Out’ aka FOMO, and partly due to a real need to address arisen issues with incomplete or unbalanced adoptions of DevOps. The most recent variants of <X>Ops are a whole another story, but the Dev<X>Ops variants address specific needs in a given environment. Basically, the question is – which X is suboptimal in your team or organization and needs reinforcements? Dev<X>Ops isn’t the new DevOps, it’s DevOps with a single emphasis based on a dire need. If you are now considering starting DevOps, don’t start with Dev<X>Ops – the original is still the key.
- The X in Dev<X>Ops is okay, but do not try to come up with a new DevOps – the original one still applies.
- DevOps requires highly skilled experts, so be prepared to have both Seniors and Juniors in the team.
- DevOps is a highly capable and refined tool, but you need to know your way around it.
Since DevOps as practice has gained huge amount of attention and popularity, new variants keep popping up like in the game whack-a-mole. Amidst this whirlwind, it is important to remind yourself – What is True DevOps?
In a traditional setting, a gap exists between producing value (development) and delivering that value to the customers (operations). This gap slows down the production of any customer value, and in many cases, what is being produced does not have anything to do with the real value that is actually expected.
The second root cause for issues is speed and feedback. Traditionally, programming is a slow task to complete and so is the integration and deployment of changes. Also, the chasm between developers and end customers can be too wide to provide any sort of tangible feedback for the produced solutions, at any scale.
The third major problem is that Agile has become the driving force in software development. This new Lean based methodology promises distributed decision-making and faster time-to-market. Unlike traditional frameworks like ITIL, success is not built on a command-and-control structure. This means that development teams often try to optimize their work and outcomes without understanding the bigger picture. This has resulted in major issues in the operations of these solutions. The promise of faster time-to-market is often misinterpreted as releasing new features as fast as possible while ignoring everything else.
Along came the idea of DevOps. The gap between Development and Operations is eliminated by combining the two into a single team. This team now has the responsibility and the freedom to develop and operate their software product without external dependencies. The foundation of DevOps was built on top of Lean principles and automation. The core pillars of DevOps are Flow, Feedback and Continuous Learning.
The Flow is based on Lean’s continuous flow of small batches. Continuous flow is often described as shortest sustainable lead time. Small batches refer to the fact the smaller the item size the easier it is to control the flow. The Flow in DevOps is achieved through Continuous Integration & Continuous Deployment (CI/CD) and Infrastructure as Code (IaC).
Feedback is the guiding light for DevOps. There is no DevOps cycle without fast and continuous feedback on every phase of the cycle. Continuous flow requires data-driven decision-making. Fast feedback is achieved through test automation and automated monitoring. There is no CI/CD without test automation. Automated monitoring enables e.g., proactive responses to potential incidents.
Continuous Learning is about e.g., failing often and failing fast. It also about using the received feedback to the betterment of the team and the product. Continuous Learning is mindset of never-ending experimentation in search of new and better practices, product features and flow optimization through automatization.
The cornerstone of DevOps is the phrase ’Automate Everything’. Automatization is the way of reaching flow, feedback, and continuous learning in the development & operation of a software product. Automation is present from start to finish in the DevOps cycle. Automation enables the DevOps team to focus on value adding tasks.
- DevOps is about Flow, Feedback and Continuous Learning
- DevOps is a solution to a specific set of problems when creating customer value through a software product.
- There is no DevOps without extensive automation.
DevOps has become the new de facto software development framework. Just like Agile years before, DevOps was adopted by teams. These teams were searching for ways to scale continuous integration into continuous deployment and delivery. Combining the development and operation of software into a single responsibility area is no small task. Scaling DevOps from individual teams to enterprise-level increases the complexity and difficulty by magnitudes. Organizations that strive for enterprise-level DevOps require a paradigm shift from traditional ways of organizing to a product and value stream-based approach. Basically, the Lean-Agile principles need to be spread left and right into Business and IT.
The Product Approach
Products have unique characteristics compared to other similar entities. Most notable of these is the product life-cycle, which requires forward-thinking, brutally objective examination i.e. life-cycle management. Traditionally we’ve viewed products as commercial entities, but a product can be anything, big or small, with a specific value proposition and a managed life-cycle. A product can be an internal operating system such as a CRM or something as small as a test automation solution. Shifting from traditional IT into product management requires an end-to-end mindset. Organizing your IT systems and platforms into a product portfolio is the first step towards adopting the enterprise DevOps.
The Value Stream Approach
The Toyota Production System, widely referred to as Lean, approaches the organization of work as value streams. One of the core principles of Lean is to organize people around customer value-adding work, instead of spreading the work across an organization i.e. a value stream. A Value Stream Map describes the flow of customer value as the required steps and information to produce the expected customer value. Combining the Value Stream approach to the Product Approach creates a systematic way to focus on outcomes: Product Value Streams. However, the Value Streams of digital products are not the same as the value streams in a factory. They are more like networks. Interconnected nodes with the built-in capability to work around problems, instead of stopping the line once an incident occurs. To harness the benefits of product value streams, we need to understand the interconnectedness of the products and their value streams as a network. These connections need to be managed carefully to e.g. avoid suboptimization. Organizing into product value streams is an organisation-wide exercise that requires mindful engagement from every stakeholder.
Scale DevOps to Enterprise level by
- Start thinking in products to make use of robust practices like life-cycle management
- Organize into value streams to organize people around work focused on customer value
- Understand the interconnectedness of your product value streams to manage your value stream network
Would you like to adopt agile and lean methodologies? Read more here