In thirteen years of studying high-performing IT organizations, I’ve witnessed the emergence of a new and unsettling trend: whenever I mention ITIL or ITSM in presentations and briefings, people in the audience snicker. When I ask why, they roll their eyes and talk about the shrill, hysterical bureaucrats that suck life out of everyone they touch, doing everything they can to slow the business down by preventing people from getting their work done. But this simply isn’t true. In fact, I believe ITSM is more important than ever.
An even more troubling trend is the dismissal of emerging movements as passing fads—movements like DevOps. It’s my genuine belief that the patterns and processes of DevOps are the inevitable outcome of applying lean management principles to the IT value stream. DevOps is poised to change IT in a manner we haven’t seen since the birth of client-server computing in the 1980s, and the same ITSM practitioners who’ve been dismissing DevOps are the ones most equipped to support DevOps initiatives and create value for the business.
My goal here is to help you become conversant with DevOps so that you’ll be able to recognize DevOps practices when you see them. DevOps complements ITSM, and I hope this article shows you how ITSM practitioners can contribute to this exciting organizational journey.
What Is DevOps?
The term “DevOps” typically refers to the emerging professional movement that advocates a collaborative working relationship between Development and IT Operations, which increases the flow of planned work (i.e., high deploy rates) and the reliability, stability, resilience, and security of the production environment. Why are Development and IT Operations singled out? Because the value stream flows in that direction, from the business (where requirements are defined) to the customer (where value is delivered).
DevOps aims to address a persistent conflict that exists at the core of almost every IT organization, one so powerful it practically preordains horrible outcomes, if not abject failure.
The DevOps movement began in 2009, at the convergence of a number of adjacent and mutually reinforcing movements, most notably John Allspaw and Paul Hammon’s “10 Deploys A Day” and Patrick DeBois’s Agile system administration movement. Today, DevOps is more of a philosophical movement; it doesn’t have a precise collection of practices—descriptive or prescriptive (e.g., CMM-I, ITIL)—but it does have an incredibly vibrant community of practitioners who are interested in replicating the performance outcomes and cultures described so vividly by organizations like Etsy, Amazon, Netflix, Joyent, and others.
DevOps aims to address a persistent conflict that exists at the core of almost every IT organization, one so powerful it practically preordains horrible outcomes, if not abject failure. The problem? Development is typically measured by feature time to market, which motivates the team to make as many changes as possible, as quickly as possible. IT Operations, on the other hand, is typically measured by uptime and availability. Until very recently, it was impossible to achieve both desired outcomes: fast time to market and sufficient reliability/stability. Because of these diametrically opposed outcomes—make changes very quickly vs. make changes very carefully—Development and IT Operations were in a state of constant battle, with ITSM practitioners in the middle. However, ITIL and ITSM are still the best codifications of the business processes that underpin IT Operations, actually describing many of the capabilities required for IT Operations to support a DevOps-style workstream.
What Are the Basic Principles of DevOps?
In The Phoenix Project (2013), Kevin Behr, George Spafford, and I describe the underpinning principles from which all DevOps patterns can be derived as “The Three Ways.”
The First Way: Systems Thinking
The First Way emphasizes the performance of the entire system as opposed to the performance of a specific silo. The system can be as large as a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, a system administrator), but the focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified by the business or IT, built in Development, and then transitioned into IT Operations, where the value is delivered to the customer in the form of a service.
The outcomes of putting The First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve a profound understanding of the system (as per Deming).
The Second Way: Amplify Feedback Loops
The Second Way is about creating feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so the necessary corrections can be made on a continuous basis. The outcomes of The Second Way include understanding and responding to all customers (internal and external), shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The Third Way: A Culture of Continual Experimentation and Learning
The Third Way is about creating a culture that fosters two things: continual experimentation, which requires taking risks and learning from success and failure, and the understanding that repetition and practice are the prerequisites of mastery. We need both in equal measure. Experimentation and risk taking are what ensure that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone before; mastery ensures that we can retreat from the danger zone when we’ve gone too far.
The outcomes of The Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
ITSM and the DevOps movement are not at odds. Quite the contrary—they’re a perfect cultural match.
The Four Areas of DevOps
Area 1: Extend Development into IT Operations
In this area, we create or extend the continuous integration and release processes from Development into IT Operations, integrating QA and information security into the workstream, ensuring production readiness, and so forth. The specific steps in this area include:
- Creating a single “repository of truth”
- Creating a one-step build process
- Extending the deployment pipeline processes into production
- Defining roles and integrating QA, information security, and operations/CAB into the Development workstream
First, we put everything we need to rebuild a service into a common repository, including both the application and the environment (i.e., operating system, databases, virtualization, and all associated configuration settings). Then, we make a one-step build process available at the earliest stages of the Development project. By using a common build process and requiring that Development be responsible for ensuring that the code and the environment work together, we’ll have an unprecedented level of production ready, even at the earliest stages of the Development project.
This impacts the ITSM process areas of release, change, and configuration management. ITSM practitioners can actively integrate into the DevOps value stream by:
- Finding the automated infrastructure project (e.g., puppet, chef) that provisions servers for deployment. We can help that team by providing release management readiness checklists, security hardening checklists, and so forth, and integrating them into the automated build process.
- Defining preauthorized changes and deployments, and ensuring that production promotions are captured in a trusted system of record that can be reviewed and audited.
- Defining changes and deployments that require authorization, such as security functionalities that are relied upon to secure systems and data (e.g., user authentication modules). The goal is to ensure that changes that could jeopardize the organization (e.g., the infamous 2011 Dropbox failure where customers discovered that authentication was disabled for four hours) never occur.
Area 2: Build IT Operations Feedback into Development
This area ensures that information from IT Operations is transmitted to Development and the rest of the organization. IT Operations is where value is created, and other departments and divisions need this feedback to make good decisions. The specific steps in this area include:
- Making all infrastructure data visible
- Making all application data visible
- Modifying the incident resolution process and performing blameless postmortems
- Monitoring the health of the deployment pipelines
The first step overlaps with event management, while the second step requires creating a monitoring infrastructure so that there’s no excuse for developers not to add telemetry to their applications. The third step then enables IT Operations and Development to resolve incidents quickly, by ensuring that all relevant information from the entire application stack is at hand to determine what might have caused the incident and then to restore service. ITSM practitioners can help by ensuring that event, incident, problem, and knowledge management are modified to incorporate Development.
Area 3: Embed Development into IT Operations
According to The Second Way, the goals of this area are to create knowledge and capabilities where they’re needed and to shorten and amplify feedback loops. To create tribal knowledge within IT Operations and foster shared accountability for uptime and availability with Development, the steps in this area include:
- Making Development initially responsible for its own services
- Returning problematic services back to Development
- Integrating Development into the incident management processes
- Have Development cross-train IT Operations
Area 4: Embed IT Operations into Development
Area 4 is the reciprocal of Area 3, and the goal of this area is to create the service design and delivery equivalent of designing for manufacturing (DFM). In plant engineering, DFM recognizes that the primary customer of engineering is the manufacturing personnel, and therefore one of engineering’s goals is to ensure that parts are designed for easy assembly, minimizing the likelihood of putting parts on backwards, overtightening parts, damaging parts during transit or assembly, and so forth.
In a similar fashion, in addition to ensuring that the needs of IT Operations are integrated into Development’s daily processes of design, requirements specification, development, and testing, the product and processes are designed with resiliency in mind. The steps in this area include:
- Embedding IT Operations knowledge and capabilities into Development
- Designing for IT Operations
- Institutionalizing IT Operations knowledge
- Breaking things early and often
This includes embedding IT Operations resources into Development, creating reusable user stories for the IT Operations staff (i.e., deployment, management of the code in production, etc.), and defining nonfunctional requirements that can be used across all projects.
* * * * *
ITSM and the DevOps movement are not at odds. Quite the contrary—they’re a perfect cultural match. As DevOps gains momentum, I’ll be excited to see what we can achieve with this winning combination. I hope you now have a better understanding of what DevOps is and why it’s so important. Think of the possibilities! And when you’re done thinking, put those ideas and practices into place in the IT organizations you support.
Gene Kim, founder and former CTO of Tripwire, is an award-winning CTO, researcher, and author. He’s written three books, including
The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (2013). He’s worked with some of the top Internet companies on improving deployment flow and increasing the rigor of operational processes. In 2007, Computerworld added Gene to the “40 Innovative IT People Under the Age of 40” list, and Purdue University’s Department of Computer Sciences recognized him with Outstanding Alumnus Award for achievement and leadership in the profession.
This article first appeared in the July/August 2013 issue of SupportWorld.