Understanding a world of DevOps from Richard Seroter

Introduction and Goals

Hi, my name is Richard Seroter and welcome to this course on DevOps. We’re going to be covering the Big Picture of DevOps, hence the title, really cover what does it mean, what’s it all about. For me, I’m the Director of Product Management for a leading cloud provider, CenturyLink, a book author, blogger, frequent trainer for Pluralsight, and a public speaker. So what’s the point of this course? Really it’s to get an understanding of what DevOps is, what are the principles behind it? How do you realize the actual benefits of it? What are the objections to it from outside parties or even yourself? What are some strategies for implementing this within an organization adopting some of those technologies? So we’ll talk about strategy, we’ll talk about technology together in a relatively brief course to give you a real good introduction. Whether you’re a Developer, Architect, System Administrator, CIO, Project Manager or Business Analyst, really anyone in a technology function there should be something for you here as you try to understand the impact of something like DevOps on your role in an organization. To some extent DevOps feels like the new cloud. Everybody’s talking about it, but what is it really? Is it just a rebranding of things we had before? Is it a fad? Is it a set of technologies, a cultural thing? We’re going to try to get to the bottom of that.

Organization Characteristics

Throughout the course we’re going to talk about a fictitious company called Globomatics to just give us an anchor point when we’re talking about different principles. This represents a typical enterprise, but it’s still going to be relatable. If you work for a small company, large company, global company, it’ll give you some frame of reference as we’re talking about some of the pain points that you feel in a large company, as well as some of the ways you can start to transform an organization. We’re going to take some time and talk about this company. What’s its organization structure? What does some of its software delivery pipeline look like? Just give us a sense for what this looks like. So we have some frame of reference to consider as we look at where DevOps transformation can make a difference. If you look at their organization structure, a pretty typical breakdown of business units, various units like R&D and Finance, HR. IT is sitting in the middle. I’m personally not a fan of this discussion of IT and the business, as two sort of distinctions within an organization. It’s like Human Resources isn’t typically the business of a company any more than IT is. It’s important to think of these things as all different business units in a company, but in this case, IT often being seen as a centralized thing that everyone else uses. If we double-click on that and jump into what IT looks like, this probably looks familiar to you in your own organization where you have some Architects in your organization, maybe Business Architects, Enterprise Architects, Application Architects, Data Architects. You’ve got Developers with different expertise levels. You’ve got Testers, you probably have a Quality Department, some sort of Project Management. Business Analysts who do some of your requirements work. Information Security and Compliance depending on the industry you’re in. People who function as system owners after a project becomes a system and they maintain its life cycle. And you may have centers of excellence where there’s a distinct set of capabilities that are carved into unique teams for things like application integration, collaboration and content management, ERP, things that have a designated set of people wrapped around them and they often get farmed out or they simply manage their collective resources in that center of excellence. You also probably see that some of these business units, whether it would be in the previous example things like Finance or R&D, might have their own mini IT organizations, sometimes called shadow IT where you’re doing things outside of the purview of the centralized IT Team, but often you have representations just like this, of things like architecture and analysis and project management. So let’s talk a little bit about the company. So if you look at their software delivery pipeline, complex, their silos across the organization, just because you’re delivering systems and software and applications. I think we all know that doesn’t mean that they’re an independent, they’re connected teams rather. Instead they’re often very independent silo teams with their own policies and procedures surrounding their stage of the deployment pipeline. There can be compliance policies, infrastructure acquisition policies, lots of hand offs, lots of teams inherently involved in delivering a product or service to an end user. Most established organizations also have this sort of COTS versus CUSTOM, COTS being Commercial Off The Shelf software where you purchase a lot of applications. If you’re like most enterprises you don’t leave well enough alone with those, you often actually do a lot of customization of COTS platforms. Just because you bought something like Dynamics CRM, you probably still customized the daylights out of it to meet your "special needs" in your organization. So a lot of the time even COTS software looks like custom. But you do sometimes have even some completely custom apps, often within the business units themselves to solve some unique or unmet needs. But this is a typical part, again, of most organizations where you do have this dichotomy of not entirely customs apps that just fit on any sort of platform, not just commercial software that has a particular license and deployment model. It’s very unique, mixed environments. Globomatics, probably just like most organizations, have project teams that leave after deployment. So after the software has been deployed the project is "done." You may have a skeleton crew or even a single system owner left, Developers, Testers, Project Managers, all go on to new projects. Operations may kind of remain because there’s going to be steady state, but the app becomes one of many that they maintain. Source control and probably outdated specifications get left behind, but any of the knowhow or custom tools built to deliver the solution, typically not. So, this is typical of most enterprise IT orgs who have a shared, centralized IT organization that delivers services, where after deployment the project is really considered done, even though the system is now going to go on and live for years and sometimes even decades.

Organizational Pain

With that in mind as we think about the organization, let’s talk about some of the example pain the company is feeling. Pain that you probably feel in your own organization, as you’re faced with the sort of complex software delivery pipelines in these siloed organizations. There are often just painful outages. Outages occurring on high-priority systems, a slow recovery. Instead of constantly trying to make applications that don’t fail, is really switching the focus to applications that can recover from failure quickly. So few system are bulletproof. So when an issue occurs a panic can ensue. Where are the people who know the system? We just talked about how teams typically disperse after a project is complete. So what about when the system is running, who knows it? What’s the impact? Who needs to be involved as part of any decision making process? Often as you have these sort of teams that have to now quickly come together, people who don’t have expertise on it, it creates a bit of chaos. This can even happen after a project has just deployed and the Project Team even is still around as you’ve still got this sort of panic of who does what. So what happens? Well, it’s a black eye for IT. It’s lost trust by the end users, it can IT deliver systems. It’s often finger pointing among the Project Team or the various silos who have a piece of the pipeline of the deployment or even of the system ownership. You’re seeing less incremental value in a lot of these companies where instead of these massive, multiyear projects that constantly miss budgets, miss timelines, CIOs and CEOs are really trying to realize that speed’s a competitive advantage. You’ve got to deliver value faster. We’re all used to getting buffering pages and things like that and while, sometimes that can be annoying, it’s great that I can start watching a movie before it’s done. I’m getting some incremental value even though I haven’t downloaded the whole resource yet. Many IT projects and enterprise IT orgs have these very long windows and if you’re not shipping it in a way that delivers value along the way, you’re waiting two to three years often and the original business objective has changed by the time you’ve actually delivered any value. There’s this new class of responsive, mobile, different apps where speed is of the essence. There’s no time for these long delays. And IT Ops Teams are responsible for more and more assets, often without an increase in the head count. So, you’ve got these multiyear projects that are getting shrunk down to something smaller that can deliver value, but Ops Teams are continued to be pressed to try to make things work in their existing pipelines with their existing teams, so a lot of tension there to make that work right. There’s simply the perception, at least of slow IT. Frustrated business users and different department users can’t get help from that other business unit called IT. They can’t get systems as fast as they need. They end up looking to outsiders for help and that’s where cloud computing has taken off, where people with a credit card can stand up a software as a service CRM application. They can work with outside developers and deploy the cloud infrastructure. It’s now easier than ever to go around IT. So, central IT at a company like Globomatics is simply solidifying their reputation as a laggard, who impedes progress, doesn’t enable it. And this is across teams. It’s Ops for not getting resources fast enough and waiting weeks to get servers and taking forever to deploy code that’s been sitting ready to go for weeks. Information Security, who says no to everything new. Testers and QA for missing critical issues gets a bad reputation, even Developers for unreliable shipments and slow progress. So all of IT kind of gets blasted with this poor performance label and it’s tough to come back from that. So organizations’ different departments end up looking at one-off cloud solutions, or departmental solutions that outside consultants deliver. It’s a challenge for most organizations, like our fictitious example here. And a lot of that simply leads to the infighting. It’s this increasingly toxic culture of us versus them between the IT and some of the end users. IT Teams don’t collaborate themselves and treating IT as separate from the business isn’t good either. Where you start to get this sort of us versus them or calling other users the clients, that really creates this weird, disconnected relationship. Instead of treating it as every team together, it’s every team for itself. There’s very little transparency, even less trust between some of the IT Teams themselves. So you’re not getting a good relationship between IT and some of the other parts of the business, and within IT itself you have the challenge. And sometimes this even based on rewards, the classic example of a VP of Development and Engineering who might be getting bonuses and incentives based on how much did they ship code and what kind of features can they get out there. And so their motivation is to push as much as I can. But then you’ve got an Engineering or Ops Team who is incented by stability and high availability and they’re motived to stop any deployments because they want to maintain something stable. You’ve got infighting between different competing interests and different compensation models.

Identifying Waste

So one of the things you see in organizations like this and a lot of organizations is just this heavy waste and it’s either the result or cause of, depending on your perspective, of Globomatics’ problems. You can argue waste could be the cause of some of these problems or that it’s the result of some of these relationships, but the key is it’s wasted opportunities for revenue, wasted time on non-value-added activities and more. So it’s really improvement as you’re looking at waste to name and shame the waste. What are the specific ones we’re talking about? Really waste being anything that doesn’t add value to the client. Anything the customer wouldn’t pay for. So let’s talk through a few of the different types of waste and examples of those that you see in a typical IT Department. There’s knowledge waste. What causes this? This is when you’re disrupting the flow of knowledge, you’re disrupting absorption of knowledge. Marketing may know something and not tell Engineering. Engineering doesn’t share something with Marketing. Operations and Development not talking. You’re disrupting the flow and this could be because of physical barriers, it could be through reorganizations where data gets lost as teams constantly reshuffle. It can be different professions that aren’t talking to each other, different disciplines where QA or Testers aren’t communicating well with Project Management. It can even be software systems that don’t talk and you’ve got this manual data entry where knowledge isn’t getting in-between the systems. It even can just disrupt the absorption. You’ve got constant review meetings where people simply start to tune out and they don’t absorb what’s important because they’re in so many other useless meetings. Complex reports, complex specifications where it’s surrounded by so much other fluff, you’re actually losing some of the core knowledge by people not absorbing what matters. So knowledge waste is a big problem when you’re losing key information, teams don’t have the knowledge they need to maintain systems and you can imagine the results of some of those things. It’s more time to get systems back online because you don’t know what’s going on. Or even to make changes, you’re terrified to make changes to a system because any of the core knowledge no longer exists anywhere in a visible place. Waiting waste, we’re all familiar with these. I’ve yet to meet anybody who doesn’t have some sort of waiting waste in their organization. You’re waiting for environments to be created or updated if you’re a Developer. You’re waiting for code to finish so you can test it when you’re in a Test Team. You’re waiting for approvals before you’re allowed to make changes. Customers waiting, internal teams waiting, teams on the same project waiting. Waiting waste is a huge problem and when you realize how much time you spend on it, clearly it’s a great area of focus for how can I eliminate this and add some significant value. Another type of waste is called over-production waste. This is the idea of producing more of a product than necessary. So I’m asking for more capacity than I need, just so I don’t have to go back again for example. How many times have you requested a super big, physical or virtual machine because you knew asking to go back later and ask for more of capacity was just going to be too much work. And so it’s easier to ask for more than you need. It’s wasteful, but sometimes because of the way the system is set up you’re incented to over produce. Sometimes you over produced just to keep people busy. To make sure that hey, it looks like we’re busy so we’re going to produce more than we need. Another type of waste is over processing. Simply doing more work than is necessary. I hate opening Microsoft Office software at some point, because it was so complicated. You just simply did way more than you needed to do. Other examples, exceeding the specification, right? If I’m doing more than I had to, excess capabilities, gold plating things. It wasn’t necessary. We know, I think within software most people realize that most features don’t get used and so sometimes not having that focus and you’re simply over processing. And doing more than you need to do for an individual feature than you should’ve is wasteful. Entering data more than once is this sort of over-processing waste. All of these sort of things, again, as you think about them you realize and you can identify them and these become an easier source of efficiencies. When you know that this is what this is, let’s get rid of it. There’s motion waste. This is the toll on people who create the product. So the long nights pushing software, is motion waste. As you’re wearing out your staff of people who are staying up for six hours to update software. Repetitive, manual testing where you have hundred page documents you have to run through every time there’s a code change. You’re killing your team when they’re doing that sort of thing. These are ergonomic problems, fatigue problems, you have less capacity, a less able team. It’s the waste of excess motion to create the product, service, software that IT is trying to deliver. We’ve got the idea of transportation waste. This is the waste of moving the product in ways that add no value. Clumsy movement of code between environments is a transportation waste. I’m packaging it up and zipping it and moving it out to this server and unpacking it. I’m sliding things between different environments when I didn’t have to. Any of these sort of moving costs, it’s lost time, you’re misdirecting it, you could ruin something in transit. Maybe as I’m moving from environment to environment I forgot to grab a certain config and I didn’t get that into the next environment. It’s the waste that happens while moving things around, especially unnecessarily. There’s the waste of having to fix defects, in this case having a stair to nowhere. I mean I’m rushing to hit a deadline so one team could say success, but another is stuck with subpar deliverables. How many times are Development Teams incented to ship, not necessarily incented to ship well? And so this happens when there’s just too little emphasis placed on prevention, acceptable quality limit is okay. You’re allowed to have a certain amount of defect that’s actually pretty significant. And we’ll talk more about building in quality from day one, but if you’re just inspecting things on the way out and that’s your way of doing quality assurance, your process is wrong. It should be quality built-in from the front, not just by random check at the end. And then finally there’s an inventory waste. It’s those things that pile up that actually aren’t producing income. And you might think digital assets don’t really apply here, this is more of a manufacturing concept, where I’ve got stuff sitting in a warehouse not making me money, that’s a clear inventory waste. But don’t we have the same thing in IT? I’ve got code that’s waiting to be shipped that could be adding value to my users and it’s sitting here in the source control repository or it’s sitting on a staging server and I can’t push to production because of one of the other wastes we talked about. Maybe I’m waiting or what have you. But having that sit in inventory, I might be losing revenue. I might be losing something that’s material to the business because my digital assets are backed up and they’re not being delivered, they’re sitting in inventory.

Introducing DevOps

So with all that set up let’s talk about DevOps now. We’ve talked about some of these pain points, we’ve talked about the wastes. Let’s talk about what DevOps is and how can DevOps help someone like Globomatics and then obviously hopefully you as well. So the whole goal is adding value and improving flow. How can I deliver more customer value by improving the flow throughout my organization and reducing waste? Recognizing some shared pain and trying to collectively improve some of that flow so that, frankly, I can make more money for my company. That’s really the point. The point is how can I add value, typically revenue, money, financial impact to my organization? If we take a step back and think about Lean, this is something that you’ll see in a manufacturing space and worthy of much more than a slide, but if we quickly talk about what Lean really means it’ll set things up for DevOps. So, in Lean, you’re focusing on customer value. The goal of Lean isn’t to cut costs, it’s to free up resources so you can actually focus on adding value. Significant attention on time being a very important thing, but being make sure I’m freeing up time to deliver value. It’s about eliminating waste. It’s working systematically to get rid of any non-value-added process. So I’m achieving these goals with the least amount of effort and waste. It’s really a key part of what Lean means in manufacturing. Cycle times, reducing cycle time, how long work takes to accomplish is a cycle time. How do I shrink that so that I’m able to accomplish work faster? In software this can be something. How long does it take to actually ship software and to deliver software from feature to deployment? Really the philosophy here, I mean reducing that time from the customer order and manufacturing to delivery by eliminating any of the sources of waste in that production flow. Shared learning is a big part of it, continuous incremental improvement. It’s sharing information, it’s sharing across teams, not having lone experts or siloed knowledge, but actually collectively learning together. Avoiding batching. This is concepts of work being pulled, not pushed. I’m not pushing it to each thing where it piles up, but instead having an efficient process where I’m only pulling work when I’m ready for it and what I need. So I avoid batching work, moving it from piece to piece. I’m thinking of the flow of the system so that things are really hitting each stage when they need to. I don’t do things before they’re required by the next step. And then finally the theory of constraints. Really the idea here of finding bottlenecks, not trying to improve the whole system, but going from constraint to constraint. I can only go as fast in my system as my constraints. It’s great, if I try to optimize other stuff I guess, but if that constraint, that bottleneck is keeping me from doing something, optimization elsewhere doesn’t really make an impact. I can only go as fast as my bottleneck. So secondly, you try to exploit that constraint. Get as much out of it as you can. Subordinate every other decision to that constraint, get everyone else on board. And finally, elevate it. Even throw more resources or capacity if none of those other things have worked. So that’s a quick look at one Lean is. There are courses on this. Obviously there’s a whole discipline on really adopting Lean. But why does this matter in a DevOps discussion? Mainly from the principle, Gene Kim is a great thought leader in the space of DevOps, but "IT is the factory floor of this century" was a recent quote of his in a Wall Street Journal article. But that’s a big thing, it’s not just for manufacturing companies, this concept of Lean. IT is increasingly how all business acquires customers and delivers value to them, is what Gene said. So, in his words, DevOps is the result of implementing Lean principles to the IT value stream. How am I shortening lead time? Reducing the friction between Dev and Ops? How am I doing those things we just talked about with Lean? Increasing the cycle time or reducing the cycle time rather. Removing waste, doing those sort of things, shared learning, all of those are key in Lean and DevOps is really Lean for IT. Super-smart DevOps guy John Willis explained DevOps with this CAMS acronym, Culture Automation Monitoring and Sharing. Really the idea of thinking of DevOps is a cultural movement that aligns people, processes, tech, towards a common goal of eliminating waste and increasing value. That’s what DevOps is. DevOps isn’t a product someone’s going to sell you, DevOps isn’t an edict from a CIO who says we’re doing DevOps now. DevOps isn’t a new team that’s going to go ahead and run around and automate things. DevOps is a fundamental cultural change, but also with some accompanying technologies. So, I mean the culture piece, people process has to be there first, right? If you don’t have the culture your automation is minimal impact, it might even be fruitless at the end of the day because you’re simply not going to deliver to the core principles. But automation is huge. Once you’ve been able to get your culture in line, then you can start to actually use tools to build together a DevOps capability from an automation standpoint and actually realizing the real value of some of those principles. Measurement is huge. The old saying, if you can’t measure it you can’t improve it. And so, in good DevOps you really do measure so that you can do continuous improvement, so you can do shared understanding of the health of your pipeline and your project. Sharing is huge. Sharing is about really having true feedback loops, not just Dev telling Ops things, not just Dev telling others, it’s Ops being able to feedback to Development. It’s all of the organizations, having a way to share information to improve things. And so that’s a huge part where I am sharing ideas. Where other organizations are sharing their pain, so everyone can collectively feel it. Why does all this matter? Why does DevOps matter? These surveys, you look at the Puppet Labs recently did a state of DevOps survey and others, where you’re seeing that DevOps Teams and organizations that adopt DevOps spend less time putting out fires, more time on customer tickets and actually addressing pain points. Much faster at recovering from failures, which if you assume they’re inevitable, that’s huge. Even improving the company valuation, that companies are more valuable. You can make some inferences from these surveys, because of this sort of capability. And just even higher job satisfaction. People are happier in these environments because they are realizing value faster. There’s a better culture of learning and sharing and collaboration, not of silos and head-butting. So, the goal is to make companies more money, if you’re in any team that isn’t the one generating the revenue for the company. And so as I look at DevOps, the real reason you should care about this is because it is something that many organizations are adopting. It can make a huge difference within an organization because you can spend more time on value-added activities.

Summary

So I want to start down the journey, let’s talk about this. Over the next couple of modules we’re going to go through this journey and talk about a bit of an action plan for how Globomatics, or you, tackle this. First we’re going to talk about the cultural changes, organizational changes, common objections you’re going to hear from this. Then the final, third module we’re going to talk about automation portion. We’re going to review the real DevOps tool chain, what you should be exploring, what you should be adopting. Why do some of those matter? I’d be hard pressed to come up with any organization that wouldn’t benefit from some of this. Even if you disagree or think there’s no way this can actually work at your organization or for your client, if you’re a consultant, I encourage you to take this next hour or so, challenge that assumption. If you still disagree I’d love to hear about it. This is an emerging space. Every single day I see new news, new interesting ideas, new angles coming up on this. Lots to explore, you’re never done in a process like this this is just a never-ending journey in general to continue to continuously improve. But there’s a lot of thinking going on in this space and I would love to hear back from you on your thoughts. Let’s get going.

Making a DevOps Transition

Introduction

My name is Richard Seroter. Welcome to this second module on the course on DevOps, the Big Picture. The goal of this module is to talk about some of the cultural organizational considerations for adopting a DevOps mindset. So, what do you really need to do set the foundation to properly use technology to actually enact change in an organization? In our last module we really discussed how, to some extent, speed and reliability and things like that that are a competitive advantage. And many organizations struggle with the results of a siloed organization. You end up with waste, you end up with inefficiencies. So, we talked about a fictitious company, Globomatics, and the pain and waste that they faced. We then talked a bit about Lean and DevOps as a way to focus IT on delivering value, kind of setting up this conversation.

Change Culture

So we’re going to cover three things. Virtually changing culture, is the first one I’m going to talk about. We talked in the last module about CAMS, the acronym about Culture, Automation, Monitoring, Sharing. And that was an acronym I pointed out and so we’re going to focus a lot of the Culture one here, as well as some of the Sharing in this first piece. Again, what does that mean when we’re talking DevOps?

Change Culture: Start With Why

First you start with why. You reestablish the purpose. What are the shared objectives of the organization? Sometimes that seem really trivial, but does everyone have a shared goal? Can everyone feel a shared pain? Whether it’s Globomatics or your company, are you able to detect those values and make sure that, just because a Development Team might be feeling pain, does Operations feel it? Does another business unit feel it? Is there a shared objective? Is everyone trying to chase the same thing? Or is everyone measured and compensated and chasing different objectives in the organization? So what are those values? And if you’re doing it in Agile Development for example, you can’t just do things because a book says to do it a certain way. Your values map to that. Do you value hitting dates over quality? Do you value a success of a silo versus collective? Do you think staff is interchangeable and easily replaceable? If either, if so, you either have to change your values or give up the effort. Right? There’s got to be something where underlying, you really value the thing that DevOps means or else what’s the point? Likewise, you also have to value what your organization is chasing or else you’re never going to achieve that sort of shared success.

Change Culture: Empowerment

One of the first characteristics you’re looking when you’re changing the culture is empowerment. Am I empowering employees to do what’s needed to maintain the service? At any point, can Developers or Engineers stop the alarm, sound the alarm, stop. Say, you know what? We’re going to stop delivery. This isn’t of the quality I expect. Or, I see an issue. Can they do that without getting yelled out or fear of speaking up? Are they empowered? Can you trust the team? You have to let them take some ownership. And that’s really a lot of this, is empowering individual groups. No more victims, no more, hey I couldn’t do it because this other group didn’t let me. Instead, you empower teams to form in a crisis, use their judgment, do those sorts of things that you’re hiring high-quality technology professionals to do.

Change Culture: Accountability

Along with that though, you have accountability. If you mess up, you’ve got to take ownership of that. You can’t really have empowerment without accountability and so you hold teams accountable for the product, the project, the entire service success. There’s a composite team that maintains collective responsibility and accountability. And a lot of this is a focus on quality. And that’s a big part of Lean and that sort of mindset, is your focus on quality from the beginning. You’re building this in from the start as well. You’re not relying on a QA or even a Testing Team to inspect and catch defects, you’re investing in that upfront and you’re making Developers accountable for delivering good code. You’re making Operations accountable for all the various environments. Whether it’s Dev, Test, or Prod, you own those things. Take an ownership of it. It involves that sort of commitment to excellence. You have to embody and protect it. The team has to be accountable to each other as well, where there’s actually a shared social pressure to step up and deliver and be accountable and call out bad form or call out some mistake. Because everyone’s accountable together for this and individually, you also are taking some ownership. And it comes down also to rewarding those who deliver or take responsibility. As a manager in an organization, you’re rewarding people for shipping, for execution, for stepping up and fixing a problem, not just for coming up with ideas.

Change Culture: Teamwork

Teamwork is a huge part of the cultural aspect of what it really means to be doing DevOps. This is across teams, this isn’t one team does a nice job. This is Development, Project Management, Operations, Testing, QA, all working together both in quiet time and in crisis. So this isn’t, we’re a great team when we all have to shove a project over the finish line over a weekend. It is, we work together at all times and we work together when there’s something bad and something great. As a team, and as a manager, and as an individual you’re looking for excuses to put the team together for planning efforts, for daily standups or things like that. Brown Bags, where an Operations Team may explain some new hardware they’re putting in or some new layout, educating the Development staff of some of those implications that their code may have. Developers may walk the rest of the IT Organization through a new platform feature or a project they’ve launched or a service that they’re maintaining. Take lunch breaks together, even as a Developer, grab an Operations person for lunch and catch up on them. Who are they? What’s going on? Co-mingle. Really, by doing this you start to respect each other’s skill set and realize that they bring unique value to the table. So it’s not just us versus them within IT. It’s truly a, we all have some different skills and there’s actually a lot of value in this. And we aren’t just, hey it’s their fault, it’s your fault. Instead, it’s collaborative. What that means is that things shouldn’t break down then when emergencies happen. Teams can quickly come together. It’s a natural formation of teams, because now you’ve already formed this. If something goes wrong, it’s really easy to use that previous collaboration and some of that deep relationship to actually bring a team together.

Change Culture: Learning

Learning, continuous improvement is a huge part of what it really means to be doing DevOps and that culture is important. As an organization, if you’re not valuing learning, if people are expected to truly just learn things on their free time and if they happen to pick up a new skill that’s not acknowledged, that’s an issue. Because really, you want people to be sharing their knowledge as they learn it. At my own company we do Brown Bags with new technologies, both on the Dev and Infrastructure side. Ops Team will go share something new that they’re looking at. Developers, the same. It’s all optional, but really, there’s great attendance on both sides because there’s curiosity and we realize both teams are doing things interesting. Doing postmortems when things go wrong. So instead of shying away when something fails and everyone kind of quietly not talking about it, embrace that as a learning opportunity. Invite a broad audience. Learn from what you did so it doesn’t happen again. You have to give the teams access to resources it needs to learn. Hey, a Pluralsight subscription, is a great way to facilitate a learning organization. It’s not a huge expense, it’s a way that people can actually learn and share. Conferences are really important. I know it’s easy to do things online now, and never attend something in person. But, I think most of us realize that there’s a lot of social aspect to a conference too. You’re brainstorming ideas, you’re simply absorbing new information in a free setting, where you’re not competing with other work responsibilities. So, encouraging some of that. Likewise, encouraging public profiles and engagement where people can continue to learn from the public. Don’t lock your people away in a fear that if they learn a lot, they might leave. Instead, it’s about having a learning organization where you’re encouraging people to experiment. Our team had a hack house, or has one once a year where we go and live together and learn new stuff and try to deliver features, but it’s really both a teamwork, as well as a learning opportunity. Use these sort of places. How can I put my team together to kick off proof of concepts of new technologies? Encouraging that fail-fast, which is a big part of innovation.

Change Culture: Trust

Trust is a big aspect. And it’s going to be, how am I going to be successful if I can’t trust the other pieces of the organization? And realistically the other parts of the supply chain. Again, I’ve got this huge pipeline that goes from business idea to technology solution that helps solve it. And if all of those people aren’t in line, if there’s not some trust between them and some relationships between them, I’m going to struggle with that. And of course, also it leads to happier employees, more efficiency. When there is trust and it’s not constantly this battle of hey, did they give me this specification knowing it wouldn’t work right so my team might look bad? Or, did they give me half-baked code that’s not going to run in production so they can meet a deadline? When I trust that everyone’s doing things in the best interest, on the same page, big value add there.

Change Culture: Reinforcing Values

And one of the big pieces as you wrap all this up, is about actually sharing, living, rewarding the right values. What values do you care about as a company? I can tell it really easily by seeing how you promote people, how you fire people, how you hire people. That demonstrates really easily to everyone what you actually value. You know, if the people getting promoted are the ones that are there a while, okay, I mean you’re telling me you value longevity, versus potentially individual excellence. Because it’s hey, it’s whose been here the longest. If you reward the person who does a lot of work, but a lot of it’s not high value, but you like the volume, again, you’re easily letting the other team see that. If you badmouth another team, so will your employees, as a manager. If you reward firefighters, people who come in at the last second and rescue something, even though they might have caused it in the first place. You know what? Teams won’t worry about consistency, because you reward people who do heroics. If you reward people who do quality work, that does get noticed. So, it’s not just a memo that goes out that says, we now will value collaboration and teamwork. It’s proving it. It’s as a manger, making sure that you’re interchangeable with your counterparts in some of the other IT Teams, it’s making sure Dev and Ops really are on the same page. If you’re not, it’s really obvious. So, it comes down to the whole team to actually live these values, management to reward those same ones, and teams to share them between.

Change Organization

So we just talked about changing the culture. Really next up is changing the organization. How does the organization now make some of these changes, so that you can realize the value of DevOps from some of those cultural changes?

Change Organization: Gain Understanding

Three things I’m going to briefly walk through. First, gaining understanding of what you’re working on. Altering the team structure potentially, and streamlining some of your procedures. Some of all these coming together to change the organization to set yourself up for DevOps success. First is having clear vision, really gaining understanding. You can’t automate or optimize what you don’t understand. So, it’s building empathy for users, understanding what’s going on in the system. DevOps Leader, Jeff Cessna, mentions that you can’t design anything truly useful unless you understand the people for whom you’re designing. You can’t skip right to automation tasks, where hey, we’re doing DevOps, we’re just going to write a bunch of scripts. That doesn’t make sense. You really need to learn to "see the system." And this is focusing on different perspectives. What’s the value that this system provides? What’s the waste that it has? What’s the process? What’s the flow? So looking at things from a waste perspective, we talked about that in the first module, all the various types of waste. You could look at value. What’s the expected value of this particular system and flow? What’s the specified value that’s been asked for? What’s the delightful piece that makes someone happy? Doing process, things like process maps and value stream mapping. Look at the process to see where, sometimes you have necessary waste that’s part of the process, that’s fine. But where are there steps in the process where you can actually add value to the customer? Where don’t you? And looking at flow, do I push or pull work? At which steps do I do that? What’s the flow among Dev, Test, and QA, Production, Support, Projects? How does everything flow like that? You’re really looking for how does everything work together? What’s the logical architecture? What’s the process architecture? How do you explain that everything’s working right? You can’t do that, unless you look and see the whole system. And then even repeating that process for the people in groups involved in supporting that system, so really getting a clear vision of what you’re working on.

Change Organization: Recognizing Bottlenecks

It’s okay to understand those bottlenecks. Identifying the constraints, trying to get the most out of them first. Realizing that, maybe your first step isn’t that you just throw a bunch of money at people at the constraint, you try to actually, how do I get the most out of it? How do I have other things that may give some resources to it? Or, how do I change my process? Because, this is, maybe QA is just really complicated and there is only so much you can do there at a certain speed. So how do you do things that make their process easier? Worst case, you elevate the constraint and do something else. These bottlenecks can be technical or social debt, they can be people, they can be tools, they can be outside parties. All those things can be constraints. Let’s talk through a few examples that you see in most organizations. What are some of those things that lead to bottlenecks, because of the way the organization has set up some IT functions? Things like inconsistent environments are a really big bottleneck. So, if you’re Globomatics you probably have Developers who don’t have production-like environments to develop against. So you get the typical, hey it works on my machine. Or you’re doing everything as a local administrator and then surprisingly it doesn’t work in production. But why is that? Why can’t we do those things? They may not be the same in size, you’re not going to have a production quality data center on your desktop, but why isn’t the configuration the same? We waste so much time getting code tested and deployed correctly when each environment is different. So where’s the bottleneck here? Often it takes a long time to get these environments. Can you easily recreate production and development? This is a bottleneck often for Developers who are waiting to either get that right environment or it becomes a bottleneck in deployment, where now you have to do all of this testing and extra work because you didn’t have everything done upstream. So, it could be a bottleneck for Devs, who aren’t getting what they need. It could be a bottleneck for Ops, who can’t turn the system live until they’ve properly got it working in this new, different environment. QA becomes a bottleneck when they hit defects that Developers can’t reproduce. Because QA may be working in a production-clone environment that Devs are. And so now they become this bottleneck because they’re that first stage of actually testing this in a legit environment. So some of the solutions for this, looking at the technologies we’re going to talk about in the next section to help you maintain a consistent environment at every step. Development, working with Operations to use different cloned environments on their Dev work stations and in each subsequent location. Everything should look the same everywhere and that’s a lot of the responsibility of Ops in a DevOps process. That’s what they own. Ops working to deliver on-demand environments. I need a perf test environment for three weeks, that shouldn’t be a big deal. That should be something we can create and configure very, very quickly. Another place you end up seeing bottlenecks is when you have a lot of custom manual builds in your organization. There are systems that companies, like Globomatics, that people are scared to touch or update. But how fast can I deploy code? Is there an inventory of useful features, sitting in QA or in Source Control, that has to wait for scheduled deliveries because it’s so cumbersome to do an update? Is that the bottleneck that’s caused by this, where because you have these long-running custom-builds that are kind of unreliable and inconsistent on each time, the impact of that is I build up this inventory. I create a bottleneck. It doesn’t matter if I develop any faster or I do QA any faster. Everything backs up behind the deployment process. So if I’m throwing money at adding developers to my team and I keep slinging more interesting code, but it keeps stopping, at that point where I do deployment because that’s where my bottleneck is, I haven’t fixed anything. Now I’ve just put more behind that bottleneck. So some of the solutions are some of those continuous integration, continuous delivery tools we’ll talk about next. Collaboration between teams that have responsibility for aspects of deployment pipeline. So that Information Security, QA don’t accidently become bottlenecks because they weren’t aware of what was coming. So, this constant collaboration where people can pull the work they’re doing, you’re not just shoving things over walls, surprising people. Instead, you can eliminate these bottlenecks by having some automation, having some better communication. Poor quality definitely results in a bottleneck and something you see in organizations. Without proper accountability for shared excellence, Devs may hand off code that has known bugs or is incomplete. This is often done to hit a deadline or because the mentality, that hey, Testers will catch it, I don’t have to worry about doing everything at the lowest level. I don’t have to do the best unit testing, I don’t have to do my own integration testing locally, someone else will do it. The impact is QA and Testers become the bottleneck again. So as deployments fail and there are no automation scripts and things like that, they’ve got to run through their own code inspection. They have to look at these things. So when poor quality is there, you again, end up with this bottleneck where I can’t continue to move code through the software delivery pipeline because it’s going to get stopped at multiple places. Or, worst case, it hits production and I’ve got this defect waste of I’ve got to go back and fix things. Inevitably this causes problems. So the solution to this is quality from the start. Rigorous integration testing, where Developers can’t commit code that fails to meet certain standards. Baking in more and more quality upfront, where it’s not relying on last-second inspection to actually check and see if this is good stuff or not. The final type of typical bottleneck you see as we talk about understanding the system and recognizing these things, is a lack of communication. When teams don’t share their plans, they don’t share their objectives, even on a project-by-project basis. You end up with this system where things just overall push to each environment, where teams aren’t pulling work as they have readiness for it. Instead, things are just pushed and shoved to them. The impact of this is code may be ready to go and there’s no servers ready yet. Devs might be waiting on work to do, because upstream processes haven’t actually handed over enough requirements for them to work on. QA has nothing to test, but then all of sudden gets a deluge of things to test and they have three days to do it before a production go live date, because work is batched. When Developers often see the outcome of their work, via the feedback, the monitoring, the other parts of what DevOps means, they do better work. When I see how this has to be managed in production, when I see some of these things, communication, that feedback loop, is huge. I don’t want siloed tribal knowledge. I want to make sure that teams are communicating their plans, what they need to be successful, and that makes a big difference as each team that’s upstream in that pipeline is going to communicate the goal, for them to communicate better and be very transparent. When you’re doing that, when you’re changing the organization, everyone understands then better what’s coming, what to expect, and you have a much cleaner flow through the system.

Change Organization: Alter Team Structure

The second step as we talk about changing the organization is potentially altering the team structure. And what does that mean? So, a typical silo team, this is what you have in most IT organizations, that I’ve at least seen. You have these share pool of Developers who work across projects and after deployment go work on another. We talked about that, that’s a problem our fictitious company has, is you have people who leave and you’ve lost all your tribal knowledge. They leave and you’ve often lost some of your tribal knowledge. You often have teams of specialists. People who have very specialized skills and get brought into certain projects, not generalists. Even startups have roles that get carved out into this above org. This has nothing to do with large team sizes. If you have a 10-person technical team you probably have carved things out to some extent like this. Now sometimes as projects get deployed, you still have a skeleton team that looks somewhat like this. Or, you end up with just a system owner who is responsible for a lot of things. Ops is typically pulled as well. One admin responsible for dozens of servers, dozens of apps. You can try in this sort of model to increase the feedback loop. And that’s going to be important. If you can have a good feedback loop, you have more tribal knowledge shared. You’re going to have more trust. You’re going to have more collaboration. But this is a tricky model, but this is how a lot of organizations are set up. Recently Adrian, at a conference in the last few weeks, made this statement, "DevOps is a re-org, not a new team to hire." So, some believe that you go hire a DevOps Team and if you have a DevOps Team, it doesn’t necessarily mean you’re doing it wrong. Some people think it means you’re completely doing it wrong, because the whole point wasn’t to create yet another silo, it was to actually infuse the idea of DevOps across your teams and have these kind of collaborative, virtual teams that all work together. Not just have a new team that tells people what to do. But there’s no real wrong answer here. You may end up keeping all the same people, even the same teams, but change the philosophy. That’s what a lot of people do, promote in this. But you do occasionally see DevOps Team popping up, but really if you think about it more as a reorg, not as I have to just hire a new team, that can change your thinking. So some of the thinking in this space, is that having more permanent teams around services, treating these things as services, that on delivering, not projects. What’s a project? I mean, a project may be towards a service, but I’ve got something that’s going to live for weeks, months, years, decades. And that’s a service that I maintain. So it’s not a deploy and disband sort of model, but a purposeful organization of teams around internal products. You retain knowledge, you do quick acting. Guess what? I need a lot less documentation, when my team is consistent. When I’m not writing these 400-page specs so that the next poor sap who gets ownership of this system knows what to do with it. When I have a team of Developers and QA, and all those who retain some ownership of that system and service moving forward, I don’t need so much documentation. I don’t need all this, sort of extra material. Instead, I have that retained knowledge. I can act quickly. If there’s a problem I know exactly what’s going to go wrong. I know exactly who to bring in. Sometimes this works better with more generalists than specialists. Specialists encourage silos. I can only participate in one unique way, I only own one capability. Instead, is you have generalists. People who can work across services if need be. People who can own a capability when they develop it end-to-end. But thinking this sort of way is interesting. Now obviously, this is not a fit for every service and app in an enterprise. You’ve got small, departmental apps that can’t justify a team who stays around it. That requires a lot more headcount, which it may in this model. But also could deliver more value as you look at your bigger systems and services. And how do I rearrange, maybe some more persistent teams around them. You may, to do this successfully, most likely will have to incubate this, in parts of the organization. Not doing this organizationally wide. Heck, my company is doing that right now, is we’re trying to reorient our various product lines around specifically maintaining dedicated teams instead of these pools of resources who bounce around the various products. Actually organizing teams at the broader company around the service lines, so that you do have consistency, shared knowledge, quicker delivery. Actually putting Ops and Dev and all those teams together. So, we’re trying to live this right now and there are some people doing some good thinking in this space. It could be radical for you, but it also may be a way to incubate some of the ideas of DevOps in your organization. Even if you can’t make that sort of dramatic change, at least have Project Teams with all of the functions with the same objectives. Not as many of these hey, we’ve got 30 people on this team, all dedicated 10%. It’s really tough to pull that off. So, can you really have every group represented, but with actually more time committed? Are they involved in planning? Is there transparency for each other’s plan? Do they define success in the same way? As you’re bringing in consultants and internal teams, you have this mishmash of matrix teams, often with very, very different objectives, especially as you bring in outside parties. Is that the best way to work here? No, not typically. Especially as I’m doing DevOps, I need a clean communication channel, I need shared objectives, and I need everyone working on the same page. So, I want the whole team onboard with deployment processes. I need to identify gaps in automation, and have the whole team work together to fill them. Have people who don’t just airlift to the end of the project and derail it, because they don’t understand the flow. Have a higher commitment, more people involved, more dedicated on these big teams. And the last piece of this, if you almost take nothing else away, consider the handoffs. You’re part of a pipeline. If you’re in Development, you have a customer downstream who is most likely not your end user. A customer of Dev is often Operations. Maybe upstream it’s the other business department who has needs, but it’s rarely, sometimes not the end user. Operations often interfaces with the end user. So, just always think about your handoffs. Think about the post-deployment. What happens after this project goes live? Ideally, it’s the same team that built it that works together. But even if you don’t have that, make sure you’re always empowering the next person in line, streamlining things, giving them what they need so that they can be successful.

Change Organization: Streamline Procedures

This last part of changing your organization really relies on streamlining procedures. And so how am I taking some of these procedures in my organization, streamlining them a bit to actually enable DevOps to work? And not have to have the same cumbersome processes that were sometimes put in place before you could really do some of the automation, before you could really have this sort of collaborative environment, actually build this up. So first step, really is some of the self-assessment. When seeing the system, make sure IT is part of that and the processes and procedures. Turn the mirror on yourself. So as we talked about earlier, look at the flow between Dev tests and QA, and Prod, Support. How projects interface. Look at how you do change management. Look at how you do deployment management. Look at how you do any knowledge management. How are you storing information and sharing information? How do you do incident management? What do the policies and procedures you have around some of those things, how are they either encouraging or discouraging the flow of value through the system? If you’re recognizing when you’re doing value mapping that you’ve got all these cumbersome processes and procedures, that were in place to maybe protect yourself a long time ago, but now are really a hindrance, really have the courage to inspect those. And see where they really do and don’t add value. Which really comes down to eliminating waste. You know, you may have policies and procedures that you need to eliminate. Now again, it may take technology demonstrations, it may even show practical demonstrations before it’s possible to go remove policies and procedures. I don’t expect that you can come in and just say we’re not going to do this anymore. But looking for those and saying we need to prove that we don’t need this same sort of rigor around this. You may have procedures today around requesting hardware, new or upgraded. Requesting a change, adding Developers and giving them access, getting monitoring alerts. Do you have policies around production access, remote access to resources? Things like that. Are they adding artificial delays? Do you have excess approvals, to kind of cover yourself because you need this audit trail, manual double-checks. As we talk through the automation tools next, think about each one helps eliminate a waste you already have in the policy and procedure realm. Sometimes those again, were put in place to make you feel better. But in essence, they can be removed by automation, a better technology audit trail, better quality that now they’re just getting in the way. Now they’re just preventing you from pushing value faster. You’re waiting for all these approvals to ship valuable code to a production mobile app. Can we get rid of some of those? And that you’re only able to do it when you can prove that you’re delivering the right value and the protection and the compliance. And we’ll talk about that here in just a moment.

Addressing DevOps Objections

So we talked about changing the culture, changing the organization. Now let’s talk about the objections. This is not all unicorns and rainbows. There are things that you can listen to this and go, some of this sounds great, some of this is completely unrealistic. So how do we address some of those, both in a good way and bad way? Not every answer is going to be well, just be quiet, DevOps is great. You do have to ask hard questions. The problem with many IT Shops is that they all think they’re special. Hey, we have special technologies and processes, our clients are different than anyone else’s. But in reality, it’s not really the case. You know, a lot of IT can be interchangeable. Instead of treating yourself as, I need all these special policies and procedures because we’re unique. Instead you almost have to admit to yourself, let’s assume everything I’m doing is wrong, and then how would I do it better. And be open to change. So at least keep that in mind as you think of your objections. Am I doing this because I think we’re unique, or because I’m scared of change, or because there’s a legitimate reason this wouldn’t work for us? The first one is security. So clearly when a lot of people hear about DevOps at first, they think, I can’t have Devs touching production. There are all sorts of information security problems, code security problems. But just not, this is a nonstarter for me, I can’t have a bunch of Devs who now have production access, I can’t have things flowing willy-nilly through the system. But really in the response of this, there are a lot of dimensions to this. Focus on quality, reduce the security vulnerabilities. When I’m focusing upfront on good quality, I’m probably taking better care of what I’m building. Focusing on teamwork and communication means Info Sec isn’t the last second partner, instead a key team member upfront, that’s part of the collaborative model, it’s part of the communicative model. Is making sure people are brought in early. That you’re assessing this throughout the pipeline. Pushing code quickly means any vulnerability can actually be patched fast. There will always be defects. It’s almost impossible, I don’t know if we know of software that doesn’t have defects. But when I have a streamline pipeline I can find the problem, fix the problem, push the problem fix in minutes. And that’s a huge difference and a huge security win. Automated, mirror test environments mean that I can adequately do security testing well before production. Doing proper penetration testing can happen in all these other environments, because they’re actually clones of the same configuration, permission model setup as production. So again, DevOps if I do it correctly, can give me so many more ways to test prior to the last second. When I have instrumentation in my code, it helps all Service Teams track down issues faster, including security ones. So again, that sort of model is making me more secure. In reality, DevOps shouldn’t open you up to security issues, it should actually close most of them. So, while there are legit reasons for some of these things, recognize that actually you end up more secure in many of these environments. Now the related one to security is the compliance piece. I can’t be compliant if everyone’s touching everything. You know, I have audits. I have to do certain things. That’s why we have all these policies and procedures that you’re checking off that you’re doing things. That four people have approved something. But again, many of the same answers I just gave for security. Compliance frameworks often look at information security, access, audit history. But most of the time legacy processes were put in place because it was impossible to track down changes across systems. When you have streamline automation and few or more meaningful policies, you should have less chance of being out of compliance. How many of us have taken these courses internally, where you’re supposed to read a document or check off that you’ve read this compliance or this policy, and it’s this massive, cumbersome thing or there’s 50 of them you have to read on a Friday. I’m less compliant when I face that, because I’m simply not absorbing it all. There is a knowledge waste there where I’m reading through it so I can check it off and say I’m done. Versus few or more meaningful polices, I’m going to absorb that more. And so as I look at compliance and automation actually can make compliance much, much easier. By having true audit trails, by having true security controls in place, giving me the right least privilege in my systems so I can still prove that certain people can’t access certain things. Unless the application’s monolithic, I can do this least privilege approach. I can have security zones where access to the most sensitive information is forbidden. It doesn’t have to be all or nothing. Where my Devs have production access to sensitive patient data in the Healthcare Industry, or financial information if I’m dealing with PCI. I can segment certain things so that I still use role-based access to who can do what. Just because they can’t access a certain piece of data, shouldn’t mean that they can’t push code to production using automation tools. So really siloing and segmenting out who can do what, where, doesn’t have to be all or nothing. I can still be completely compliant in a DevOps environment. The next real concern is remote teams. So let’s say co-location is not possible. I’ve got an offshore team, I have nonlocal teams. I’ve got some things outsourced to different providers who are my operations, things like that. You know, response to that is that definitely can be tricky. I do it with my own team, I’m remote, where my Engineers and Operation staff are mostly centralized. It’s not ideal, but a shared culture and vision does make it more possible. Technology makes it more possible. Where I can actually be collaborating with this team in real time that does make a huge difference. But if Globomatics, our demo company here, or you have this organization where any major function is completely handled by some sort of faceless third party, frankly it will difficult to optimize your entire pipeline. If you know you just hand things over the wall to this partner, who actually owns your infrastructure, and you have no say in how that’s managed, that’s a challenge. You know, given the theory of constraints, you may actually be optimizing areas that aren’t the bottleneck. You may be doing a bunch of things in your org that aren’t moving the needle, because that next part in the pipeline can’t deliver as quickly as you are. So you may need to change either the nature of your partnership or regain control of aspects of that pipeline. Even though they may own the infrastructure, maybe you get the permissions possible to do application pushes on your own, without just handing over code or things like that. But, remote teams can work in a DevOps model, but everyone still has to follow those other cultural changes to make it possible. Another objection you’ll hear is the Ops impact. What is this impact in Operations? Is this a sort of no-ops model where I have to fire all my Admins since now everything’s done by robots or done by developers? In reality that’s not the case. I mean, historically of course, Ops roles, most IT roles are constantly changing, those roles do. But Ops takes on the extremely important role of delivering the environments everywhere. Giving production quality environments for Dev, Test, Prod, and others. Configuring those, monitoring, maintaining, tuning. Teams are still working jointly to deliver this pipeline. Most organizations are not asking Developers to suddenly know how to construct an optimal network topology in the cloud, or on premises, or how to troubleshoot all server OS issues. DevOps isn’t about Developers now simply doing everything. It’s about both teams understanding their part in the pipeline, having those right feedback loops, and optimizing each other, as well. So, Operations may not be loading punch cards or racking servers as much anymore, but they’re still keeping that infrastructure running and at scale and working on these systems and services and delivering things as part of this pipeline. It doesn’t mean to make them any less critical. But it does potentially have a change in their role, and may modify, to hopefully more value-added activities. There’s definitely the legit question of legacy systems. Hey, these new-fangled stuff can’t work with my crazy-cots product or my legacy platform. I can’t do this at all in this new model, I’m stuck with these old things. I guess it will only work for new systems. You know, large companies can’t add new code or string new code to systems, often because they haven’t built the necessary flow in front of the production release. Ignore the technology, do you even have the proper flow leading up to those legacy systems to release stuff to effectively, efficiently test and verify these code changes? Probably not, right? I mean, that’s one place you can continue to optimize even if you can’t have automation tools to update your legacy platform. You can still do some continuous integration upstream. You can still do some more-scripted deployments, even with older environments. And in a DevOps culture you’re still trying to build Service Teams that have knowledge and shared ownership of the service. So you can still do things like introducing monitoring, having teams that are really maintaining this service in a different way. Even though it might not be the most modern app that supports one-click deployments to production, It’s not just about pushing code to prod. One aspect of DevOps is about increasing that flow of work through the system. And that’s by optimizing all the different places where value is added. And that can happen with a legacy system or a completely modern system. So don’t always just think about, I can’t take this system and update SAP live in prod. Therefore, it’s not part of our DevOps cycle. Baloney, you can absolutely do that. Make sure you’re optimizing all of the aspects of DevOps, not just thinking about code pushing. A big, reasonable objection is, my team doesn’t know any of this stuff. We’re not some hipster startup who loves using Chef, and Puppet, and writes Node.js. We’ll leave that for them. We’ve got all our nice, old legacy processes. My response is kind of, frankly, learn it. I mean, this is exciting stuff. Some of these skills you do already have. Your Admins are probably pretty good at scripting. And that’s some knowledge they can either share with Development or simply have that as something they do well like configuring and scripting some of these systems broaden that. So that becomes part of what they’re doing for development environments and others. Some of these skills are soft skills. This is better communication, teamwork, continuous improvement. This isn’t hardcore technical skills for a lot of these. We haven’t talked about any hardcore technical skills yet. We’ve talked about things around how do I frame problems? How do I understand the whole flow? How do I communicate across teams? How am I constantly learning and thinking about improvement? Anybody can do those things. This doesn’t require a 12-week course. So it’s about collective learning and applying it immediately. You know, we’ve all taken training, of course not this course, but we’ve taken training and you haven’t had an excuse to use it right away, and forgotten it three weeks later. No, instead, make sure your team is learning and applying this interesting information. Build that skillset. It’s an important thing for you and your team, that shouldn’t be a barrier to adopting DevOps.

Summary

Alright, I hope you enjoyed this module. We talked about changing DevOps culture by focusing on why. Changing accountability, trusting each other to work towards a shared objective. We talked then about changing the organization, bringing the view of the system out in the open, clearly identifying bottlenecks. We talked about changing the team dynamic, how do we think, how do we maybe rearrange more around services then just this sort of pump and dump strategy of hey I’m going to do this project, I’m getting out of here, I’m working on the next one and moving on. Instead, how am I delivering persistent value. Then we discussed some objections your going to hear when adopting DevOps, you’re going to hear a lot of those when adopting any sort of major change in an organization. But when you see the value, when you see how companies are more valuable when they’re using a DevOps mindset your employees are happier, your systems have less defects that you can fix faster, really important. The next module is one of my favorites, this is where we go through the technology that underpins a DevOps tool-set. What are the things you can practically go look at today to learn more about how DevOps works and start optimizing your company.

Introducing DevOps Automation

Introduction

Hi, my name is Richard Seroter. Welcome to this third and final module on a course on DevOps, The Big Picture. In this one we’re going to bring some things all together and talk about some of the automation components of DevOps. Specifically, we’re going to talk about some of the DevOps friendly technologies that complement the cultural changes, organizational changes, that we’ve talked about in previous modules. What are those things that really start to bring that into focus? If we rewind from the previous modules, we talked about the cultural and the organizational changes needed. We talked through a lot of those. That you can’t do technology alone if you don’t have the teamwork set up. If you don’t have a team structure set up, that even has some of the shared knowledge and some of the system ownership and accountability and empowerment that’s necessary. What we’re focused on here, on continuous improvement. How am I using technology to continue to help with improvement? I don’t like the, if it’s not broke, don’t fix it mentality. That doesn’t work in DevOps. DevOps should be about automating repetitive tasks, the feedback loop between Dev and Ops and other teams, always learning. Just because something’s not broke, doesn’t mean it’s optimized, that there’s not tons of waste in it or that it doesn’t actually work well in the whole flow of the system. So, we should be constantly looking at improvement and how do we grow? How do we make sure we’re doing things in our system? Hopefully often now, using technology to make that possible. The stages of maturity, as you think about this progression of technology, you go from using just source control, and sometimes that’s a big step. You might just have source control sitting on a file share somewhere, on your local machine. So even using a centralized source control is a good first step. Then, having builds actually triggered by committing your source control and doing integration testing right there. Then moving into some automated deployment to testing. Have that next step of maturity of actually taking these builds and putting them somewhere. Then actually automating some functional testing and having that baked into your process. And then really getting to this point where you can do continuous deployment to production. So as we talk now about these tools, keep some of that in mind.

The Tools

So now we’re going to talk about the toolset. We’re going to talk about, I’m really trying to put this into a reference framework where you can think about each of the sets of tools. DevOps is not just, hey I’m using tools to configure servers, which you may think of as just the first thing that comes to mind when you think of DevOps. Instead, what are all those categories that touch on all the things we’ve talked about so far in this course? You’ll see the technologies that complement each one of those focuses.

DevOps Technology Categories

So as we look at that reference diagram, you’ve got collaboration tools, planning tools, issue tracking tools, monitoring tools, configuration management tools, source control, dev environment things. Continuous integration tools, and then various set of deployment tools. So we’re going to walk through each one of these categories and some of the technologies you can use, today, to actually realize value in that area.

DevOps Technologies: Collaboration

Let’s first talk about collaboration. So, it’s not about having more meetings. It’s about actually collaborating in a way that results in action. How are you making sure that all of the teams are communicating regularly and with meaningful information? Whether it’s in Agile, using things like daily standups. Whether it’s having downtime exercises and actually simulating failures. How are you encouraging collaboration? Rapid communication, with the people needed to make a decision. That’s one of your key criteria. Am I making decisions? Am I learning things? Am I properly collaborating across teams? Tools cover a wide range of things here. It could be real-time chat, discussion rooms, knowledge repositories. Things that help me avoid the knowledge waste, but also things that encourage the collaboration. And it’s using the right type of tool for each situation. Why does this matter to DevOps? Because collaboration is critical, right? It eliminates the waiting waste. It keeps you from doing things that impact others without their knowledge. A really big part of DevOps success. So an example, of course you have things like Skype. And I’m trying to make a decision or I’m trying to simply walk through some information. Things like Skype and Lync, and so forth, great tools for figuring out a way to quickly collaborate, typically one-on-one, but even easier now to do kind of cross-team things. A great way to make those sort of quick decisions or quick question and answer. I can use tools like CampeFire or I can use something like Slack, which is a great way for keeping a pulse of what’s going on, quickly bringing a team together to make a decision. So let’s create a quick room, set this up, set up this channel, get in and discuss a problem. There’s an outage, let’s have people sitting in 10 different locations, quickly jump into a group, collaborate, figure out what’s going on and move on. As a team leader, even a team member, it’s a great way to use tools like this to keep a pulse on what’s going on. What are teams thinking about? What are some issues that might not even have bubbled up yet to management, but I can see people talking about something that will become an issue? So a great way, again, to remove some of that friction between the teams, be able to easily see what’s going on even though it hasn’t made it to a status report yet, I’m keeping a pulse. Using blogging tools is a way to actually store information. This is something, if you read the great book, The Year Without Pants, by Scott Berkun, you can see how WordPress themselves uses internal blogs to be able to share team progress and notes and so forth. So I have a persistent record. Things like Slack and things like Basecamp and Lync and Skype are great. Not always the same sort of persistent information. Of course, I can go back and search things, but ideally when you’re thinking about how do I store decisions? How do I make it easier for a new person to onboard and learn something? Something like a blog is a great way to give a new-hire a way to see something important. As are things like using GitHub Wiki. How do I have persistent information for teams, that might want to be, here’s our coding standards for APIs? Here’s how we do X, Y, and Z. Here’s our exception handling strategy. How do I make that stuff easy to find? Make sure I’m not just putting that in an email somewhere. It doesn’t really even belong in these kind of ephemeral chatroom sort of things. I want persistent ways, SharePoint wikis, blogs, things like that. I need persistent information. Think about those. GitHub has a great solution and there are others you’re going to find as well.

DevOps Technologies: Planning

As we look at planning tools, how do planning tools play into DevOps and what are some things I have here? Really the key is, how do I get the collective team participating in the planning? Transparency, are everyone’s priorities in harmony? If not, why? So that’s a big part of DevOps success, is again, that transparency, the communication, the teamwork. How am I using tools, typically online tools, not static documents, to help me do that? Static documents are out of date. The minute someone emails you a project plan it’s probably already out of date. So you can have formal project management tools, Kanban boards. You have a lot of ways you can share this information, hopefully online in real time. You’re doing this to distribute ownership of tasks, bite-size activities. Transparency between team members and simply visibility of what are these guys working on? I can see what’s going on in a crisis or in a bug triage process. Where is my thing in a priority list? What did they end up capturing in their postmortem session about action items? I can see all that sort of information in some of these shared planning tools. In my own example, at my company, we use feature review meetings once a week. Where, people from QA and Development and Design and Project Management, Product Management, and Operations sit together and review what our external customers are asking for. Shared vision, shared idea of what’s coming in and then we can group, have feedback on the priority of it. Does this matter a lot, a little? And so doing that is great. Instead of me in Product Management just saying here’s a feature, this is what I think it is, and moving on, the transparency is great. So that’s really why this matters to DevOps. That transparency reduces knowledge waste and confusion about the blockers upstream and downstream and what the priorities are. You’re seeing that in real time. Tools like Trello have been great in this space. A great tool for actually creating these sort of boards. We’ve done this as a company ourselves a lot, as we’ve stood up boards for different functions. You invite different teams to it. Real-time collaboration, who owns what? What’s the progress on it? All those sort of things. It’s visibility, it’s transparency. It works across teams. It works on mobile devices and so forth. A great way to actually do collaboration. Visual Studio Online actually has some good features for planning as well, where again, I’m using an online view to do things like manage my product backlog, assign things to individual Sprints. Different ways to see things. A really great way again to use a tool to do collective planning. Great visibility. Where’s my progress? Let me jump in and change something. Great tools. These are a lot of things, again, as you think about planning in a DevOps world it’s about breaking down these barriers.

DevOps Technologies: Issue Tracking

Next we think about issue tracking, really the rapid response. How am I doing things in a single way to collect, triage, and respond to issues? Shared systems. Not transportation waste, of passing information between multiple bug tracking systems. So if you’re in an organization where a tier 1 support person collects information and then hands it off to another team who enters it into their bug database, and then it maybe goes to a dev as part of a feature. And all of this happens, by copying data multiple times. Stop doing that. You end up with transportation waste, motion waste, knowledge waste. The information continues to degrade as it travels between these systems. So having a shared way to collect, view, and see what’s going on is great. You’re closing the loop by having everyone looking at the same issue or problems, not by passing it around. So this matters to DevOps because it’s better responsiveness, not having knowledge waste, make sure you’re giving a better end user value experience, by having ensure everyone can see what the defects and problems are. Some tools in this space are things like Zendesk. A great tool for collectively managing both knowledge and some sort of knowledge repositories. You’ve also got where you can manage tickets and see threads, and so forth. So a great, kind of industry standard, tool for that. You also have products like JIRA, which is great not just for doing tracking in this case and keeping issues and such front and center. It also has some tools for supporting continuous deployment and continuous integration things. So, JIRA’s a great tool as well. But, looking at these tools, where are my shared lists of issues? How can people join and view these issues and interact with them?

DevOps Technologies: Monitoring

Next up is monitoring. How do we do the smart correlation? DevOps success can potentially hinge on how you do system monitoring. If I do it poorly, developers are going to be extremely upset because they’re going to be the ones who get brought in or paged unnecessarily. Operation staff is going to be annoyed because maybe there are things that they’re spinning up alarms, thanks to developer code that actually don’t matter. So this is where Developer empathy for Operation staff is really important. Developers should be building the solution with their customer, Operations in mind. And Operations should also be making sure that if they’re involving Developers in alerts and pages, that they’re also thinking about what’s the smart way to do this. And so, when you’re doing monitoring in a system, you’re probably doing it inside out and outside in. How am I inside the system doing some monitoring? And how do I use some things outside, that might be looking for up time or overall health checks from a service perspective? Versus inside, maybe monitoring some of the individual components. It’s key about seeing the health of the whole system. Not just single servers in most systems nowadays. Especially as you think about that whole pipeline and think about services, not just servers. So why does this stuff matter to DevOps? It’s key to know what’s going on in the system. Transparency is such a big part of this so that all the parties can see what’s going on. A great monitoring solution means Operators aren’t getting randomly paged, Developers aren’t stuck in a silly troubleshooting phase where they’re trying to figure out what’s going on from some vague information. Cloud environments can expand, contract, without raising any false alarms. So as you have a well-thought-out monitoring strategy, that’s really going to help some of these other parts of the DevOps pipeline with deployments and maintenance and so forth. It’s up to Developers to do a great job upfront thinking about this when they’re designing systems. Thinking about how Operations will maintain and manage this moving forward. And again, for Operations to think of when is the right place to bring in other people to help troubleshoot a problem? Lots of technologies here. Things like Logstash are popular for managing events and logs, collecting, parsing, storing, exploring various logs. This is free and open source. You can access that. You’ve got things like Microsoft System Center. So when I want to be able to do some group-based, collective management, as well as monitoring of multiple systems in a Microsoft environment. It’s a great tool for doing that, so if you’ve already got that in place that can play part of your monitoring story. Check out things like Kibana. So when you’re trying to make sense of information. You’re trying to chart it, rank it, explore numbers. You know, you’ve got this raw data, and I’ve got to turn it into information, turn it into knowledge. How am I using things like Kibana to actually take raw data, raw logs and make that into something useful? I’ve got to do that by actually reading it, charting it, giving me different views on it. So, it works really well with Logstash, as well as Elasticsearch. If you’re using that in your environment as well, Kibana is a great tool. And there are other things in this space as well. Things like Graphite, StatsD, other tools that will help you listen for stats or render graphs, or do time series data. Really thinking about, what is the information that your system is emitting? Your service is emitting? What is some information that you might want from that so you can make smarter decisions? From a pure monitoring perspective, things like New Relic, where I want to be able to monitor, again, outside in and ping some of my services, see what’s going on. Look at the response time, health checks, latency. All those are really important things as I look at the holistic view of what my service is. And as Developers take more ownership for the application in every environment, whether it’s Dev, Test or Prod, Operations takes more responsibility for the infrastructure in all those places. Developers need to care about this. They may be the one getting alerted because latency is high and they need to figure out if it’s an application problem.

DevOps Technologies: Configuration Management

Configuration management is what people typically think of when you think of DevOps. This is really, oh how do you enforce a state to avoid configuration drift by using automation to achieve some consistency? So, forcing a particular state of a server, because we all experience that problem. As you have a server online longer and longer, the configuration can potentially change between servers as you go to quickly fix one thing and you forget to keep it consistent elsewhere. And so it’s using automation to enforce consistency. This is also where the compliance story can come into play. As you make sure that you’re maintaining consistency, and can prove it, by having a Configuration management tool. So how do I manage large sets of servers at scale with repeatable and automated configuration enforcement? Configuration management tools. You’re treating infrastructure as code that can be source controlled and deployed. A huge benefit in DevOps where I’m really treating environments, like a configuration file. Or I can easily spin them up reliably. Any changes are applied systematically versus manually. I’m not going in and updating individual servers with a new policy. I’m updating a policy centrally and either pushing or pulling that information out to each affected server. So why does that matter to DevOps? Pretty clear, this enforces consistency. So I’m not wasting time correcting manual defects, I’m getting better compliance with systematic state enforcement, versus a lot of manual work. So, one confusion with DevOps is that Developers are just jumping into individual production machines and tweaking things. If things are set up right from a technology perspective, no one’s ever touching anything themselves. They’re using configuration tools, they’re using deployment pipelines to push things out with a very clear audit trail. And so that’s really the big part of this, is it’s not people with access, it’s often systems with access that you’ve kind of broken down the barriers and silos and allowed controlled access to a bunch of things. So a lot of tools in this space, a very mature space. There are tools like Chef. You can create these cookbooks and recipes with Ruby, manage tens of thousands of Windows or Linux machines from a central server. Or you can do a Chef solo, without a centralized server, and have servers that kind of maintain their own configuration that they can just have stored locally and you can push locally. It does have Azure integration now. So you can even do cloud integration with a lot of these tools. There are things like Salt where you can set up this sort of agent list model, that uses SSH instead. So not this sort of idea of the same central server per se, but that has agents on individual boxes. You can still have a centralized server, a master server, or you can just run these minions all over the place on Windows and Linux machines as well. So I could do Master or Master Lists. The key with Salt is to kind of focus on the more high-speed communication between the nodes versus the polling sort of strategy, that things like Chef or Puppet use. Speaking of Puppet, so it’s a great tool for creating declarative infrastructure definitions. And I can manage this life cycle from provisioning to runtime and deployment and reporting. This is open source, again, like most of these. There are commercial offerings where I can have more baked-in and support experience, but I can do thousands of machine management. It’s something that works with Windows and Linux. And it comes with a lot of prebuilt modules that you can see, that plug in and let you support things like IIS on Windows or Apache on Linux. And it already has a lot of these modules for managing and maintaining commercial software and open source software. Ansible is a great tool. This also uses SSH instead of agents on boxes, meant for automating configuration management, provisioning, and more. It’s got this concept of playbooks for configuring and deploying and orchestrating deployments. Not a lot of Windows support yet. There is some there. You can’t run the Master piece on Windows, but again, a great popular tool. There are a lot of things that are coming up in this space. Chef, and Puppet, and CFEngine, and others have been around for a while. But you’re also seeing Ansible and Salt, and some other things coming through as well. So think about what you need. Really play with some of these. This is where the learning organization will come in handy. A newer entry into this space is also PowerShell, Desired State Configuration. So you can actually set things up as well now with PowerShell and enforce configuration across Windows and now pretty recently, Linux machines. So this lets you deploy and manage configuration information for these machines. Enable and disable roles and features. Manage files, start and stop services. Manage the registry. Deploy software. All those sorts of things, you can have a central configuration with a pull server or again push this thing out to individual machines. This is a great space. If you’re not looking at configuration management, you’re missing out on a lot of interesting tools. This will be something that’s great for internal Brown Bags as you learn more.

DevOps Technologies: Source Control

So a big piece of that is once you have configuration management though, thinking about source control. This is where you’re having controlled access. You’re closely guarding your software assets. It may be the most important part of DevOps frankly. It’s about realizing the power of the source control systems and storing configuration along with code. So instead of just thinking of source control for code, it’s about actually using this for all of your configurations of all of your environments. So this may be the key piece as I’m doing infrastructure config. I can have confidence that what I’ve tested is what got deployed. And I can track the history of changes. Why did this build fail after it got in production? Let me go ahead and compare the configurations in source control. Why is this working differently? I can have a source control tree, where I can easily see what changed between one and the other, versus people manually changing boxes. If you have a process that says, I have configurations that my configuration management solution uses, those are always stored in source control. I can really, really easily have an audit trail of what happened, who did it, when did it occur, and what’s changed. A great, great way to have more security and compliance in your environment. So there are lots of tools in this space. One of course to point out is something like GitHub. Where you can have public repos of course, like Netflix, in this example. Clearly your private ones, where I can store this information that I can pull in for configuration management tools, continuous deployment tools. Source control really ties a lot of this together, especially more modern distributed ones where again, I can have things that are offline and things can still work, much better than these sort of centralized engines where everything can grind to a halt if something goes wrong.

DevOps Technologies: Dev Environments

There is a lot of great work in the Dev environment space. This is about accelerating social development, enforcing consistency. How do I get Developers up quickly, using standardized images, and no more of the, it works on my machine answer? Of course this would matter to DevOps because it means all parties can quickly and reliably share key environments. How are Devs working in production-quality environments? Some of this is configuration management, right. We just talked about some of those tools. Imagine baking that into some of these images, that when a Developer stands up a box it immediately enforces a particular state of security roles and firewall port hardening. All the sort of config that looks exactly the same as production, configuration management tools can definitely help with some of that. In addition, there are new tools like web-based IDEs. Things like Codenvy, which lets you actually do completely browser-based development of a whole host of languages. Where I can build, and test, and deploy my code all from a browser. This sort of stuff is valuable as I’m doing social development, I’m doing collaborative development, between teams. I’m pair programing, maybe not even in the same room, because I can see real-time each other’s code changes. So that’s a pretty important thing as I’m looking at, again, how do I onboard quickly? Reduce friction? Streamline the pipeline? Browser-based IDEs can be pretty interesting in this space. One of the most exciting things in this space is Vagrant. You can go to vagrantup.com. Again, most everything I’ve shown you so far is free and freely available. But this is about portable work environments, based on standard technologies controlled by a single workflow. Regardless of what your hypervisor technology is. So, if I’m a Developer, I can have a consistent environment that can be source controlled regardless of my host OS. I can run this on MAC, Windows, Linux, whatever. And I can stand up with a single command, Vagrant up, a particular box, defined by a Vagrant file, which really defines what my box looks like. Here’s the base image, here’s the Chef script to run when it stands up so it gets configured. Which underlying hypervisor maybe it runs on, which I’m not exposed to as a Dev, I don’t have to care about it. I can configure networks, set up multi-machine configurations, very, very powerful stuff. If I’m in Ops, I can do quick dev and test locally and use the same workflow to test on the cloud. For a Product Manager or a Designer, I don’t have to worry about standing up every environment myself. I can just Vagrant up from a Vagrant file that’s in source control and immediately have a running environment of multiple servers that might be running my app, without wasting a day setting things up or configuring it. A really, really, really powerful way to run things on Hyper-V, VirtualBox, even in various cloud platforms to easily extend that. You can create your own boxes that have base OSs like obviously Ubuntu and Windows and other Linux flavors. You can have ones that have software already on them, to experiment with them. And now you can even use the Vagrant cloud to do things like sharing. Where I could open up my HTTP port to any one person in the world so they can actually access my box or let them SSH in, things like that. Pretty, pretty powerful stuff. Likewise with multi-server. So if I want to set up a database web environment, multi-tier environments, try out DR, I can easily Vagrant up the whole environment by pointing to one Vagrant file, that again, is sitting in source control and defines my server. I’m really treating infrastructure as code, as a config file. Powerful stuff, definitely try this out.

DevOps Technologies: Continuous Integration

Continuous integration is about incremental progress. How am I merging Developer copies with a mainline many times a day? I’m preventing last-minute integration problems. Anybody who has worked on big enterprise projects, knows that sometimes the most painful part of the project is that last phase integration testing, where we take all of these independently developed components, try to see if they actually work together, and then inevitably spend a lot of cycles fixing it all. So it’s a big problem with these non-Agile projects, where you’re not constantly integrating and that’s a brutal phase. Instead, in continuous integration, having these centralized build engines, coupling with automated test suites and even automated security penetration and other code inspection suites, that verify code quality before committing the build. So you really, I mean, the reason you’re doing continuous integration you’re constantly building, releasing, and testing it to catch problems before it gets released to production. Or you end up with a bottleneck in QA where everyone’s merging all of this stuff, and now it’s stopping up because it’s trying to collect everything together. You’re removing that bottleneck. You’re really simply trying to make deployment boring, as much as possible, because releasing changes, pushing changes, is done in an automated, consistent, reliable fashion. Why is this good? Again, I’m getting the early warning of bad changes. Constant availability of working code. Immediate feedback to Developers if something didn’t work. This should be obvious why this is good to DevOps, is code quality assurances, reduced defect waste. Increase automation and reduce the friction so I can constantly be getting code ready for production. A lot of things I can use in this space for .NET and Java shops, something like TeamCity is a great tool. You can get this from JetBrains. It’s good for .NET solutions, works with Visual Studio, has some code analysis baked in, supports NuGet. It lets you do testing before committing any changes. It shows you progress reports and a lot of different ways to be notified if something goes wrong. Things like Hudson, now Jenkins, been around for a while, really good tools for monitoring software builds, quickly integrating some changes. Even distributing builds across servers for complicated builds. Something to look at again, open source thing that you can pull down. Products like Travis CI, also very popular. A lot of integration capabilities. Spinning up databases, caches, and so forth as part of these builds. It integrates well with GitHub. You can actually do it on the .com-hosted builds and supports for a lot of different programing languages. So another good tool to inspect there.

DevOps Technologies: Deployment

So the final category is we think about deployment tools and this covers a lot of different things. But the goal, as I mentioned in continuous integration, deployment should be commonplace, it should be boring. Now continuous delivery, you’ll hear that term, means that software can be released to production at any time. Continuous deployment means that every change actually goes right to production. So, subtle difference there. Continuous delivery, I could be deploying constantly, continuous deployment I am. And the key is it runs through that whole deployment pipeline with some of these tools. Where it’s actually running and deploying it out, getting everything configured. The goal is to make deployments more reliable and less scary. It encourages a thoughtful collaboration between Devs and Ops. When I’m pushing things from a check in, all the way into production, everyone needs to focus on whether those things that should be happening along the way, because a person may never intercept any of it. From the minute I check in code, it could realistically be in production seconds or minutes later. You really do think about that whole pipeline. It’s important to work, obviously, between Dev and Ops, that’s really realizing a lot of this vision. So I’m going to be talking through here a bunch of tools that make deployment easier, not just continuous deployment tools, but this matters to DevOps because this is what enables that Agile delivery in a way that involves the whole org. I can have a bug fix, that’s been asked for by somebody in HR. I can have a defect that somebody else noticed internally. There could be a compliance or security problem, and I’m able to make those changes, get them pushed all the way to production, with a full audit trail, catching any errors that I might have made along the way. All in a reliable way that I can easily trace back. Super powerful stuff. This is really what brings DevOps to life. Implementing that consistent deployment pipeline, where I’m seeing that all come together, by then taking the result of all that stuff that exists in this reference before that, and then actually getting that to a place where it’s realizing value to the end user. Whether that’s an external customer, who pays for a service, or an internal user who now can do something more efficiently, this is where it really pulls it all together. So this space is full of lots of different types of concepts. There are things like CloudFormation, if you use Amazon’s web services, and the idea of orchestrating out deployments of environments. So it doesn’t handle code deployments, that’s more of their Elastic Beanstalk service, but it’s environment deployments. How do I automate and build out complete environments via templates? How do I make deployment of environments a boring thing, where I can click a button and create a multi-tier application? There are other tools in this space. This is just one example of, how do I use a cloud environment to quickly build out environments? There are cool tools like Packer. You can access this, it’s free and open source. Create machine images for platforms. So, build gold images, using provisioners that configure the running machine before turning them into templates. So I can take a running machine, put a provisioner on there that configures it, use a command line tool, maybe at the end of my continuous integration process that actually takes that configured machine with the latest code on it and creates a Vagrant image, a VMware template, an AWS machine image. Whatever it is and more, gives me all of those different templates for every environment and I can run it as a command line build. So I can end up with these gold images that always have the latest code. This gets me more towards that immutable server concept. Where I might never patch a server, I may never update a server, I just replace them. So, because it’s so easy to simply build gold images, as part of my actual delivery pipeline, I don’t have to deal with configuration management to some extent. I don’t have to deal with constantly patching machines, I just replace them. And using tools like Packer are ways to make image template creation much, much simpler than it’s been in the past. If you are paying any attention in a technology space the last even six months, let alone the last couple of months, you’ve heard about Docker, a Linux technology for creating isolated containers that are really portable between machines. This is really OS virtualization versus just kind of server virtualization. I’m carving up the OS into all these individual containers with isolated resources, CPU and memory, network, virtual interfaces, content isolation. I’m doing that all with an individual machine, getting even more density of use out of a VM. This can run on any hardware, VM bare metal. And so you’ll see this often baked into a lot of continuous deployment pipelines and continuous integration pipelines where I can actually integrate my code, push it out to a bunch of different containers, test it out. I can do perf testing. I can do a lot of different things, even in production now, using things like Docker. So it’s a great way to containerize things and I’m seeing teams do this more and more as part of their deployment pipelines, using containers for all sorts of reasons to test and scale things out. As we look at an actually deploying code, things like Octopus works great for ASP.NET and if you’re a Microsoft shop, taking those build artifacts from your continuous integration run and deploying it to all sorts of different environments. It integrates with TeamCity if you’re using that. You can deploy to Azure, you can to deploy to On-premises. You can see what you last deployed, the version of each component, the state of the last deployment. Great visibility into what is running in production. What was the last run when I pushed stuff, and how does it all look? I can manage application configurations that might be environmentally sensitive. This one has this connection string, this one doesn’t. IIS settings, things like that. Again, the Microsoft environment. It can also introduce approvals and manual intervention, because sometimes everything can’t be automated. Maybe there is a last gate in certain scenarios where someone approves something before it goes to prod. And I can also encourage self-service deployments even with role-based access. Tools like this really speed up deployment delivery by giving me that audit trail and automation. And finally, if something like Go, which recently got open sourced by the folks at ThoughtWorks. So this is for continuous delivery as well. One click deploys. Configured with XML configurations, I can see what’s building, ensure that tests have run against the code before I push it. And I can even see what’s in production. So, some great tools for actually doing continuous delivery. Again, many of them open source and freely available.

Summary

This was a fun course for me. This is something my company does, in terms of a DevOps focus. We’re constantly changing and improving, but it’s good to see this in action. Hopefully this is something that’s interesting to you. This particular module, we did a walkthrough of some of the key technology categories and this is far from a complete list of all the different tech that makes up a DevOps pipeline. Each of those categories have another dozen things you can do to do collaboration, or do to issue tracking, or to do source control management. But really, as you think about DevOps, how are you making some of those incremental changes? How can you start automating things you do manually, more than once? Focus on sharing knowledge and collaboration, breaking down some of those silos, working across teams, towards a single objective. Realize who your customer is. Think about the next stage in the pipeline. Ensure that you don’t become the bottleneck. I would love your feedback on the course. You can tweet me @rseroter, leave a comment here. Chase me down on my blog, seroter.wordpress.com. I hope this at least made you think that there was some interesting concepts in here for you and that you challenge anything that you didn’t agree with. I hope we continue to move this conversation forward.

• Author
• Recent Posts

Follow me

Rajesh Kumar

Mentor for DevOps - DevSecOps - SRE - Cloud - Container & Micorservices at Software AG

Join my following certification courses...
- DevOps Certified Professionals (DCP)
- Site Reliability Engineering Certified Professionals (SRECP)
- Master in DevOps Engineering (MDE)
- DevSecOps Certified Professionals (DSOCP)
URL - https://www.devopsschool.com/certification/

My Linkedin - https://www.linkedin.com/in/rajeshkumarin

Follow me

Latest posts by Rajesh Kumar (see all)

• Apache Lucene Query Example - April 8, 2024
• Google Cloud: Step by Step Tutorials for setting up Multi-cluster Ingress (MCI) - April 7, 2024
• What is Multi-cluster Ingress (MCI) - April 7, 2024