DevOps Deep Dive

In this episode of The IJYI Way we chat to the IJYI team about the what, why and how of DevOps. The line-up this week includes:
Chris Pont, CEO
John Nicholson, CTO
Alan Jackson, COO
Tim Naish, Consultant.

With years of experience in software development and delivery the team give us some fascinating insights on what has changed in the world of IT. They discuss how DevOps can enabe IT teams to break down barriers and blockages to improve outcomes.

Transcript

Andrew:   0:00
Welcome back to The IJYI Way. Coming to you live and direct from IJY's boardroom here in sunny Ipswich and it is sunny. It's been, pretty rotten weather all week where you are but here in Ipswich, it's fantastic. That's why you should move your business here. We might use that we might not, I'll try it again.  Hello and welcome back to The IJYI Way Podcast the first one of January. So it's a big happy New Year from us at IJYI we are starting off the year as we mean to go on talking about DevOps. Yes, it's probably the biggest thing that's happening in the world of software that you've heard off. Or perhaps you haven't heard of it. It's certainly transforming way people make software the way the software integrates with business and the way that coding will continue to drive digital transformation across all different areas of the business world. And joining us today to talk about this we have consultant Tim Naish, and with him, we've got CEO and co founder Chris Pont, John Nicholson, CTO and Chris's co-founder good  and also resident DevOps guru along with Tim. Can I say, Guru?  

Alan:   1:25
No, no, it's not true that's why! *laughing* Look at the other three in the room for that one!

Andrew:   1:31
That, of course, is the voice of reason Alan Jackson, Chief Operating Officer and I'm Andrew Walker. I'm a freelance writer, and normally at this point I would make some sort of lame joke about coming here for the coffee. But the truth of the matter is that I'm passionate about new technologies and new ways of doing things on Also, biscuits let's not forget the biscuits...... and the tumbleweeds. Clearly!  *laughing"

Andrew:   2:03
You have a January challenge here, don't you? Which is you've got to do as a company 2020 minutes of exercise a week. 

John:   2:12
You say as a company, but it's really Chris has to do 2020.

Chris:   2:18
I think my Strava score is looking right right now, but it can still do with some improvement! *laughing*

Andrew:   2:23
Okay, there we go. So Chris is on Strava, if you mail IJYI or send us a DM on the Twitter ID, that's @IJYILtd we'll give you Chris's Strava ID and you can race someone....let's see if he's as agile as he makes out! *laughing* So we are talking about DevOps and to start can I get a definition of what Dev Ops, actually is, because you hear a lot of acronyms, don't you? I've heard develops of heard Des Ops that's a new one. We could maybe talk about that a bit later on.

Chris:   3:00
DevSecOps

Andrew:   3:00
Really, is that a thing?

John:   3:04
Yeah. And SecOps

Andrew:   3:06
Okay, OpsOps?

:   3:07
  

Chris:   3:10
Marketing ops

Chris:   3:11
That would be MarkOps. Okay. Is there one we could do with a si so it could be Cyclops? Is that you know, I suppose that there will be a psychology one.

Andrew:   3:30
Okay, fine. So rowing it back now.....please don't write in!Rowing it back. John, you were the first person I heard talking about DevOps. What's your definition of DevOps?

John:   3:44
I'm actually not gonna give my own personal one. I really like a definition that's been used by one of the Chief Architects and head of DevOps over at Microsoft. So Donovan Brown coined it in this way "DevOps is a union of people process and products to enable the continuous delivery of value to our end users" It's really useful. It is very specific as well you'll notice. It talks about people, process and products the union of those three things, but it talks about value, not software, and we're not talking about systems or software in this definition. We're talking about things that give direct value to the end user, the internal business user. And essentially, we're talking about something that our lines to the businesses goals, not necessarily IT's goals.

Andrew:   4:31
Okay, so is that. Does that mean DevOps sits in the realm of process engineering, not software engineering?

John:   4:37
In many ways, yes, it's very much around a process that you can take things through. The biggest part of DevOps often is about aligning departments to the business's fundamental goals. We've had years of management consultants doing this with every other department and IT has been left in a backwater where it's very much cost, centric and cost focus. Instead of being focused on what generates the sale, the transformational companies out there learned this lesson early on and have driven their sales through aligning their departments with this.

Andrew:   5:19
Tim what's DevOps for you?

Tim:   5:23
I think for me it's a lot about culture bringing together two  different, well at least two different teams and having a kind of shared responsibility of delivering that business value. Often particularly in large companies there could be a lot of siloing and things are affectively kind of thrown over the wall from from developments into operations, and that really doesn't help deliver that value. Often, teams don't talk to each other. There's a lot of finger pointing when things go wrong rather than taking ownership as a whole team.

Andrew:   6:00
Okay, so there's taking ownership. There's this union of people, process and product. Chris, what's DevOps for you?

Chris:   6:09
I think it is that focused on delivering value. It's making sure that everyone's working towards a common goal, and I think, you know, DevOps is often misinterpreted or misused within the industry. So people try to silo it into a specific job roll into a specific person or they consider it just automation of infrastructure. Where is you know, it is about taking on the whole team and making sure they are all driven towards a common goal of delivering value as quickly, as often as possible to the customer.

Andrew:   6:46
So we're getting down into the real shape of it here. Alan, what's your take on DevOps as the Ops part?

Alan:   6:54
So to actually deliver software using it. I think Tim sort of really encapsulated what what I would say about it probably said it better as usual. So it's about that sort of collective responsibility. So when you're resourcing a team that's using DevOps processes, you're looking for people that have the they're sort of will to take on what we're traditionally be parts of multiple, different roles. So people that are prepared to not only write the software and write great quality software but also take on the tooling and processes that allow them to deliver that software regularly and quickly and reliably and to a good standard to the user so that they so they deliver value early. So it's about taking ownership, the staff taking ownership of the end to end not just the build and lob it over the wall to someone else who's responsible for the next bit, but ownership of it end to end right through to delivery to the client, and they've gotta have that will and they're gotta want to be involved across the board. So, you know, specific type of person we have to hire for that, I think, and then going on to that at IJYI particularly the guys then have to support their own work as well, so it's not just the development, not just the infrastructure and the deployment out prod, then walk away. They then have to live it afterwards. So they're on call 24/7 some of the guys are supporting the work, so there's quite a large incentive there for that to work properly unless they like their on call payments and don't care about sleep! But they have a big incentive to make sure that the quality of the product is high, and it's doing what the customer wants.

Andrew:   8:29
Okay, so we're kind of getting a framework around what DevOps is now. But doesn't that kind of ask another question, which is what was wrong with the old way of doing things? Because, you know, it has worked pretty well up to now, hasn't it?

John:   8:48
So the old way.....we talked about ownership before. Within IT organisations specifically, there were normally two distinct teams working. We had an operations team. They were charged with keeping the lights on. No matter what the cost, everything had to be running no matter what. Now, the traditional way of managing that was to not allow change, control the change as much as you can and restrict it as much as possible. And then you had the development or the new product teams, depending on how they organised. And this was all about what's new and shiny. What's the business asking for? How can we get this going? What? What can we get out there and being blunt in many organisations there was direct conflict between these teams and that's where this over the wall mentality came from because as soon as the delivery team was finished, they would throw it at the operations team who would often know nothing about what was coming, and they'd be expected to keep the lights on on this.

Andrew:   9:46
Okay, so, to put a sort of example around that. I've been involved in companies that have deployed entire new end to end purchasing project management systems and the software is very impressive, but the number of tickets,  support tickets it raises as you know, the people who are working remotely suddenly....they wake up one day, they have a new system, and suddenly they're saying, Hold on a second. What's happened to my old purchase number generator? And you know, why can't I sign off this invoice the way I used to with the check box or whatever it was? That's the tension between the new system that's coming out of the dev side on the supporting users and making sure everything keeps running from the ops, right?

Chris:   10:34
Yeah, I think you know the traditional way of releasing was to have one Big Bang release every six months. Nine months, 12 months. Whatever. I think nowadays, people expect features to be rolled out incrementally, and it means that the extent of that change is much smaller, they're just gradually getting new features dripped to them every every couple of weeks, if possible, but that does mean is that there needs to be that relationship between the development team and the operations team so they can handle those releases as often as possible. It does mean that the change is less risky. There is not one huge change that just happens in one go,

Alan:   11:13
I think with your example. What you're describing is a relationship between IT and the user base. I think what John was describing is a relation with the night itself, between the guys who produced the software and then the guys who have to live with it and maintain the servers and keep the lights on behalf of the users. So the conflict, the conflict is in within IT as a traditional department. And that's what you can try to break down and the inevitable thing. I mean, if you imagine being the Ops guy and then the developers come along with something new, shiny, chuck it over the wall, say, look after that, mate, and then you have to deal with it. What would you do as a team is an ops team. You would put paperwork in place. You would put run books and long meetings about how you're gonna absorb this thing. You would put long lead times on change in order to assure that your team's got the right resourcing and processes and capabilities to take on this new  asset. And all that does is put barriers between the business realising the value, realising their return on investment and realising the value that they're trying to deliver through the software. So you just putting IT itself through the traditional process is putting bottlenecks and blockers in the way off the business and that's what you want to try and avoid. As John said that businesses  over many years have stopped doing this in every other place in their business and IT kind of got left behind a bit, you know, it was seen as a cost centre. So DevOps as a process is supposed to break down those barriers and building on what Chris said one of the ways it does that is by incrementally changing things in small increments. So not big change, lots and lots of lots of little changes which are easy to understand, much less complex and they have much less impact on the end on the end piece of software if there is a problem so you know, you you kind of just de-risk the pipeline by actually changing more often which is counterintuitive because traditionally they were de-risking the pipeline by changing less, often but that doesn't help anyone.

Alan:   13:15
Yeah, and essentially, the more you practise change the better you get at it. So if you could do lots of small changes rather than fewer big changes, then yeah, you can de-risk your deliveries.

John:   13:26
Yeah, in line with that. I mean, it's a fairly common rule. All systems suffer entropy. The common way of maintaining stability was to not change. But that's a fallacy. It's based on a false assumption. We have to change to maintain position by making those changes smaller by de-risking them. You make change part of your business as usual, and it makes it more comfortable for the business, not just for IT. If the business is accepting of change and understanding how it's going to occur, they embrace it more. You get far less opposition and bluntly, they'll expect more as well.  

Andrew:   14:12
I've heard this amazing statistic that comes out of big Silicon Valley players.  Amazon we've got....they release new code every 11.7 seconds or something like that, and we've got Netflix are releasing something like, is it 1000 times a week or something? And we seem to hear it a lot. Google as well have this DevOps approach. Tim, I want to ask you a consultant how do you get the idea across to people who haven't moved over into the DevOps world yet that you could be releasing new code every minute, which would be quite slow compared with Amazon. We'll be releasing new code every minute and that isn't going to, you know, send everyone into a sort of a panic because it sounds like quite a major thing to do.

Tim:   15:10
With the examples you've described are kind of very large scale organisations. And so they've got very large and complex systems so by making small changes to different parts of the system that they're able to change and adapt very quickly. Obviously not not all companies are going to do new deployments every every 10 seconds but by building your system so that they are scaleable does allow you to grow. If we think of the traditional sense of how software would be developed you might find a bug in your system. You then deploy a fix for that and instead of immediately deploying that into your your production environment. You would gather up all those bugs into a software release, and there will be a big kind of release cycle that would happen in a big bang and you'd have a lot of change and so the potential for things to go wrong is a lot higher. In a DevOps environment you would do that small change, you would push that small change through your different environments, say your QA, your user acceptance testing environments and so on. Then, once it had passed user acceptance testing, it would go straight into that production environment and would be live for your customers. So by splitting things up into much smaller chunks, you're isolating that change, which, if for any reason, it might have gone wrong once it got through your environments, you could quickly reverse it because you know just what you've changed.

Chris:   16:44
Yeah, I mean, essentially it's about addressing risk, so it's making sure that you can you can automate. You can automate those deployments, but also automate testing on the governance side of things to satisfy the business that the code that you're releasing is gonna be acceptable, it's gonna work the way it should, it's not going to cause problems with finance or payment or or any regulatory issues it's making sure that you can work with the business to address those risks using some kind of automation so you can release speed.

Tim:   17:15
It's also delivering value to the customer a lot sooner because rather than gathering everything up and then releasing in a big chunk as soon as you've done that release, you've made that change whether it's a bug fix or a new feature you're adding that's moved down the pipeline into production much sooner rather than being held off.

John:   17:37
Yeah, an example of the way this change could be used if we look at standard practises for testing designs like A/B testing or canary based deployment. So A/B testing, we're going to release two different versions of a UI to prove which one gets the better traction. The differences in those UI may be very small. We talk about the scope of change. It could be the size of the by now button if we take an Amazon example or the position of it, but they'll measure the impact of that on their sales to decide which route they go. When we talk about Canary deployments, they'll measure it on a 5% of their customer base will then increase it so 10% are seeing it and 20% so by reducing scale of the releases and the scale of the change it enables new strategies for the business to look at how they handle their marketing, their sales approach and what they're doing. It doesn't just have to be large scale, logic-based changes in the system,

Chris:   18:33
And it does mean that when you roll out code, you don't have to roll it out to everyone. So when you're rolling out code to a customer base the size of Amazon, you might only rollout as John said, to 5% of the customers and just check it. Make sure it works, monitor it, make sure it's is behaving itself and if it is acceptable, then roll it out to everyone.

Andrew:   19:00
It's the kind of thing, we see this a lot I know with Cryptocurrency exchanges there's a lot of talk about the fact that if you follow them in the news, there will be some complaint that "oh look this new coin has emerged but they're only trading it for customers in the US at the moment" and you think well, hang on a second it's international, it's not that there's a regulatory reason why you can't buy that particular Cryptocurrency here in the UK, but you know, exchanges will roll it out in very specific areas first. Presumably, that's a DevOps approach.

Alan:   19:32
I think the DevOps approach enables them to test things like that. Like traditionally, you wouldn't be able to......if you use the traditional method the test cycle would be months, if not years long for you to find out which of four options is working best for you. With DevOps you could test something for a day, an hour, you know, potentially in the Amazons example in a matter of minutes and then turn it off again. And you know you think about Amazon. If they're, they're throughput is millions and millions of hits. You know, a minute whenever they go through, they could do a lot of testing in a minute, you know, so being able to do this allows their business, too, to just be so dynamic and so responsive and boil, their decision making not only the risk but boil their decision making risk down to minute components. So they're not making, they're not being forced to make wholesale decisions on the look of a page because they won't be able to change it again for six months. They can literally, you know, increase the size of the button by 10 pixels, corner to corner, and see if that makes a difference. Because they've got that ability to reverse it so quickly and try something new.

Alan:   20:36
Yeah, that does really highlight the monitoring side of DevOps because, you know, that's really important. You want to make sure that you're able to see the impact of your changes and that means that potentially you have a lot more monitoring a lot more logging in place to enable teams to see the impact of those changes. Is it affecting number of customers? Is it affecting our conversion rates, or is it just causing areas in the code? So, you know, do we need to roll that back or roll out a new fix to sort that issue out.

John:   21:05
Yeah, in line with that is very much worth highlighting That DevOps is an empirical process. This isn't a touchy feely guru led process. It's about measuring and seeing the impact and understanding what's going on now those measurements happen in multiple places. Chris was giving some really good examples around business level measurements that can be done, but you also take the measurements all the way back to when the first line of code is being written and measure the process from here through deployment measure each work station Because one of the big changes that it brings is we treat the journey from code being written through to being deployed into production as a production line like you would in a manufacturing business. So you are able to measure and optimise each stage, understand what causes complications, what causes errors, what causes delays and then remove them over time.

Tim:   22:02
Yeah and continuous improvement is a big part of this, so we're always looking at ways in which we can improve things all the way through the process. But we talked a lot about deployment, but it's very much part of the development process as well. Looking at work how it travels through the pipeline, through developments, QA et cetera. We're always looking at what's happening and using measurements. Not just kind of this touchy feely or this doesn't feel right. It's okay. What evidence have we got for this? How can we measure that?

Andrew:   22:37
We're gonna have to break it there because this is almost too awesome for one podcast and so we're going to tie it up, but come back in the next podcast to talk about automation. And also, how do you transform your organisation to leverage the advantages of a way of DevOps working and that kind of process design. So we're going to just wrap it up and say, a very big thank you to Alan Jackson COO. This is, by the way, in no particular order. That sound sound bad. Alan, I'm sorry about that and  to Tim Nash. Consultant. Tim's gonna be joining us next week for the next exciting instalment of automation and cultural change for DevOps. It's funny, it sounded more exciting in my head when I said it than it did then!! Just please come back for the next podcast. Because on that podcast we will have a CEO and co founder, Chris Pont. CTO and Co-FounderJohn Nichols. Thank you for listening to this episode off The IJYI Way. Join us online at ijyi.com. Find us on Twitter @IJYILtd Now, at this point, we're gonna wave at the camera, which makes no sense. Hey, makes no sense at all on the podcast. We did just wave. You want to see the proof that this actually happened on that? We're not all just AIs come and find us on Twitter on Say hi and we will see you next time for part two of DevOps  on The IJYI Way. Thank you were all waving now too!!!