Five years ago, a small team of developers at American financial services firm Capital One began experimenting with becoming more agile. Now, in an enormous leap, the bank has rolled out devops practices to all of its branches worldwide.
Speaking during an interview at the Devops Enterprise Summit in London, Tapabrata 'Topo' Pal, senior director of agile at the firm, explains just how significant this has been for the organisation.
"The whole purpose was to remove manual steps," says Pal. "So it started with a small team with small scope - all I want is to automate my build, and a few test automations in the pipeline.
"But success spread on our good news and reached the leadership, and at that point in Capital One there was a lot of transformation ready to take off."
That transformation included changing the organisation from one that outsourced its IT to working internally. There was a lot of noise about agile and working in a more lean fashion.
"It was good timing to put all these things together - the leadership had already bought into these practices from a conceptual perspective, but when they saw these things happen they bought in and the push came from the top: thou shalt go agile and devops."
According to Pal, it took about three and a half years for devops practices to spread out from that one team to the whole enterprise. Now, aside from a few areas, most of Capital One is in-sourced and it's pushing for an all-cloud environment by the end of 2018.
"About 40 percent of our production workload is on AWS now," Pal says. "At 40 percent we are larger than Netflix on the AWS footprint, it's huge. We are not running a hybrid model - our focus is everything on cloud. There are data centre-based applications being transformed, re-written, re-engineered, thrown out, to get to cloud."
Now Capital One is mostly an open-source tooling shop and automates what it can. The company found that using open source tools allowed the teams to contribute fixes quicker than closed-off solutions, and this offers clear benefits in shutting down bugs while also going to production as fast as possible.
It is, of course, phenomenally useful for enterprise-wide transformation if it's got that leadership buy-in. And the principles of devops really clicked with that leadership: speed, repeatability, the fact that nobody had to wait for source code to build and deploy, and the added trust of using tooling rather than humans.
"There were pockets of resistance but since we got leadership buy-in those resistances didn't last long," says Pal.
To illustrate the time benefits of moving to the cloud, Pal mentions how long it used to take to get a server up and running.
"We had a manual process for building a server in production and I think we listed 62 steps, and it needed people from all different organisations," Pal explains. "They all had their own check-list, so we found that for a developer, to get a server I needed to complete these 62 steps and none of this is owned by me.
"It took anywhere between two to six months to get a server. Now, developers are just clicking a button or running a command line tool, the ROI is right there - and if I don't need that server, I just shut it down, as opposed to if I had a server before I owned it for a lifetime. To maintain that, manage that, or if the server is not up to date with patches I get nasty emails. I don't get that anymore, if I don't need a server I kill it."
Pal adds that there were challenges in achieving the cultural mindset across the organisation - of course, this is to be expected with any global company that has thousands on the payroll. There were a lot of silos that needed to be broken down and handing over projects between teams, but in addition to this, a lot of retraining was needed, too.
"Not only organisational change but also to retrain people - right now we do not have a Unix admin because there's no Unix admin needed in the cloud," says Pal. "What do you do with those people? There's no admin as such, so a lot of retraining, education, certification, pushing people to get up to date. It's a lot of these kinds of silos and how to get people embedded into the team that requires a lot of work from a people-management perspective."