Over the past decade Bank of America has grown by leaps and bounds internally and through an array of mergers and acquisitions. From a technical standpoint, that growth has created a complex and disparate set of data centers, computing architectures and vendor relationships.
For CTO David Reilly, there was an obvious goal: Standardize on more efficient infrastructure. For a company that spends $3 billion investing in new technology development each year – nearly double the amount it did five years earlier – any reduced expenditures translate directly to improved bottom line profitability for the bank. Transitioning to a shared virtualized computing platform not only drove savings in the IT organization, but net profit for the bank. But soon Reilly realized that standardizing and virtualizing was not enough. He wanted to start all over again.
Do like Facebook does
Bank of America had to get to a point where it is as agile, flexible and efficient as young, web-scale companies, Reilly believed. “It’s not enough to make what we’ve got better,” he concluded. “We actually have to get to a point where we could almost completely discard (our infrastructure) and move to a brand new, greenfield environment, unencumbered by any of the history or legacy of our existing footprint.”
Reilly isn’t alone in thinking that legacy infrastructure just isn’t cutting it anymore. In 2011 social media juggernaut Facebook started a community foundation named the Open Compute Project (OCP), which aims to create a set of standards for commodity infrastructure components that organizations can assemble together themselves. That’s opposed to buying racks and stacks of infrastructure from the traditional IT vendors.
Reilly liked the idea. But just ripping and replacing was not an option. Bank of America has 100,000 technology and operations workers and contractors serving 19 million mobile banking customers – growing at 5,500 per day. Reilly’s technology infrastructure team manages 169 petabytes of data. The legacy system couldn’t just be shut down.
So about 18 months ago Reilly created a team that began rethinking everything – from how a machine was built to how it was managed. “Software-defined technology was the way to go,” he says, harkening to the philosophy of the OCP – whose board includes executives from Facebook, Intel, Goldman Sachs and the co-founder of Sun.
The new cannot look like the old
Bank of America took a two-pronged approach to starting what Reilly calls the bank’s greenfield project. One path used a vendor’s proprietary infrastructure stack (Reilly would not disclose which provider). The other followed standards being developed by OCP and other open source projects.
Both embrace software-defined infrastructure and follow core guiding principles: Compute, storage (and eventually network) resources are provisioned via APIs; applications use containers; and data centers are built in modular components.
“We’ve held a very hard line to say that the new cannot look like the old,” he says, even delaying the project at times to comply. The goal is to one day have a system built entirely on commodity servers that are uber-flexible: They could act as compute servers one day, a network switch another and be part of a pool of storage resources if needed. Containerized, micro services-designed applications automatically provision infrastructure resources they need without requiring any human hands in the process.
The proprietary stack is running about six-months ahead of the open source one because technology innovation by private vendors has been faster than the methodical yet essential work of the open source community.
About 95% of the applications that have migrated to this new greenfield environment have thus far been written to the proprietary stack. But Reilly hopes in the coming years as the open source innovation picks up, that will become the bank’s dominate platform. Open source software-defined networking is one of the laggards, Reilly noted.
All eight lines of business within Bank of America are already using the new platform including wholesale banking, development and quality assurance. There are plans to migrate additional grid-based workloads, while pockets of the wealth management division have use cases for it.
“We are now convinced that this software-defined model, where an application presents a manifest for the assets and facilities it needs, it acquires those assets with a call, uses them, then lets them go when they’re done so they can be used by someone else – that’s going to work for something like 80% of our technology workloads,” Reilly says. The other 20% of workloads will run on more statically provisioned machines and a mainframe that the bank still uses.
Matt Eastwood, senior vice president at IDC, says financial services firms are the next logical candidates to embrace this type of technology after the web-scale businesses. Businesses that do drug discovery, oil and gas exploration, aerospace and defense could all have workloads that favor a dense, scale-out, cluster-model compute environment.
“Any cost savings on the CAPEX and open side of the equation can be re-invested back into the computing environment in an attempt to get a competitive strategic advantage in the market,” Eastwood says.
Already Bank of America’s greenfield system has created enormous efficiencies in how application developers do their jobs. Reilly positions developers as pseudo micro-CIOs who serve their individual business units, and are charged back based on infrastructure usage. “A developer will move to this environment initially to get the price benefit,” says Reilly. “But the developer stays because of the flexibility and agility.”
Change isn’t easy
Oddly enough, Reilly says the biggest challenge in executing this greenfield program has not been a technical one. “This is not a criticism of the way we’ve been doing things,” he says. “It’s an embrace of the disruption that’s occurring in the market.” Still, unsurprisingly, it caused some workers to question, “What does this mean for me in the future?”
Workers aren’t judged based on the number of boxes they manage, or how virtualized an environment is. “’How am I delivering against the goals of the firm, how am I managing risk, and how am I contributing to overall profitability?” are questions he encourages workers to consider now.
Pulling out the team that worked on the greenfield project as a separate entity was one key to success. Gartner has encouraged this bi-modal IT approach of dedicating a team to work on a new project while still ‘keeping the lights on’ for the rest of the organization. Reilly said it took about a year to get everyone on the same page.
The project would not have been possible without Reilly’s bosses – Chief Operations and Technology Officer Catherine Bessant and CEO Brian Moynihan - embracing a long-term, five-year strategic planning initiative. At a recent all-company meeting Moynihan cited the technology infrastructure group as an instructive example of simplification driving growth within the enterprise. “You can imagine what that did for our team,” Reilly said.
Having engaged and committed workers and higher-up support is key, but staying abreast of new technology and innovation in the market is another. To help find those young startups that could be a benefit to the bank, for the last seven years Bank of America and Merrill Lynch have run a Silicon Valley Technology Innovation Summit. Hundreds of startups apply each year to attend. BoA invites a couple of dozen. On average, the bank ends up engaging with 17% of the startups who pitch at the conference.
Hindsight is 20-20
If there’s one thing Reilly wishes he had known before he started this process – one lesson he could impart on enterprises who want to go on this journey themselves -it’s to not underestimate the cost of legacy systems. It was essential for BoA to scale down the costs of the existing, legacy system (called the brownfield) as that new, greenfield environment was being spun up. If you don’t cut brownfield costs as you’re investing in the greenfield, then you’ll just be adding cost.
Doing so is not easy. Physical assets need to be depreciated off the books, along with associated maintenance contracts. Software agreements need to be written in a way where you’re not carrying more licenses than needed. Doing all this allows an organization to “monetize the shrinkage,” Reilly says. Doing so will free up money to invest in new, greenfield projects. “In large enterprises, we haven’t seen a problem we couldn’t engineer our way out of,” he says. “We are great at adding stuff, but we’re not so great at taking stuff away.”