As part of Data Week for VB Transform 2022, Patrick Barch, senior director of product management at Capital One Software, took to the stage to explain why operationalizing data mesh is critical for operating in the cloud. Then, on the second day of The Data Week, he sat down for a chat with Matt Marshall, CEO of VentureBeat, to dive into the governance piece of cloud strategy, and why a holistic approach is key for managing the influx of data in a new environment.
About six years ago, Capital One went all in on the public cloud. The company shut down its owned and operated data centers, and dove into modernizing the data ecosystem for machine learning.
“How do you manage something like that?” Barch asked rhetorically. “And by the way, you have to get this right, because — pick your phrase. Data is the new oil. Data is the new gold. At Capital One, we say data is the air we breathe. Companies recognize that the key to success in today’s tech-driven landscape is making use of their data. So, no pressure.”
Moving to the cloud means more data, from more sources, stored in more places — and a whole company of users demanding self-service access to all of that data in the tool, format and consumption pattern of their choice. That’s all happening against a backdrop of patchwork privacy legislation that’s popping up all over the world.
“There are a lot of challenges when you move to the cloud,” Barch told Marshall. “There are challenges with publishing, getting data into the cloud in a well-managed way. There are challenges with consumption. How do you help your teams find all of this data that’s exploding in quantity, that’s in all these different platforms like AWS and Google and Snowflake and others? How do you govern all this data, especially against a patchwork of emerging privacy legislation that’s popping up all over the world? Finally, this is a new paradigm for infrastructure management. You’re not responsible for servers anymore. You pay as you go. How do you put the right controls in place around all of that?”
Early on in the journey, the company invested in product management and user-centered design in the data ecosystem to address the specific challenges of all their customers and users — how they use data and where they struggle with it. That includes everyone from the people publishing high-quality data to a shared environment to the analysts and scientists leveraging that high-quality data for critical business decisions. There are the data governance and risk teams, worried about defining policies and enforcing them across the enterprise, and the teams responsible for managing the underlying infrastructure that powers all of those use cases.
Organizations often end up with an array of point solutions to solve for some of these user needs — and a single person might have to hop between six or seven different tools and processes to complete a simple task like sharing a new data set, or finding data. But that simply doesn’t work, Barch says. Scaling this ecosystem gets extremely complicated for both the engineering teams that have to build and maintain these integrations, and the users who have to navigate across this map.
“For me, the heart of this thing is treating data like a product,” Barch said. “Once your company makes that mindset shift — and it truly is a mindset shift — the rest of these principles fall into place. You need to understand how to organize all of those products, and you need to figure out the right capabilities to enable self-service across a variety of factors.”
That’s where data mesh comes in: an operating model that can help scale a well-managed cloud data ecosystem. Capital One approached their own ecosystem through two prongs. A centralized policy tooled into a common platform that enabled federated data management responsibility. The aim was to put more control into the hands of the teams that were closest to the data themselves, because data mesh doesn’t work unless it’s operationalized through self-service. And the overarching goal is to get your data practices operating at the speed of business.
“When you combine common tooling and centralized policy with federated ownership, you make it easy for your practitioners to do their job,” he said. “You transform data from something that’s a bottleneck into something that can turbocharge and power your business.”
Captial One engineers built these tools and infrastructure internally, but Barch recognizes that not every company has the luxury to build things themselves. Fortunately, a vast array of solutions exist today which didn’t exist when when the company was starting its journey.
“You just need to make sure that you’re creating a user experience that works for your user base,” he explained. “The days of a single central data team and data being the IT team’s job — those days are over. Think through the UX. How do you enable your teams to get their jobs done?”
To help other companies navigate the cloud journey, the company created a new line of business: Capital One Software, which is bringing to market some of the internal products and platforms the company developed to help navigate its own cloud journey. Capital One Slingshot, the first product, is designed for companies trying to adopt Snowflake in a well-managed way.
The product tackles one of the biggest challenges with any cloud provider: the risk of unexpected costs due to the pay-as-you-go, usage-based consumption model. Slingshot offers a way to create the rules of the road for infrastructure provisioning and management to most efficiently use the cloud resources, particularly Snowflake, which cuts costs, and simplifies the experience of your critical data users — and levels up your data optimization.
“The data transformation was all about better serving our customers,” Barch told Marshall. “Being able to create more real-time experiences around fraud detection. Being able to level up the skills of Eno, our intelligent financial assistant, has been a huge win. Reducing the amount of time it takes our analysts and scientists to find new data for new projects has been a massive time saver and a massive win. We’ve been able to onboard thousands and thousands of new real-time data streams to the platform, all via self-service thanks to these tools.”
For Barch’s deep dive into Capital One’s epic data transformation journey and to catch up on all Transform sessions, register for a free virtual pass right here.
