Amplify Partners’ Sarah Catanzaro on the evolution of MLOps

Note: This interview was edited and condensed for clarity.

As part of our media partnership with Tecton’s apply(conf), RTInsights recently had the opportunity to speak with Sarah Catanzaro, General Partner at the venture firm Amplify Partners. The firm has invested in data startups OctoML, Einblick, Hex, among others. Prior to venture capital, she was the Head of Data at Mattermark. She started her career in counterterrorism.

RTInsights: I listened to your podcast interview with James Lee from last year. And your journey to VC is really fascinating. For those who haven’t listened to the podcast, can you talk about some of the data challenges that you had when you were Head of Data at Mattermark?

Catanzaro: Yeah, absolutely. At Mattermark, we were collecting data on other startups, and then selling it to investors, primarily VCs. We had data challenges ranging from that to collecting and integrating disparate datasets at scale. 

And we were collecting data from CrunchBase and AngelList, and Twitter and Facebook and LinkedIn. And we needed to build a consistent data model that would be useful to venture investors to know challenges associated with transforming those datasets into machine learning models, or into insights using machine learning models. 

So we didn’t just want to collect a bunch of data and sell that to investors, we wanted to be able to generate data, thereby kind of creating value add, this was back in around 2013, 2014. So there were even fewer tools that were available to practitioners for building and deploying ML models.

And some of the challenges that we experienced pretty regularly were monitoring those models to make sure that no performance degradations were happening and scaling those models to new use cases. 

For example, at the time, any time we wanted to build a new system to classify companies by industry, we had to develop a new model. So there was a model for healthcare, there was a model for e-commerce, and each of those was kind of developed from scratch. So it could be very tedious as well. Maybe kind of a meta challenge, too. 

But as a data vendor, we wanted to ensure that we were selling a high quality product. But there aren’t super consistent definitions or standards for evaluating data quality. So we had to come up with our own data quality scorecard. 

Those are just a couple of examples of data related challenges that I recall. But I caveat all of this with it’s been about six or seven years.

RTInsights: How have the challenges evolved and changed since then? What are you seeing now?

Catanzaro: I think it’s become a lot easier to develop and train ML models. So it’s become easier to prototype ML models. There are a lot of deep learning frameworks, their architectures that are available, you have a bunch of these model hubs. 

So it’s easy to experiment with those new architectures. But I actually don’t think it’s gotten any easier to build ML-driven applications. I don’t feel like ML in production is really easier, is really more efficient to do. 

I think where we’re at is that we have kind of a first generation of tools, but many of those tools actually solve fairly narrow and specific problems. And those tools are not well integrated with each other. 

So in order to build an ML-driven software application, in order to move from prototype into production, ML teams need to kind of cobble together all of these pieces. And it’s not just implementing all of these pieces that’s hard. Even understanding how they ought to fit together can be really challenging. 

So I’m hopeful that there will be kind of a new generation of ML tools on the horizon, that make integration extensibility a primary goal, like tools that are designed to work together, I also think that these tools can be better aligned with a set of standards and best practices for building ML-driven software applications. 

And I think many of those standards and best practices are just starting to emerge now, as more and more companies move from ML experimentation into software development into building. 

RTInsights: The more things change, the more they stay the same. 

Catanzaro: Yeah, I think the obvious thing that has changed is that there are hundreds of ML vendors now, the non-obvious thing that has changed is that it’s still just as hard to get a model into production.

RTInsights: How so? You mentioned tools, best practices, and standards. What role do teams and experience play?

Catanzaro: Yeah, that’s a very relevant question. The beginning of this most recent wave of ML, development, ML experimentation, I think it probably started around 2012. That’s what I recall was the beginning of Alexa. So a lot of the ML, or, scientists and engineers or data scientists who are working at companies, they’ve only been working for 10 years. 

And, well, 10 years might sound like a long period of time, [but] 10 years is hardly enough time to really establish standards and best practices. 10 years is not really enough time to learn from experience. 

I think we are hitting this tipping point, though, where there’s a handful of people who perhaps move jobs every three years or so the seen ML in practice at at least three employers, they’ve seen what works, what doesn’t work, what problems constantly recur, and can start to really drive this discussion about best practice. But there are some phenomena that we just can’t shortcut. And the accrual of experience is certainly one of them.

RTInsights: Right. It’s interesting what you’re saying about them moving every few years, and, especially now with the acceleration of the great resignation. It seems like a lot of engineers are moving every year, or maybe even less than every year. Do you see that as potentially more valuable than if they stayed at a company longer? Because they potentially are seeing this in more environments?

Catanzaro: Yeah, I think one of the benefits of having more liquid talent markets where ML practitioners move jobs, is that when they move jobs, they can bring both their experience and their tools with them. 

So having people move jobs often actually helps accelerate the adoption of the tools. Because you started off with one company that was using your tool and now you’ve got two companies that are using your tool. So that’s certainly one thing.

I think the other thing is that having that sort of liquidity fee can actually drive Innovation because you find these practitioner-turned-founders who maybe built an ML platform at one company then they built an ML platform at another company then they built it at another company. And they’re like, “I’ve solved the same damn problem three times, I might as well solve this problem for hundreds of companies, and commercialize the thing that I built.” 

So I think that having more liquid talent markets enables these potential founders to identify and really recognize a broader market for the thing that they would have otherwise built at just one employer as an internal tool.

RTInsights: Can you talk a little about the companies you’ve invested in and what you look for in founders?

Catanzaro: Yeah, absolutely. I think I’ve already alluded to this paradigm of the practitioner-turned-partner, which is very clear in the Amplify portfolio. As an example, one of the companies that we invested in is called Outerbounds. They’re commercializing a data science or ML framework that the founding team initially developed at Netflix. 

So Metaflow, the open source project that they developed at Netflix, was initially intended to be an internal tool, something that would enable their data scientists to work much more efficiently, particularly those who weren’t working on recommendation engines and some of the kind of more complex ML projects that required half your enterprise infrastructure. 

But I think the team recognized, including through their experience in other roles prior to Netflix, that this framework could have utility beyond Netflix, which is what drove the initial open sourcing of the project. 

And then after seeing how they were creating value for companies like Zillow, or Expedia, ultimately, they did decide to commercialize the project, and really focus all of their effort on building this tool, not just for Netflix, but specifically for a much larger audience. 

I think one of the things that we really liked about this kind of practitioner-turned-founder model is that they have a deep understanding of what problem they’re trying to solve, as well as what motivates the people who adopt the solution to that problem. So they tend to have a very clear product vision and very strong user empathy, which I think matters a lot when you’re building things for technical practitioners.

RTInsights: Right. And, and it seems like there’s more of its discussion right now in terms of end users and the systems that they’re trying to build. There’s a discussion around what systems need to be real time or if they should be real time. What are your thoughts on that?

Catanzaro: A lot of ML applications, I think, will require real time inference. So things like personalization, things like recommendation engines, you have user data that is arriving in real time. And you need to make predictions at the very least in near real time. 

Although, if you can generate predictions in real time, I think it affords a much better user experience. For software developers to build real time applications is really hard, like managing streaming data. I think this is still way too difficult. And again, some of that will be addressed through the passage of time, tools like Kafka are still relatively new [for example]. 

But right now, I think the burden on ML engineers to connect to real time data sets, like just connecting to real time data sets is really tough, to understand the impact of schedules, to understand the impact of attributes of real time management, data management systems, consistency guarantees on the performance of an ML application is also challenging. 

So I think the thing that we should constantly remind ourselves is that whenever we say real time application, whether it’s in the context of a more traditional software application, or an ML driven application, we are talking about a very serious engineering effort. And, a lot of investment. I’m hopeful that streaming will get easier for everybody, not just ML engineers, but today, it’s super challenging.

RTInsights: What do you think can be done to get more women into data?

Catanzaro: One of the things that I think a lot about, both for myself and for others, is having more role models. 

I remember a couple of years ago, as a woman in venture, looking at the boards of many of the companies that I was investing in and thinking, there are no women here. And sure there are. 

There’s structured bias and reasons why there are not more women on boards, more women in data, but I remember worrying, “Oh, is it because of pregnancy or motherhood…” And I wish there were people who I could’ve talked to about that. 

So I think if we want more women in data, if we want more women in venture, we need to have more role models, more women who are willing to make the time to talk through these things with other women as they navigate their career. 

RTInsights: What do you think the most important trend around cloud data will be in the next few years?

Catanzaro: It’s a tough question. Because on one hand, I feel as if data teams are still trying to master the basics. I see so few companies that are good at like, just basic measurement and reporting. 

And frankly, I think that becomes even more important in a downturn when companies are really going to be judged on their metrics and you know, not just sentiment. 

So I do think, paying attention to these kinds of boring topics, like how do you decide upon the metrics that matter and ensure that you can calculate and analyze those  metrics in a reliable way is an important topic.

I think the thing that is very sexy though, in the data and machine learning space right now is how you build applications. How do you build new user experiences with data and ML, including by leveraging some of the work that you’ve already done? 

So if you’re investing all of this effort to do basic reporting, like basic, you know, MRR reporting, etc., you’re going to have to create a whole series of data transformations and dashboards, etc. Instead of starting from scratch, when you’re building an application, including an emulator with an application, like, how do you repurpose what you’ve already done? 

I think this whole notion of a data science Hierarchy of Needs built upon rigorous counting is an idea that teams will start to revisit as they master counting, if you will.