The interconnected and digital nature of our world has left businesses with more data at their disposal than ever before. This data tends to exist in the form of continuously-generated streams, fueled by our never-ending consumption of digital experiences. The opportunity to use streaming data to make better decisions and improve the customer experience is both obvious and critical to the long-term success of any modern business. However, it has proven elusive to all but the most technically-sophisticated companies to date.
As an investor who spends much of his time thinking about the future of software and data infrastructure, understanding the aforementioned “gap” has been a major focus of my work over the past 18 months. I was convinced that “streaming” would be the next wave of opportunity after cloud, and the proliferation of Kafka was only the beginning. After countless conversations with database systems researchers, engineers, and founders, a clear set of takeaways emerged. First, traditional approaches to analytical data processing (ex: data warehouses) are ill-equipped to unlock the value of streaming data. Second, newer open-source technologies for processing streaming data have known technical issues that limit their applications absent significant custom engineering work (microservices!). Lastly, the lack of availability of a true SQL interface to access streaming data imposes an unfortunately low ceiling on its impact. SQL is, after all, the lingua franca of data and most commonly required skill in tech.
This is why we are excited to announce our partnership with Materialize, the company behind the first streaming SQL database. Unlike current stream processing technologies, Materialize incrementally re-computes the results of a query as new data comes in, correctly and at low-latency. This is a transformative capability for any business with streaming data in that it enables the correct result of a query to be available whenever needed. Materialize also provides the first and only standard SQL interface for streaming data, and shines in its ability to carry out complex queries and multiway joins. This means that instead of having to develop microservices to carry out the parts of computation the stream processor is incapable of handling, a developer can express their logic in a single SQL query. Ultimately, Materialize delivers the ease and simplicity developers have grown accustomed to when working with batch data, to the world of streaming applications.
Materialize was founded in early ‘19 by Arjun Narayan and Frank McSherry, in NYC. Arjun was an early employee at Cockroach Labs (thanks for introducing us, Spencer!), where he helped develop what has emerged as the leading multi-cloud, transactional database. Frank was previously at Microsoft Research Silicon Valley where he co-invented Differential Privacy, and led the Naiad project. Frank also created Timely Dataflow and Differential Dataflow, which serve as the technical basis of Materialize. They are joined by a stellar team who were early employees at companies like Cockroach Labs, Datadog, Dropbox, Pure Storage and YouTube.
As more business data moves to streaming, we see Materialize’s ability to democratize access to it as the foundation upon which they will build the industry’s next major data platform. Materialize’s growing list of customers are using it to support real-time visualization, financial modeling, and in-app analytics experiences across a variety of industries including logistics, marketing technology, financial services, and ERP. My partners and I are honored to be supporting the company on their mission to make the value of streaming data accessible to all. If you are interested in joining them, they are hiring!
ーBucky