Smart Talk Episode 9: Apache Iceberg and Streaming Data Architectures

hello and welcome to this episode of smart talk a day motion leadership series I'm your host denh Chandra Shaker chief analyst and founder at sta joining hats with the RT insights uh we are pleased to bring toor shiran uh to this episode we're going to be talking about uh dro and Apache Iceberg and the importance of that in today's modern data architectures uh uh toor is a well-renowned technology leader having worked in companies like mapar and Microsoft he started Dro almost like nine years ago and uh we are excited to have D um toor on this particular episode uh welcome toor thank you I'm I'm excited to be here so let's begin with Dro tell us about Dro what was your vision behind starting Dro um nine years ago yeah so Dro is the hybrid lak house um for the business and uh I think we'll have time to unpack that but at a high level uh we allow companies to um you know analyze their data regardless of where it is right so they don't have to put it into a proprietary and expensive um you know cloud data warehouse um they can Corry the data wherever it is and we really appeal to that uh kind of business persona so you know a business analyst a business user folks that use bi tools or write SQL queries um and uh and that's in contrast to others in the market which are more focused on a very technical user that is using python or you know spark Scala things like that um and there's a lot of things we do in the product to make it so that it can be you know appropriate for a less technical user and that's everything from the kind of the entire experience all the way to you know taking away some of the things such as data optimization like we we make that fully autonomous all happens behind the scenes you know the data gets partitioned in the right way and organized in the right way based on the workloads uh and so we do a lot of work and a lot of innovation to make it so that you know anybody can really take advantage of data awesome and there is the undeniable uh elephant in the room which is Apache iceberg that we have to talk about because that is largely Associated when you talk about Dro so tell us about Apache Iceberg and your uh driver behind being supporting that for so many years sure yeah so actually Iceberg and you know other comparable kind of uh what we call table formats didn't exist when we started Dro um uh but really uh I'd say I could get this wrong around five years ago I think um uh was really the beginning of kind of this new era of the the data Lake um and so and prior to that you know Dro was a query engine right uh you could query data but it had to already be there right it had to be you know if it's if it was in your Lake and in object storage then you had to have the data already there um ideally in a format such as Park so you could quer it with high performance um but increasingly what we kept hearing in the market is that companies you know they were really excited about being able to use an open data architecture where data is in their own you know S3 bucket or their own Azure storage uh you know account um but they wanted the kind of the all-in-one Simplicity of a data warehouse where they can do inserts updates and deletes as well not just kind of readon queries um and so table formats provide that capability right they they enable uh various engines such as Dro to not just be able to read data but also to be able to update it right and so it think of it as kind of a metadata layer on top of files so historically you know data Lakes were just a bunch of files right in object storage but the problem is files are immutable you can't change them right um and so a lot of kind of you know use cases were were just too complicated once you think of data and kind of this higher level abstraction of a table um you can start to interact with it you can you know change things you can you can update records you can delete records you can change schemas and do all of that you know just the way you would do it with any kind of database um and so we uh you know we identified Iceberg as a potential way to kind of solve this problem at the time that we did that it was just a project an open source project that Netflix was working on and um you know apple as well and you know some some of these companies us used Dro and so we had a lot of kind of inside knowledge of of kind of this how that was going and and their experience with it and so we decided that we were going to kind of um you know really focus on on making these kind of Technologies a reality um we initially started evangelizing Iceberg right when it was kind of entering the the pach Pache software Foundation as an open source project we we put some um kind of analysts sorry um evangelists on the project to to really get the word out um and you know long story short within a couple years Iceberg became um you know broadly adopted not just by our engine but by all the major players in the market you know whether it's you know Amazon Google snowflake um confluent um basically everybody um kind of decided to to embrace Iceberg as their standard table format and I think that's really important also when when you think about the value of an open data architecture and kind of data lakes or or lake houses as they're called now having the data in an open format that different engines can operate on that's really critical right it's not something that one company can just you know create their own format and nobody else supports it because that defeats the whole purpose of an open architecture and then it becomes like a proprietary Thing by itself and without any kind of uh just like warehous uh I get that and congratulations on you know the support that the entire Community has been showing for Iceberg I every single conference every single you know vendor that I talk to uh have been widely appreciative of like Iceberg and their support for Iceberg so I think it's definitely growing for sure but I would also say that we've also seen an emergence of other such table formats in catalogs right so what has led to these different variations and tell us about interoperability from your Vantage Point how do you see that yeah well I think we should separate the discussion into you know like table formats and cataloges and I'll address each of them separately you when it comes to table formats Iceberg was not the only game in town actually data breaks um had created their own table format called Delta Lake and for a while it was kind of like this two horse race um there's actually a third project called hoodie which Uber had created um but that kind of became I think more in kind of in the background didn't get as much traction in the in the community um so it was really a two-horse race between Iceberg and Delta Lake and you know we were all over you know Iceberg from an evangelism standpoint you know on social media blogs conferences and so forth um but you know datab break has a lot of customers and and many of them were using Delta Lake um and I think what ended up happening and what ended up kind of tipping the scale here is the fact that the rest of the the technology providers you know they they wanted something something that was open right that that was a lot of the value again of of kind of the This Modern data architecture is having the data in open formats right and so you know going with something that is uh kind of an Apache project that's open source that different companies can contribute to I think that was super valuable and once once all the kind of the key major players you know Amazon Google snowflake Etc once all of them decided you know that Iceberg was their format as well I think it became clear that that was kind of winning um you know data bricks then went and bought u a company that was founded by some of the craters of Iceberg um they spent uh something like $2 billion on that um and uh kind of I think that was the the final maybe stamp uh that kind of sealed that that two horse race and and uh declared Iceberg the the winner um I I think the longterm what what's going to happen there's still Delta Lake out there but I think what's going to happen is the two will end up kind of being closer and closer uh um to each other where at some point most people will just think about it as as Iceberg and and it'll be kind of one format um for for everybody the catalog is an important ingredient in all of this because when you are thinking about table formats there has to be some service that has the list of tables in the in the environment and can point to the metadata for each of these uh the iceberg metadata for each of these tables right that's kind of the core foundational principle of of these cataloges which are also called metast stores right they're kind of technical cataloges to discover everything else well I'd say that's even a higher level you know beyond that but but even at the technical level you need something that can kind of manage the transactions the updates to these tables H and there were different options there like you know some people still use the hive meta store from back in the Hadoop days that's still pretty popular Amazon has a hosted version of it called glue um there's just different options out there um you know Google had you know a couple of these meta stores one as part of kind of big query one as part of their data proc process so all sorts of options you know what we ended up doing is partnering with snowflake to create an open source option and that's called the Apache Polaris and so Apache Polaris is an open source um Iceberg catalog basically that anybody can download you know for free it's it's an open source project um you know it's not just us and snowflake contributing there's now committers from other other companies as well including the large Cloud providers and I think what's going to happen is that Polaris will become the standard um Iceberg catalog that that everybody uses ultimately awesome so one of the things that we talk about in this particular uh Smart Talk series is about real-time data and so forth um particularly realtime data uh so we we uh understand that you know one of icebergs you know promises is also about like know unifying all different types of data badge streaming everything you know into this Common Table format and so forth um how do you think it helps technology leaders you know look at you know data architectures you know currently from how they were looking at before and does it even reduce like operational overhead and things like that you know tell us about like some benefits of like iceberg in this particular context of bringing together all the data in one single view sure well I think there's a couple a couple aspects here one is in general if you're going to create a platform where you're going to centralize data right you probably want something that is going to stand the kind of the test of time right and um and Iceberg and in General open kind of formats um are much more likely to do that right if you think about proprietary Technologies and vendors right there's you know every five or 10 years there's something new and people want to adopt that and they they end up moving to that but if you can have your data in an open format and that is supported by a variety of different tools and engines right you have you know Dro and Spark and you know data brakes and Snowflake and you know confin lots and lots of tools that support Iceberg Andor there will be new startups and new technologies that support it well then you continue to enjoy all this Innovation over time right and so you're kind of future proofing Yourself by by choosing something like this and not putting your data inside of a a warehouse right like like a snowflake or a you know something of that nature right or or redchief um and so I think that's one thing is just having this infinitely scalable way to store data in an open format that encourages you to kind of centralize and and standardize and I think I think the second thing is um you mentioned kind of streaming data so for uh you know for a very long time data Lakes couldn't really play nicely with streaming data because you know that a data Lake was kind of defined by just storing a bunch of files right and these files are you know there's no transactionality to them there's no kind of consistency that's easy to kind of manage between you know readers and writers um and with Iceberg that's different because operations such as updating a table you know adding records is an atomic operation right and so you can stream data into an iceberg table let's say you know do something like CDC from some database into Iceberg somebody else could be reading these records as they're as as they're being written and so that having that clear kind of transactionality in the in the format um unlocks these types of use cases and I think that's one of the reasons that you know you see someone like conflent right which is the the company behind Kafka um adopt Iceberg as their storage format now as well yeah and I I I feel that you know it brings about that uh notion of data democratization in a way where suddenly this becomes like the single central place where people can push data into and of course as you said others that need to like read data off such tables can do so as well and you're bringing about a diverse set of uh tools and Technologies being able to intercommunicate so easily because of this one common open standard that we put in place yeah that's that's exactly but but then that also begs the question about you know data governance right so that becomes almost like an instant uh question if you know I can push all my data into this from like all these different tools and then I can also read from it and all that then the question about data lineage and consistency and auditability compliance all those become um kind of like Prime importance how do you you know advise organizations when it comes to iceburg and like you know subsequently with data governance yeah well you know one of the roles of the catalog right the lak house catalog such as Polaris um is to provide capabilities such as Access Control right um and so of course you have to make sure when you have a platform like this with all this data that you know the data is only being accessed by the people that should be accessing it right um now what we've done in Dro right with our uh you know Lakehouse engine is we provide all these capabilities kind of for the Enterprise right so Enterprise authentication Enterprise Access Control right integrated with all the different systems you know Azure active directory OCTA Etc um uh you know auditing for example who did what and you can query these Audi audit logs and get a really deep understanding of who's touching what data is somebody accessing pii where they shouldn't be accessing pii and you can understand lineage across different you know data assets and and so there are all these Enterprise kind of or or governance requirements that need to be solved and I think that is addressed through kind of the the catalog layer and and Dro provides kind of a play powered Enterprise catalog um and also through kind of the engine itself which uh obviously is like responsible for things like the the auditing for example okay um you spoke about um efficiencies when it comes to again uh from a data architecture perspective and all that but when you talk about storage and compute um in the context of Iceberg uh how do you see this you know how should like you know data leaders consider this um in their larger data architecture designing you know how do they fit in this so that it brings in cost efficiencies particularly when it comes to storage and compute yeah look we you know we we see when when customers move to Dro from something else such as a cloud data warehouse right um often their costs go down by at least you know 2x right like 50% um and and so there's a a huge cost saving just from like the efficiency of the the the engine and the fact that it's an open architecture um already makes it much more efficient provides a much lower TCO and so you know storage is cheap at the end of the day that's not where most of the costs are um um and so what we've also done for example at Dro is realizing that you know storage is cheap and compute is expensive we will actually optimize the data in the storage in different shapes and forms so you may have a table um uh you know of like transactions or uh you know some some kind of event data right we might maintain different aggregations of that data and different sorts or partitioning schemes of that data so that different workloads can actually come and and get much higher performance and not have to scan the entire data set for every query that's being run right and so you know we can store with some storage overhead let's say 10% storage overhead we can give people you know 10x faster performance on average for their entire workload just by by doing things like that um and so there's I think a lot of opportunity out there to um kind of optimize performance and you know usually when you're optimizing performance you're you're really optimizing cost right um um especially in this era where you can scale as much as you want it's question is how how much is it costing you right can you maybe share a specific example of a company a customer yours maybe that has uh leveraged Iceberg transformed their data operations some of the business outcomes you know from such an implementation yeah you know we I mean there's hundreds of these these companies that we work with that are you know kind of Enterprises for the most part leveraging leveraging Iceberg and so forth yeah one recent one that I was talking to a large large Bank um they had you know combination of data um some in the cloud in a cloud kind of data warehouse um and the costs of that had been growing kind of year-over-year to a point where you know they just realized hey this is this is just not sustain aable anymore we can't um we can't keep you know growing the these costs and at the same time they also had data in other data sources including some that was on Prem and so they brought in Dro um deployed that in the cloud um in a way that actually the compute runs in their own VPC um and they connected that to their various sources and so for the data that was in the cloud data warehouse they just moved that into Iceberg tables um uh on on S3 and kind of object storage and then because Dro has connectors to all these other data sources as well including their on-prem data um they're able to kind of bring all that together Define views that include even joins across different data sources and now they have kind of one pane of glass and much lower costs as well for all their analytics and so um that that's a great one because you know taking advantage of all sorts of capabilities in in the dro platform um you know often when we're starting with a new with a new customer um they're kind of Landing Dro in as kind of a a unified analytics layer kind of like data virtualization right they they can Federate across different sources and then gradually they'll take data that is sitting in more expensive systems and move that into a kind of a more modern open data architecture right see a lot of migrations from Red shift for example into kind of Iceberg tables now and um just you know doesn't scale gets really expensive and hard to manage and then you bring the data into Iceberg you put Dru on top you know you're able to serve all those use cases uh you know with far less you know operational challenges and you know 10x lower cost that kind of thing okay I I had a question about you know Silo Data Systems which is like very common when it comes to Enterprises and they all struggle with it and you spoke about connectors that you know allow you to like you know plug in various different data sources very easily into Remo yeah uh my my question would be more from a sense of uh the flexibility or like the the diversity of such data connectors and maybe you can talk about that but also from a query optimization and Effectiveness perspective because there's always that tug of war between like you optimizing the query then you compensate on something else and so forth or there like benchmarks that you like you know the audiences can like you know look for look up and like you know find out more about that yeah you know historically you know people have thought of like data warehouses and and you know data lake houses maybe and data virtualization or Federation as two separate kind of like systems and and what we've done at Dro is we've kind of blurred the lines between those two things we've we've built the best we started with just building the best query engine right we we invented Apache Arrow which is now downloaded like 70 or 80 million times a month for really high performance kind of inmemory processing so that's the foundation of our engine is being able to run cor's extremely fast on on object storage but we also built the ability to connect to these other databases um and to be able to push down query processing into those systems and so you might have most of your data now say in iceberg tables or parket t parket files in in you know object storage but you still have data sitting in Oracle and mongodb and these other systems right and you know some data warehouses and you want to incorporate all of that into kind of one shared kind of semantic layer right um and so Dro can do that and when a query comes in we automatically figure out what part of it can be pushed down into the underlying system and that includes you know really complex you know SQL queries right joins and you know correlated subqueries and all sorts of kind of constructs that are most systems can't push down into an underlying system so we take the query we break it apart maybe a simple example would be I have two tables in postgress um and you know two tables in the in in the object storage and I'm doing a four-way join will push down the two-way join of postgress into postest let postest do that join there and if possible if there's filters that can be pushed down as well so that you know the minimum amount of data actually has to come back from the postgress system and so that's the default mode of operation H but we give customers the flexibility because sometimes they don't want you know they they have uh maybe too much load on that database right and they don't want to put a lot of load on that um or maybe that system just by the nature of it you know like like say elastic search is is just really slow at scanning data right and so asking it to do a lot of processing is actually kind of counterproductive and so what we uh what the system can do is it has this caching technology called Data Reflections which allows to maintain different caches of the data that's in these different systems so we can cache that and the user is not aware of that the user is running the query like from a bi tool or just running a SQL query they they're not aware of these caches the caches are actually stored as Park as Iceberg tables inside of inside of the object storage but to the user they're just interacting with data from all these systems and we will go periodically and and update kind of the cache based on those systems um and the cach doesn't even have to be as simple as just a raw copy of that a table somewhere right it could be maybe that raw copy but also different aggregations on that data so we're constantly analyzing the workloads and figuring out okay you know this is what we should be persisting at least for now in object storage we're we're constantly reevaluating that those decisions um so that users are just getting this really amazing kind of subsequent response times when they're dragging the mouse and Tableau or powerbi or something like that you know we don't want them to go and have to create extracts in their bi tools because those are a nightmare to manage and they really restrict kind of the kind of analysis you can do and and it's about data freshness as well you know you also touched on a very important topic which I want to bring up which is when you talk about these multiple data silos the concept of the data caches is like really resonating well with me right so that every time you're not doing the data hop and like trying to get data back and forth and all that and if there is a way to like cach it more closer to uh where the you know access is I think that's making it more fresh in a way but at the same time there are probably uh data sources that uh keep updating like as I said you know a realtime source for example if that be the case how do you you know compensate for that how do you you make sure that the data still you know refreshes in a timely manner so that there is no latency uh from um where D fitches the data yeah well first of all we give users the ability to you know control that at some level it's a business decision like am I am I okay with like the data the results being 15 minutes stale or am I not okay with that and that depends on the source of data right for some things in the use cases so for some things it may be okay for others it's not and so it's very easy to just set that in the system but the second thing is in the last year we've actually released something called Dro live Reflections and the idea with live Reflections is that we can update these different cached elements these Reflections um you know kind of live as soon as as soon as some data changes in an iceberg table we immediately know what changed and all the reflections that are kind of derived from that unit are automatically refreshed in an incremental way and so we spent a lot of time building kind of the ability through these incremental updates in all sorts of kind of different situations and all sorts of crazy scenarios right um because the the ability to to update cashes live is really depend on how good can you do that incrementally because the moment you have to do a full kind of update it just takes time right in a large data set right if we're talking about a million records sure you can do the full thing every time but when we're talking about billions or trillions of records and and many of our customers run at that kind of scale then you have to make sure that you can do this stuff much more intelligently and so we spent a lot of time built a lot of Ip in this area of incremental uh reflection updates and being able to understand also with the query Optimizer when a new query comes in what is it okay to use and what is it not okay to use right and so you can imagine a a you know sometimes I look at these things our support team shares with me of like queries that are like 10 pages long right and so our Optimizer has to be able to take this this SQL query that's massive right break it apart figure out okay I can take this if if I rewrite the query plan in this way it's still logically the same but then I can replace this part of the tree with you know some data that I've pre-calculated already and this other part of the tree with data that I've also pre-calculated and it's all going to be consistent and the user is going to get the same result they'll just get it you know 100 times faster right and so yeah huge amount of Ip and patents that we've built in in kind of in this area I think you brought up also another point that I think is uh very appropriate for talking today which is when you talk about optimizing a query that is like 10 pages long uh that also begs the question how much um have you Incorporated like some of the newer uh AI advancements and Innovations into these offerings uh and how has that helped in you know speeding up some of these things yeah you know AI creates um you know tremendous amount of opportunity I think uh at multiple different layers and so you know in terms of the how we build the system and how some of these things work which is kind of what you're talking about um in terms of how users can interact with a system right like we allow them to ask questions now in natural language and give them responses based on you know our understanding of the metadata and actually the the values and so forth as well um and also in allowing users to build their own kind of AI applications right like utilizing unstructured data and so forth and so you know we we have a lot of investments in all these different areas um specifically in the area of kind of the the core kind of capabilities of the system and how they operate um one of the things we launched recently is reflection recommendations where we do analysis of the query workload and we learn from that and you know the system gets smarter and smarter as it sees more queries and it figures out okay like through this kind of learning what are the right set of materializations to create automatically in the system that will maximize the amount of cost Saving right or the amount of uh performance increase and then as the workload changes because the workload is not a static thing it's not like you know you ran this today and tomorrow you might you're guaranteed to run the same things as the workload is changing over time those decisions are constantly reevaluated right and so we look at okay each of those constructs that were created each of those Reflections that we created what is the actual value of that what has it been over the last 24 hours what has it been over the last seven days and you know we can then figure out okay well maybe we shouldn't be doing this anymore we should be doing some other set of things instead right and so that constant learning and that's just something people had to do in the past right um and it was super complicated of course people can never get it right in most of our customers if you look at it most organizations ations the team that's running the infrastructure the plat the data platform is not the team that's cing the data they're not the ones using the data right and so there's this disconnect where the the the platform team doesn't understand the workloads they can't right there all these different people and different teams and use cases um and so they had no really good way to figure out how to optimize the data right because they just couldn't they didn't know what people were doing other than looking through a big log of of SQL queries and certainly couldn't anticipate things that were going to happen in the future um and so just for the most part data didn't get optimized and the data would be partitioned by a date column and then 80% of the workload was actually quering by the device ID column or something like that right it's just like terrible results um and so by making this all intelligent and automated and you're not just taking work away from people that had to do it manually but you're actually getting to an outcome that just couldn't be achieved before right before that kind of automation very cool and I'm glad you spoke about that part of you know the Innovations that you brought into the product Set uh because you know almost every vendor today talks about you know natural language to SQL and that kind of stuff you know which is can appreciated but at the same time it doesn't like you know quite differentiate itself but the real problems are like what you said about like the reflection re recommendations and things like that I think that's where operationally there's a ton of optimizations that could be done and uh more than like manual intervention something that could automatically analyze the workloads and like make those kinds of recommendations is absolutely uh beneficial so thank you for sharing that um one last question maybe uh would be if I were to be an Enterprise data leader out there and I'm looking at revamping my data architecture I'm thinking about iceberg uh as like a strategic uh important thing to be uh part of my data architecture um how would you advise like what kind of strategic considerations should they prioritize maybe like the top three or something uh that would maximize the benefits of the data architecture particularly from like future proofing it right so because we are thinking this is going to be open standard and it's going to be for the future and all that uh how would do you recommend a data leader like that yeah well I think you you hit the nail on the head that the data has to be open when I say data like I think having and I say open I mean like open source like an Apache project and so um I would encourage anybody to make sure you're using kind of like Iceberg as the table format underneath that is par as the the actual file format behind the iceberg and then using um going forward using Polaris which is an Apache project as the as the metast store right as the catalog I think those kind of those are the the three the or the core pillars of kind of being open um and then you know I think the next step is you need a SQL engine right that's fundamentally you know the key kind of piece of dro right is providing that ability to run SQL C ques um on top of dro I'd say use something like DBT on top of that you know DBT scheduling the the dro work kind of the workloads the queries to do the Transformations that you need to do and but that alone I think gets you pretty far in terms of like okay you can replace your data warehouses you can run all these workloads and have a very modern kind of uh modern data architecture maybe maybe one more question then because I think uh yeah I'll be remiss if I didn't ask you this sure given the kind of uh you know technology leader you are you know Hands-On and you've been in the space you know building up uh some of these you know open source Innovations and so forth what are you more excited about in the next couple of years in this particular data space you know what are some of the things that are cooking that we as Outsiders probably don't know that you as an Insider you know know and are excited about yeah um look I think there are you know different a number of different areas but but yeah I think we kind of touched on it AI is is creating a new opportunity right the the AI models that are out there you know all of them really like everything from you know GPT to you know Claude to you know Gemini they're all getting really really smart right and that creates opportunities to do things to automate things that in the past required you know people to do and people to do kind of things they didn't really want to do right um and so I think that is going to allow us as an industry to get closer and closer to kind of that Holy Grail of you know allowing people to really take advantage of their data and not constantly kind of be stuck waiting on some transformation to be done before they can access the data not constantly questioning the quality of the data I mean there's just so much stuff that needs to get better in data despite Decades of kind of innovation and new technologies um you know we're just not far enough along today in data it's still kind of a mess um and I think there's a lot of opport opportunities to um that that we're working on and and and others I'm sure are working on um to to really change the game very well said well toor thank you so much for joining us today this was a fantastic conversation we like to keep our episodes you know short and susing but you know sometimes I go a little overboard primarily when the conversation is like super exciting like this so thank you for sharing all your insights about Remo Iceberg and how you see the world of data and Beyond so thank you so much yeah for sure thanks for having me here

This transcript was generated automatically from the video's captions and may contain errors.