First Look Video: TIBCO’s Apache Spark Accelerator

we're here today with Hayden Schultz global architect for tipco and we're going to talk about some new products that are coming out let's start with uh tell me a little bit about yourself and how you came to tipco I was working for streambase a a small startup in the Boston area that makes uh the streambase event processing product that tipco acquired in 2013 I've been working on event processing for the past almost 13 years it's with streambase um doing mostly uh um Financial Market things and high frequency trading uh currency trading um a lot of that a lot of asset classes now that tipco own streambase we' moved into all kinds of things um vehicles uh internet of things and today we'll talk about big data we see a lot of solutions that we find repeatable um but that's not necessarily obvious from the customer point of view okay know we have a lot of different products and rather than just giving the customer a bunch of pieces a bunch of parts they can build complicated things together what we're doing is we're putting them together in common application framework so that we build an application in the case we're going to talk about today uh Big Data application um and what it does for customers it gives them an example of how the products fit together and how they work with other Technologies in in a scalable reliable uh best practices um solution so that you can take this and customize it to something else and we've been calling it a product but this is something that we're giving out this is not something that we're selling sample implementations so there are full systems which you can take and and customize to use yourself we're we're releasing them with an an open source license and do whatever you want with them so we started focusing on Hadoop um and Hadoop is in there it's you know part of a fundamental uh a fundamental uh building block that Apache spark uses but when we look at what's the dynamic um growth in the Big Data Arena right now we see it in Apache spark one way of thinking about this is while the accelerator is a new thing that we're we're giving out the various components are not really new in the sense that any of our customers could have taken our products and built systems on top of Big Data clusters built systems on top of spark um they do that without us but what this does is it lets someone who's new to it use these components so we have um an example of customer what they're doing with uh with streambase and Spark is they have their um their uh currency trading algorithms are written in streambase and what they want to do is they want to do back testing uh which means they have a large amount of historical data and they want to know whether their new algorithms perform better than their old algorithms now if they knew what the new currency data was going to be like that would be easy it would be no challenge at all so but they don't so what they do is they store in their big data cluster they store you know all their financial trades that happen and they store all of the raw uh um the raw FX quotes you know that happen every day and so what they do to train their new algorithm to evaluate it and compare it against their old algorithm they take six months of data they partition it into one day chunks which is turns out to be like 136 uh different eight hour chunks of data okay and then they run the new algorithm where streambase is a is running in a worker thread inside a spark cluster and so they take 136 simultaneous uh um partitions of data and they run them all in their cluster and they end up training six months worth of data in under an hour interesting so it used to take them you know when they started doing this it was you know you know many many hours you know they would typically think of it as in days and now it's under an hour so they can do um a lot more experimentation they can they can evaluate different changes and and and get their um their new versions of their algorithm tuned faster and executed so the the accelerator is a sample solution it's assumes that you are are using the tipco product set now you can swap in and out other components if you want but what we start with is stream base for capturing the data so one typical example is um you have a Kafka bus yeah maybe that's the initial source of the data or JMS bus or just just a socket or web services uh there's there's a large variety of adapters we use to connect to the data SCE and so those are very simple stream based applications um that connect the data source and then write it into directly in hdfs or possibly write it out using Flume and then it's in the Big Data System so once you have a large portfolio of of data to look at what you can do is you run your your data analytics at the on the uh data scientist comes out and uses in the tibco stack they would use Spotfire and so they'll look at the the data connect using um there's a a newly certified by data bricks spark connector from Spotfire to spark system uh they can run spark SQL commands uh they could run R commands directly from um from Spotfire they can even use um tear the uh the tibco Enterprise R runtime okay uh um to analyze the data as well so in the case of the the the the accelerator for Apache spark what we do is Spotfire prepares the data understands the the relationships and once that's done it uses um sparkling water uh an H2O layer on top of Apache spark okay and that trains us to a machine learning model okay and so machine learning model is now saved it's trained and saved inside the big data so it's basically serialized out to hdfs so if I were to try and sum it up and you know if I were talking to potential customer of yours it sounds like the important thing is that when you're looking at an accelerator the acceleration what you're accelerating is the time to Value it's an overused term but it's the time to deliver something that's usable because you have this template exactly if you look at why do projects fail and and at these large customers these large companies a lot of projects fail and the reason is it takes too much investment on the company's part before they start showing some return and so the idea here is we're going to give you something that's already it's front end already and all you do instead of all of the um you know all of the plumbing that's that's done you work on the business Logic for someone that hasn't been working with you or someone that hasn't been looking at this how are they going to get started they download this uh uh from the tipco uh accelerator download sites um and they have the full source to everything a totally open product do whatever you want with it

This transcript was generated automatically from the video's captions and may contain errors.