MemSQL Showcases Real-Time Data Pipelines at Kafka Summit
Instant SQL analytics and easy Kafka connectivity deliver immediate enterprise value
San Francisco, CA - April 26, 2016 - MemSQL, provider of the fastest database platform for real-time analytics, is showcasing its work with real-time data pipelines at the Kafka Summit today in San Francisco. As real-time workloads gain prominence at companies competing in the digital economy, MemSQL provides multiple ways to persist recent and historical data from Kafka, and through its use of SQL, enables real-time analytics for enterprises.
In conjunction with Kafka, MemSQL allows companies to rapidly ingest, process, and serve data at the speeds required for today’s digital business. Kafka is recognized as a leading real-time message queue, and when combined with MemSQL, the database platform for real-time analytics, customers can immediately run sophisticated analytics on live data. By embracing SQL, the lingua franca for enterprise analytics, MemSQL makes real-time data from Kafka accessible to analysts, as well as a comprehensive ecosystem of SQL-based tools such as business intelligence dashboards.
“As the world quickly moves to real-time applications, Kafka provides an efficient mechanism to manage data streams,” said Nikita Shamgunov, CTO and co-founder, MemSQL. “When combining Kafka with an in-memory database, users can immediately and flexibly query both real-time and historical data with SQL, providing easily accessible value to enterprises.”
MemSQL provides multiple mechanisms for Kafka to write data to MemSQL:
Application allowing Kafka data to flow to MemSQL by identifying the Kafka cluster IP address and topic name. MemSQL handles the rest. Streamliner also includes an integrated version of Apache Spark for real-time transformation of data flowing into MemSQL.
Kafka Connector with JDBC
MemSQL supports industry standard connectivity through JDBC.
With either mechanism, Kafka streams can be persisted in either the MemSQL rowstore, residing in memory, or the MemSQL columnstore which uses a combination of memory and disk. This allows for complete flexibility of media types while retaining a simple SQL interface across all data.
"Customers can quickly derive value from real-time streams flowing through Kafka when they are persisted to fast databases providing comprehensive analytic functions," said Jabari Norton, VP of Business Development, Confluent. "MemSQL provides such value through multiple integration options with Kafka and the Confluent Platform allowing customers to ingest, process, and analyze data in real-time."
End User Speaking Session
At the show, MemSQL will be featured in an end user speaking session:
Real-Time Analytics Visualized with Kafka, Streamliner, MemSQL, and Zoomdata
Anton Gorshkov, Managing Director, Goldman Sachs
5:20 PM-6:00 PM, Imperial B
Despcription: Building a real-time pipeline from scratch that is able to handle billion+ transactions per day, store, analyze and visualize it all in real-time has never been easier. In this build-as-we-go talk, we’ll create a front-to-back architecture that does exactly that.
- We’ll start with a simple producer emitting a few messages and publishing them onto a Kafka queue
- On consuming end of the queue a Spark-based Streamliner process will pick them up and store in MemSQL
- Zoomdata will connect to MemSQL for real-time visualization where we’ll be able to ask various questions and see answers change as data is flowing through the system
- We’ll quickly make the entire pipeline more complex by increasing the amount of data as well as complexity of the data, until reaching 100K transactions per second
As we walk through this demo, we will touch on cross data-center Kafka and MemSQL set-ups, speed limitations if any as well as echo back to real-life use cases of a similar set-up used in Goldman’s Asset Management division for the purposes of Portfolio Management & Trading.
Live Demonstrations at the MemSQL Booth
Drop by the MemSQL booth to see live demonstrations of MemSQL including PowerStream, a real-time showcase application collecting up to 2 million updates per seconds from wind turbines around the globe.
About the Kafka Summit
Tuesday April 26 at the Hilton San Francisco Union Square. Visit kafka-summit.org for more information.
MemSQL delivers the leading database platform for real-time analytics. Global enterprises use MemSQL to achieve peak performance and optimize data efficiency. With the combined power of database, data warehouse, and streaming workloads in one system, MemSQL helps companies anticipate problems before they occur, turn insights into actions, and stay relevant in a rapidly changing world. Visit memsql.com or follow us @memsql.