A data streaming company The confluence Just hosted the first Kafka Summit in Asia in Bangalore, India. The event saw a large turnout from the Kafka community—more than 30% of the global community comes from the region—and included numerous customer and partner sessions.
In the keynote, the company’s CEO and co-founder Jay Krepps shared his vision of building universal data products with Confluent to power both the operational and analytical aspects of data. To that end, he and his colleagues showed several innovations coming to the Confluent ecosystem, including a new capability that makes it easier to run real-time AI workloads.
Kripps said the offering will save developers the complexity of managing different tools and languages when trying to train and evaluate AI models with real-time data. In a conversation with VentureBeat, the company’s CPO Sean Close further explored these offerings and the company’s vision in the age of modern AI.
Sangam Kafka’s story
A decade ago, organizations relied heavily on batch data for analytical workloads. The approach worked, but it meant only understanding and driving value from information to a certain point – not the most recent piece of information.
To fill this gap, a series of open source technologies were developed to power the real-time movement, management and processing of data, including Apache Kafka.
Fast forward to today, Apache Kafka serves as the leading choice for streaming data feeds in thousands of enterprises.
Led by Kreps, one of the original creators of the open platform, Confluent has built commercial products and services (both standalone and fully managed) around it.
However, this is only one piece of the puzzle. Last year, the data streaming player also acquired Immerok, a key partner in the Apache Flink project, to process (filtering, joining and enriching) data streams in-flight for downstream applications.
Now, at Kafka Summit, the company has begun evaluating the AI model in its cloud-native offering for Apache Flink, facilitating one of the most target applications with streaming data: real-time AI. and machine learning.
“Kafka was built to make all these different systems work together in real time and power really amazing experiences,” Clowes explained. “AI has just added fuel to that fire. For example, when you use LLM, it will do makeup and respond if it has to. So, effectively, it’s just about I’ll continue to talk about whether it’s true or not. At this point, you call AI and its response is almost always based on the accuracy and timeliness of the data. It’s always been true in traditional machine learning and that Very accurate in modern ML.
Previously, to call AI with streaming data, teams using Flink had to use code and several tools to plumb models and data processing pipelines. With AI model approximation, Confluent is making it “very pluggable and composable,” allowing them to use simple SQL statements from within the platform to call AI engines, including OpenAI, AWS SageMaker , GCP Vertex, and Microsoft Azure.
“You may already be using Flink to build a RAG stack, but you have to do it using code. You have to write SQL statements, but then you call a model. A user-defined function would have to be used, and the embeddings would have to be returned or inferred. On the other hand, it just makes it super-pluggable. So, without changing any code, You can call any embedding or generation model,” the CPO said.
Flexibility and strength
A plug-and-play approach has been chosen by the company as it wants to give customers the flexibility to go with the option they want depending on their use case. Not to mention, the performance of these models also changes over time, with no model being a “winner or loser.” This means the user can start with model A without changing the initial data pipeline and then move to model B if it improves.
“In this case, really, you basically have two Flink jobs. One Flink job is listening for data about the customer data and it generates the embedding from the model document fragment and stores it in the vector database. Now, you have a vector database with up-to-date context information. Then, on the other hand, you have an inference request, like a user asking a question “Takes the query from the job and associates it with the retrieved documents using embeddings. And that’s it. You call the selected LLM and push the data in the response,” Clowes noted.
Currently, the company provides access to AI model estimation for select users building real-time AI apps with Flink. It plans to launch more features in the coming months to expand access and make AI apps easier, cheaper and faster to run with streaming data. Part of the effort will also include improvements to the cloud-native offering, including a general AI assistant to help users with coding and other tasks in their respective workflows, Close said.
“With an AI assistant, you can say, ‘Tell me where this subject is coming from, tell me where it’s going or tell me what the infrastructure looks like,’ and it will answer all the questions, perform the tasks. It will help our customers build really good infrastructure,” he said.
A new way to save money
In addition to ways to simplify AI efforts with real-time data, Confluent also talked about freight clusters, a new serverless cluster type for its customers.
Clowes explained that these auto-scaling freight clusters take advantage of cheap but slow replication in data centers. This results in some delays, but reduces costs by up to 90%. This approach works in many use cases, such as when processing logging/telemetry data feeding into indexing or batch aggregation engines, he said.
“With the Kafka standard, you can go as low as Electron. Some customers go as low as 10-20 milliseconds. However, when we talk about freight clusters, we’re looking at one to two seconds. Looking at latency. It’s still pretty fast and can be a cheap way to digest data,” noted the CPO.
As the next step in that work, Clowes and Kreps both indicated that Confluent wants to “recognize itself” to expand its presence in the APAC region. In India alone, which already hosts the company’s second-largest workforce outside the US, it plans to increase headcount by 25 percent.
On the product side, Clowes emphasized that they are exploring and investing in capabilities to improve data governance, primarily moving left governance, as well as cataloging data to drive self-service data. are He said these elements are much more immature in the world of streaming than in the world of data lakes.
“Over time, we would expect the entire ecosystem to invest more in governance and data products in the streaming domain. I’m very confident that’s going to happen. We’ve made great strides in connectivity and streaming as an industry. , and even stream processing is our competition from the governance side,” he said.
Credit : venturebeat.com