Confluent: Apache Kafka Use Accelerates

52% of organizations have at least 6 systems running Kafka.

Some 86% of organization are increasing their use of Apache Kafka, according to a“Streaming Data” report from Confluent.

Kafka acts as a messaging broker in real-time and big data systems, often determining where data goes for analysis.

Some 20% of respondents to the Confluent report, which surveyed professionals in a wide range of industries in 47 countries, said that their use of Kafka is increasing sharply.

Ready to Complete the Data Pipeline with Streaming Applications? [Visit Nstream.io]

Respondents to the survey held positions of expertise in areas ranging from developers to tech management and engineers. One in four work for organizations that have more than $1 billion in sales.

Some 52% of organizations said they have at least six systems running Kafka while 21% have over 20. That is in sharp contrast to last year’s report where only 41% of organizations said they had at least six systems running Kafka and only 10% had more than 20.

Kafka helps realize a unified streaming platform. It allows a centralized team to manage massive amounts of data per day and integrates with legacy technology including data warehouses, Hadoop, ETL and messaging middleware. Over 15% of respondents to the Confluent survey are processing more than a billion messages a day.

Kafka’s use in the cloud is also growing. The report found that 34% of organizations use it in virtual private clouds, 52% in public clouds, and 57% on premises. Roughly 32% said they have at least six Kafka applications in the cloud.

Kafka is also creating new business opportunities. Because it makes data available in real-time, companies can create new products or transform existing ones. Roughly 54% of organizations said Kafka enables their business to make faster and more accurate decisions; 47% said that Kafka has allowed them to reduce operating costs; and 40% said it helped them provide a better customer experience.

Organizations are using Kafka in a variety of ways:

The most common use is data pipelines, with 81% say they are using it for that purpose.
66% use Kafka for stream processing
60% use it for data integration.
50% say they are using it for microservices.
75% said they have applications that process data from websites, analytics and sensors connected to their Kafka systems.
Asynchronous apps make up 57% of Kafka use while data warehouse apps are a close second with 51%.
Other uses for Kafka include application monitoring at 41%, system monitoring at 30% and recommendation engines at 30%.

The Kafka API is also enjoying increased popularity, with 37% said they are using it, compared with just 12% last year. Approximately 59% of respondents have databases connected to their Kafka clusters but only 36% are using the Kafka Connect API with Hadoop/HDFS,

Kafka Streams API is the new kid on the block but has already been enthusiastically adopted, with 89% of organizations saying they are familiar with it.

40% are using Kafka Streams for ETL, 32% for core business apps, and 25% for asynchronous applications. Of organizations that have microservices, 28% are using the Kafka Streams API to manage them. Developers are the most common users at 85% but architects (48%) and application teams (43%) also use them.

Despite Kafka’s popularity and the high salaries people with Kafka skills command, there’s a shortage of skilled Kafka engineers. Approximately 75% of organizations said they are having trouble finding skilled Kafka engineers. Despite that, 70% of organizations reported that they are completely satisfied with their Kafka systems.