How to start Spring Apache Kafka Application with SpringBoot Auto-Configuration

The Spring Apache Kafka (spring-kafka) provides a high-level abstraction for Kafka-based messaging solutions. And Spring Boot 1.5 includes auto-configuration support for Apache Kafka via the spring-kafka project. So in the tutorial, JavaSampleApproach will guide how to start Spring Apache Kafka Application with SpringBoot.

Related Articles:
How to start Apache Kafka
How to start Spring Kafka Application with Spring Boot
Spring JMS – Explicitly Configure Spring ActiveMQ ConnectionFactory | SpringBoot

I. Technologies

– Java 8
– Maven build
– Spring Boot
– Apache Kafka
– Spring Tool Suite editor

II. Overview

As above mention, SpringBoot supports auto-configuration for Apache Kafka development:
– Use spring.kafka.* in application.properties file to modify external configuration.

Spring Apache Kafka Application

– Use Spring auto-configured KafkaTemplate to send Kafka-based messages:

– Use @KafkaListener to setup a Kafka listener:

III. Practice

We create a SpringBoot project with 2 main services: KafkaProducer and KafkaConsumer for sending and receiving messages from Apache Kafka cluster.
And export 2 RestAPIs {‘/producer’, ‘/consumer’} for interaction.

SpringBoot Kafka Application - project structure

Step to do:
– Create a SpringBoot project
– Create Kafka Producer and Consumer
– Add Apache Kafka external configuration
– Export some RestAPIs
– Deployment

1. Create a SpringBoot project

Use SpringToolSuite to create a SpringBoot project, then add dependencies {spring-kafka, spring-boot-starter-web}:

2. Create Kafka Producer and Consumer

– Create a KafkaProducer service:

– Create a KafkaConsumer service:

About MessageStorage, it is an additional implement to store Kafka-based messages after received. See details the implementation of MessageStorage:

3. Add Apache Kafka external configuration

Open application.properties file, add the configurations:

spring.kafka.bootstrap-servers is used to indicate the Kafka Cluster address.
spring.kafka.consumer.group-id is used to indicate the consumer-group-id.
jsa.kafka.topic is an additional configuration. In the the tutorial, we use jsa.kafka.topic to define a Kafka topic name to produce and receive messages.

4. Export some RestAPIs

Create a Web Controller to export 2 RestAPIs {‘/producer’, ‘/consumer’}

/producer is used to send messages from browser to KafkaProducer service.
/consumer is used to get all recieved messages that are buffered in MessageStorage.

5. Deployment

Start Apache Kafka Cluster:
– Start a ZooKeeper:

– Start the Apache Kafka server:

>>> More details at: How to start Apache Kafka

Build and Install the SpringBoot project with commandlines: mvn clean install and mvn spring-boot:run

– Make a producer request: http://localhost:8080/jsa/kafka/producer?data=Hello World
-> Logs:

– Make another producer request: http://localhost:8080/jsa/kafka/producer?data=This is a SpringBoot Kafka Application

-> Logs:

– Make a consumer request: http://localhost:8080/jsa/kafka/consumer, result:

Spring Appache Kafka - consumer request

IV. Sourcecode

SpringBootApacheKafka

By grokonez | June 8, 2017.



Related Posts


5 thoughts on “How to start Spring Apache Kafka Application with SpringBoot Auto-Configuration”

  1. Hi, i am getting error,
    Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.5.1.RELEASE:run (default-cli) on project Messaging: Unable to find a suitable main class, please add a ‘mainClass’ property -> [Help 1]

  2. spring -boot version is old here in the pom..also the jars need to be updated….does anyone have an updated pom on this project that works??
    I updated the spring boot version to
    <!– 1.5.4.RELEASE –>
    2.0.2.RELEASE

    compiles alirght but gave test failure
    [INFO]
    [INFO] ——————————————————-
    [INFO] T E S T S
    [INFO] ——————————————————-
    [INFO] Running com.javasampleapproach.apachekafka.SpringApacheKafkaApplicationTests
    [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.042 s <<< FAILURE! – in com.javasampleapproach.apachekafka.SpringApacheKafkaApplicationTests
    [ERROR] initializationError(com.javasampleapproach.apachekafka.SpringApacheKafkaApplicationTests) Time elapsed: 0.014 s <<< ERROR!
    java.lang.TypeNotPresentException: Type org.springframework.test.context.junit4.SpringRunner not present
    Caused by: java.lang.ClassNotFoundException: org.springframework.test.context.junit4.SpringRunner

  3. How would it scale ?, it would run out of memory in about 15 mins ? Reading a message queue into an object and then reading the object isn’t going to be in any fault tolerant.

    You would be better with Kafka Streams to achieve this, or at least cache them in a h2 database so its a common resource for all users.

Got Something To Say:

Your email address will not be published. Required fields are marked *

*