Steps to read Kafka topic messages from Test-consumer in Karate Framework

Hello Folks !!! I am back with my 2nd blog on how to read Kafka topic from test-consumer in Karate API Automation framework.

On reading blog title, what is the first thing that comes in your mind ?? Hmm.. Let me guess !!

What is Apache Kafka ?

Apache Kafka was originally developed by LinkedIn, and was subsequently open sourced in early 2011. This software platform is currently developed and owned by Apache Software Foundation and it is written in Scala and Java.

Apache Kafka is a distributed streaming platform for building real-time data pipelines and real-time streaming applications.

In simple terms, is a message bus optimized for high-access data replays and streams i.e. Like a publish-subscribe system that can deliver in-order, persistent messages in a scalable way.

The robust message broker allows applications to continually process and re-process stream data.

This open-source platform uses an uncomplicated and easy routing approach that engages a routing key in sending messages to a topic.

How does Apache Kafka work?

Kafka allows you to send messages between applications in distributed systems. The sender can send messages to Kafka, while the recipient gets messages from the stream published by Kafka.

Messages are grouped into topics — a primary Kafka’s abstraction. The sender (producer) sends messages on a specific topic. The recipient (consumer) receives all messages on a specific topic from many senders. Any message from a given topic sent by any sender will go to every recipient who is listening to that topic.

Message flow from producer to consumer via topic

By now you must have acquired basic idea of Kafka and its working mechanism.

How can we create a test-consumer for reading Kafka topic messages in Karate API Automation Framework ?

To achieve this, we have to follow below steps:

Step 1 : Create a Kafka Consumer java class with the default properties. A consumer starts listening to the topic as soon as it is created. This class will consist of all necessary methods to read and process available Kafka records in the Queue, close consumer after use etc.

Step 2 : As test-consumer will be consuming message stream from producer topic, we have to create a JSON Deserializer java class on Kafka consumer side which will deserialize message obtained from Kafka topic. (The Deserialization is a process of converting the serialized object back into a copy of an object)

Step 3 : Create a feature file that calls Kafka consumer class

We can also define a consumer with customized properties as follows :

Step 4 : Run above feature file using TestRunner class

Thanks for reading !!!

Was this blog useful to you? I’d be glad to hear your feedback, so don’t hold back. If you are interested in Automating your Backend service APIs using Karate framework or just have any questions related to KarateDSL/Selenium/BDD framework, you can follow me on LinkedIn. Happy Automation !!!

QA Automation engineer @ Paytm Insider| Ex-Automation Tester @ Reliance Jio Infocomm Ltd. | Ex-Software Developer @Tata Consultancy Services Ltd.