Apache Kafka for Developers #14: Order Processing with Kafka, Node.js Microservices, and MySQL

Building simple E-commerce Order Processing System with Kafka, Nodejs Microservices, and MySQL Databases.

Architecture Overview:

The above architecture consists of two main Node.js microservices

Order Microservice:

The Order Microservice is responsible for accepting new order requests, storing the details in the Order Database, and publishing the order information to the Kafka Order Topic.

Implementing Idempotency Checks

Idempotency checks are crucial to prevent the duplicate processing of the same request. The Order Microservice ensures that each request header contains a unique identifier called the idempotency-key. This key is stored in the Order Database along with the order details. Before processing a request, the service checks the database to determine if the request is a duplicate, ensuring that each order is processed only once.

Implementing Kafka Retry

The Order Microservice includes Kafka retry logic to handle transient errors. By using try-catch blocks to capture exceptions, the service can reprocess messages up to a maximum number of retries with a delay between each attempt. This ensures that temporary issues do not prevent order processing service.

Subscribing to Kafka Payment Topic

The Order Microservice subscribes to the Kafka Payment Topic to receive payment status updates. This allows the Order Microservice to update the order status in the Order Database based on the payment status received from the Payment Microservice.

Payment Microservice

The Payment Microservice listens to the Kafka Order Topic for new orders, processes payments, and stores payment statuses in a database, and publishing the payment information to the Kafka Payment Topic.

Implementing Idempotency Checks

To ensure idempotency, the Payment Microservice checks if a payment record with the same orderId already exists in the Payment Database before processing the payment. This prevents duplicate processing of the same order.

Publishing Payment Status to Kafka Payment Topic

After processing the payment, the Payment Microservice publishes the payment status to the Kafka Payment Topic. This allows the Order Microservice to receive updates on the payment status and update the order status accordingly.

Step-by-Step Implementation

Prerequisites

Install Docker Desktop using https://docs.docker.com/desktop/

1. Clone the repository

https://github.com/ramasubbareddy1224/kafka-nodejs-microservice-order-processing

> git clone https://github.com/ramasubbareddy1224/kafka-nodejs-microservice-order-processing.git

// change directory
> cd kafka-nodejs-microservice-order-processing

2. Start all the Docker containers by running the following command.

> docker-compose up -d --build

3. Use Kafka UI to view the list of Kafka topics.

http://localhost:8080/ui/clusters/d8zv92ecQk6ZrAAA35SDbw/all-topics

4. Submit a new order request using either curl or Postman

curl --location 'http://localhost:5520/v1/order' \
--header 'idempotency-key: 223' \
--header 'Content-Type: application/json' \
--data '{
    "customer_name":"test user",
    "product_id":"1",
    "quantity":"2",
    "total_amount":150
}'