Skip to content

Account Service - Spring, Kafka Streams sample application that allows users to sign up and retrieve account details and statistics. This PoC demonstrates how to run multiple Kafka Streams instances (with different consumer groups) inside single Spring Boot application.

License

Notifications You must be signed in to change notification settings

rbiedrawa/spring-kafka-streams-protobuf-testcontainers-demo

Repository files navigation

Spring, Kafka Streams, gRPC with TestContainers - demo application

Account Service - Spring, Kafka Streams sample application that allows users to Sign up and retrieve account details and statistics.

This project demonstrates how to run multiple Kafka Streams instances (with different consumer groups) inside single Spring Boot application.

Table of Content

Patterns, technologies and concepts demonstrated here

  • Testcontainers for Kafka streams integration tests.
  • Multiple KStreams inside single boot application.
  • Kafka protobuf integration.
  • Dynamic property binding in Spring Framework.
  • gRPC for handling commands (CreateAccount).
  • gRPC client via BloomRPC application.
  • Interactive queries (kafka state store integration).
  • Query endpoints for account details and statistics.
  • REST endpoints for printing kstream topology details.
  • Separating unit and integration tests with gradle.
  • Docker, docker compose integration.
  • Custom gradle task for printing docker image name and sample docker-compose file.
  • Faker with scheduling setup for generating dummy data, activated by dummy_data_gen spring profile.
  • Custom lombok configuration.

Architecture

architecture

Getting Started

Prerequisites

  • Java 11
  • Docker
  • Kafka
  • Schema Registry

Installation

Local

  • Start Kafka broker and Schema Registry using Confluent Community or Confluent Platform components.

    • Confluent community:
        docker-compose -f ./docker/local/docker-compose.yml up -d
    • Confluent Platform:
      docker-compose -f ./docker/confluent-platform/docker-compose.yml up -d
  • Check if all containers started, using docker-compose ps command.

    # Sample output for Confluent Platform
        Name                  Command            State                       Ports                     
    ----------------------------------------------------------------------------------------------------
    broker            /etc/confluent/docker/run   Up      0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp
    control-center    /etc/confluent/docker/run   Up      0.0.0.0:9021->9021/tcp                        
    rest-proxy        /etc/confluent/docker/run   Up      0.0.0.0:8082->8082/tcp                        
    schema-registry   /etc/confluent/docker/run   Up      0.0.0.0:8081->8081/tcp                        
    zookeeper         /etc/confluent/docker/run   Up      0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp    
  • Start Account Service

    ./gradlew bootRun

Usage

  • Check Create Account flow

    • Import proto files into BloomRPC client.
    • Choose 6565 as gRPC server port.
    • Create new account using AccountService.Create rpc call.
      # Sample request body
      {
        "email": "[email protected]",
        "countryCode": "GB",
        "type": 2
      }
    • Get account id from response
      # Sample response body
      {
        "id": "519faeb7-ccdf-49dd-95a4-3e2a6ac8ad13",
        "email": "[email protected]",
        "countryCode": "GB",
        "type": "PREMIUM",
        "createdDate": {
        "seconds": "1616860904",
        "nanos": 644947000
        }
      }
    • Test interactive queries using REST endpoints.
      • Get account details from accounts.store state store (use account id retrieved before):
          curl -X GET --location "http://localhost:8080/api/v1/accounts/519faeb7-ccdf-49dd-95a4-3e2a6ac8ad13"
      • Get account statistics for 'GB' countryCode:
        curl -X GET --location "http://localhost:8080/api/v1/statistics/countries/GB/accounts"
  • Analise kafka streams topology:

    • Get all available topologies.
      curl -X GET --location "http://localhost:8080/topologies"
    • Print topology for accounts-per-country kstream.
      curl -X GET --location "http://localhost:8080/topologies/accounts-per-country"
    • Print topology diagram, visit https://zz85.github.io/kafka-streams-viz/ page.

Testing Kafka Streams

  • Unit test (TopologyTestDriver) examples may be found under src/test/.
  • Integration test (Testcontainers) examples may be found under src/integrationTest/.

Deployment

Docker

Docker compose file with instructions are kept in docker subdirectory.

References

License

Distributed under the MIT License. See LICENSE for more information.

About

Account Service - Spring, Kafka Streams sample application that allows users to sign up and retrieve account details and statistics. This PoC demonstrates how to run multiple Kafka Streams instances (with different consumer groups) inside single Spring Boot application.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published