Kafka Quick Start using Docker

Big Data

Reference : Kafka Quick Start Guide

Pre-Requisites

Docker

1 Create Virtual Machine

Get the env for docker-machine, 2CPU, 2G

1
> docker-machine create --driver virtualbox --virtualbox-cpu-count 2 --virtualbox-memory 2048 bigdata

This command downloads a lightweight Linux distribution (boot2docker) with the Docker daemon installed, and creates and starts a VirtualBox VM with Docker running.

  • docker-machine create 创建一个Docker主机
  • –driver virtualbox flag to indicate which provider (VirtualBox, DigitalOcean, AWS, etc.) the machine should be created on.
  • ‘bigdata’ is an argument to indicate the name of the created machine.

list available machines.

1
> docker-machine ls

Get the IP address of one or more machines.

1
> docker-machine ip bigdata

2 Connect your shell to the machine

  • Show the virtual machine environment variable

    1
    2
    3
    4
    5
    > docker-machine env bigdata
    export DOCKER_TLS_VERIFY="1"
    export DOCKER_HOST="tcp://192.168.99.100:2376"
    export DOCKER_CERT_PATH="/Users/yourhostname/.docker/machine/machines/bigdata"
    export DOCKER_MACHINE_NAME="bigdata"

  • Run this command to configure your shell

    1
    2
    # Each time you want to use this virtual machine, you should run this command first!
    > eval $(docker-machine env bigdata)

3 Pull Images

1
2
> docker pull wurstmeister/zookeeper    
> docker pull wurstmeister/kafka

4 Run Images

  • Start zookeeper container

    1
    2
    3
    4
    5
    6
    7
    > docker run \
            -d \
            -p 2181:2181 \
            -p 2888:2888 \
            -p 3888:3888 \
            --name zookeeper \
            wurstmeister/zookeeper

  • Start kafka container

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    > docker run \
            -d \
            -p 9092:9092 \
            # IP=$(docker-machine ip ${MACHINE_NAME})
            -e KAFKA_ADVERTISED_HOST_NAME="${IP}" \
            -e KAFKA_ADVERTISED_PORT=9092 \
            --name kafka \
            --link zookeeper:zookeeper \
            -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
            wurstmeister/kafka

5 Kafka Test

  • Enter the Kafka container

    1
    2
    # here CONTAINER ID = kafka
    > docker exec -it ${CONTAINER ID} /bin/bash

  • Change to kafka default directory

    1
    > cd opt/kafka_2.xx-x.xx.x.x/

  • Create a Topic

    1
    > bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka

  • Run a Producer

    1
    > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka

  • Run a Consumer(in another terminal!)

    1
    > bin/kafka-console-consumer.sh --zookeeper zookeeper:2181 --topic mykafka --from-beginning

  • TEST
    When you type in some words in producer, it will appear in the consumer terminal. Kafka will replay the messages you have sent as a producer.

All Done! :)

Share