Apache Kafka/Pixy+Kontrol pod
Overview
This project is a Docker image packaging Apache Kafka 0.10.2.1 with Kontrol. It is meant to be included in a Kubernetes pod.
The container will just run the JVM broker and does not include a control tier: kontrol runs in slave mode which means you need to point to one or more control tiers in the YAML manifest.
Lifecycle
Turning the JVM on/off is done via a regular supervisor job. If a broker fails (e.g is not serving requests anymore) the automaton will attempt to re-start it.
The initial state will render the various configuration files including the telegraf one. The broker is configured to use dynamic id assignment (e.g from the data volume or zookeeper). The broker is started immediately.
Building the image
Pick a distro and build from the top-level directory. For instance:
$ docker build -f alpine-3.5/Dockerfile .
Manifest
It is important to run this pod as a stateful set in order to keep volumes properly assigned. Note that anti-affinity should be used to spread the pods and land the brokers on separate machines. For instance:
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: kafka
namespace: test
spec:
serviceName: kafka
replicas: 3
template:
metadata:
labels:
app: kafka
role: broker
tier: data
annotations:
kontrol.unity3d.com/opentsdb: kairosdb.us-east-1.applifier.info
kafka.unity3d.com/overrides: |
default.replication.factor=3
num.partitions=32
retention.ms=259200000
spec:
nodeSelector:
unity3d.com/array: data
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- kafka
topologyKey: "kubernetes.io/hostname"
containers:
- image: registry2.applifier.info:5005/ads-infra-kafka-alpine-3.5
name: kafka
imagePullPolicy: Always
volumeMounts:
- name: data
mountPath: "/data"
ports:
- containerPort: 8000
protocol: TCP
- containerPort: 9092
protocol: TCP
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeClaimTemplates:
- metadata:
name: data
namespace: test
annotations:
volume.beta.kubernetes.io/storage-class: ebs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Gi
Support
Contact olivierp@unity3d.com for more information about this project.