Redpanda Edge Agent
A lightweight Internet of Things (IoT) agent that runs alongside Redpanda at the edge to forward events to a central Kafka API compatible cluster. The agent is written in Go and uses the franz-go Kafka client library.
Build the agent
Build the agent for any Go-supported target platform by setting the GOOS and GOARCH variables. For a full list of supported architectures, run: go tool dist list.
go clean
# MacOS (Intel)
env GOOS=darwin GOARCH=amd64 go build -a -v -o redpanda-edge-agent ./agent
# MacOS (M1)
env GOOS=darwin GOARCH=arm64 go build -a -v -o redpanda-edge-agent ./agent
# Linux (x86_64)
env GOOS=linux GOARCH=amd64 go build -a -v -o redpanda-edge-agent ./agent
# Linux (Arm)
env GOOS=linux GOARCH=arm64 go build -a -v -o redpanda-edge-agent ./agent
# MacOS (Intel)
env GOOS=darwin GOARCH=amd64 go build -a -v -o redpanda-edge-agent ./agent
# MacOS (M1)
env GOOS=darwin GOARCH=arm64 go build -a -v -o redpanda-edge-agent ./agent
# Linux (x86_64)
env GOOS=linux GOARCH=amd64 go build -a -v -o redpanda-edge-agent ./agent
# Linux (Arm)
env GOOS=linux GOARCH=arm64 go build -a -v -o redpanda-edge-agent ./agent
The build.sh script builds the agent and adds the resulting executable to a tarball for the platforms listed above.
Running the agent
Usage of ./redpanda-edge-agent:
-config string
path to agent config file (default "agent.yaml")
-loglevel string
logging level (default "info")
-config string
path to agent config file (default "agent.yaml")
-loglevel string
logging level (default "info")
Configuration
Example agent.yaml:
# The unique identifier for the agent. If not specified the id defaults to the
# hostname reported by the kernel. When forwarding a record, if the record's
# key is empty, then it is set to the agent id to ensure that all records sent
# by the same agent are routed to the same destination topic partition.
id:
# If the provided topics do not exist, attempt to create them on the source and
# destination clusters.
create_topics: true
# Source cluster configuration. The agent will typically run as a sidecar to
# the source cluster in an edge environment.
source:
# Name of the source cluster
name: "source"
# List of Redpanda nodes (typically a single node at the edge).
bootstrap_servers: 127.0.0.1:19092
# List of topics to forward to the destination cluster.
# - Specify a single topic name to forward events to a topic with the same
# name on the destination cluster (e.g. "telemetryA").
# - Specify a pair of topic names, separated by a colon, to forward events
# to a topic with a different name on the destination cluster
# (e.g. "telemetryB:telemetryC").
topics:
- telemetryA
- telemetryB:telemetryC
# Set the consumer group for the agent to join and consume in. Defaults to
# the agent id if not set.
consumer_group_id: ""
# The source cluster TLS configuration.
tls:
enabled: true
client_key: "agent.key"
client_cert: "agent.crt"
ca_cert: "ca.crt"
# The source cluster SASL configuration.
sasl:
# Valid SASL methods: PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512
sasl_method: ""
sasl_username: ""
sasl_password: ""
# Destination cluster configuration. This is typically a centralized Redpanda
# cluster that aggregates the data produced by all agents.
destination:
# Name of the destination cluster
name: "destination"
# List of Redpanda nodes
bootstrap_servers: 127.0.0.1:29092
# Maximum Kafka protocol version. E.g. "3.0.0"
max_version: ""
# List of topics to pull from the destination cluster (to create a
# bidirectional flow).
# - Specify a single topic name to pull events to a topic with the same
# name on the source cluster (e.g. "configA").
# - Specify a pair of topic names, separated by a colon, to pull events
# to a topic with a different name on the source cluster
# (e.g. "configB:configC").
topics:
- configA
- configB:configC
# Set the consumer group for the agent to join and consume in. Defaults to
# the agent id if not set.
consumer_group_id: ""
# The destination cluster TLS configuration.
tls:
enabled: true
client_key: "agent.key"
client_cert: "agent.crt"
ca_cert: "ca.crt"
# The destination cluster SASL configuration.
sasl:
# Valid SASL methods: PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512
sasl_method: ""
sasl_username: ""
sasl_password: ""
# hostname reported by the kernel. When forwarding a record, if the record's
# key is empty, then it is set to the agent id to ensure that all records sent
# by the same agent are routed to the same destination topic partition.
id:
# If the provided topics do not exist, attempt to create them on the source and
# destination clusters.
create_topics: true
# Source cluster configuration. The agent will typically run as a sidecar to
# the source cluster in an edge environment.
source:
# Name of the source cluster
name: "source"
# List of Redpanda nodes (typically a single node at the edge).
bootstrap_servers: 127.0.0.1:19092
# List of topics to forward to the destination cluster.
# - Specify a single topic name to forward events to a topic with the same
# name on the destination cluster (e.g. "telemetryA").
# - Specify a pair of topic names, separated by a colon, to forward events
# to a topic with a different name on the destination cluster
# (e.g. "telemetryB:telemetryC").
topics:
- telemetryA
- telemetryB:telemetryC
# Set the consumer group for the agent to join and consume in. Defaults to
# the agent id if not set.
consumer_group_id: ""
# The source cluster TLS configuration.
tls:
enabled: true
client_key: "agent.key"
client_cert: "agent.crt"
ca_cert: "ca.crt"
# The source cluster SASL configuration.
sasl:
# Valid SASL methods: PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512
sasl_method: ""
sasl_username: ""
sasl_password: ""
# Destination cluster configuration. This is typically a centralized Redpanda
# cluster that aggregates the data produced by all agents.
destination:
# Name of the destination cluster
name: "destination"
# List of Redpanda nodes
bootstrap_servers: 127.0.0.1:29092
# Maximum Kafka protocol version. E.g. "3.0.0"
max_version: ""
# List of topics to pull from the destination cluster (to create a
# bidirectional flow).
# - Specify a single topic name to pull events to a topic with the same
# name on the source cluster (e.g. "configA").
# - Specify a pair of topic names, separated by a colon, to pull events
# to a topic with a different name on the source cluster
# (e.g. "configB:configC").
topics:
- configA
- configB:configC
# Set the consumer group for the agent to join and consume in. Defaults to
# the agent id if not set.
consumer_group_id: ""
# The destination cluster TLS configuration.
tls:
enabled: true
client_key: "agent.key"
client_cert: "agent.crt"
ca_cert: "ca.crt"
# The destination cluster SASL configuration.
sasl:
# Valid SASL methods: PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512
sasl_method: ""
sasl_username: ""
sasl_password: ""