Streamline Kubernetes log management with production-ready EFK stack, featuring advanced pattern detection, multi-namespace support, and flexible routing configurations. This stack brings comprehensive log aggregation and processing to modern containerized environments.

Production-ready Kubernetes EFK logging stack for multi-namespace log aggregation and processing. Uses Fluentd DaemonSet to route container logs from /var/log/containers/ to Elasticsearch with configurable indexing and visualization in Kibana.

This project provides comprehensive log collection and processing for Kubernetes environments with specialized support for custom log pattern processing, multi-namespace deployments, and configurable indexing strategies. The stack includes multiple Fluentd configurations optimized for different deployment scenarios.

• Custom Log Processing: Automatic detection and field extraction from configurable log patterns
• Multi-Namespace Support: Configurable log collection across multiple Kubernetes namespaces
• Flexible Indexing: Multiple routing strategies including centralized, per-namespace, and time-series indexing
• Production Ready: Includes RBAC, service accounts, and security configurations

View Architecture Diagram

Alt: Fluentd DaemonSet tails container logs from /var/log/containers/, enriches with Kubernetes metadata, and routes to Elasticsearch indices. Kibana queries indices for visualization.

The EFK stack follows a distributed architecture where Fluentd runs as a DaemonSet on every Kubernetes node, collecting logs from all containers and routing them to Elasticsearch based on configurable patterns.

• Fluentd DaemonSet: Collects container logs from all nodes
• ConfigMaps: Multiple configurations for different deployment scenarios
• RBAC & Service Account: Proper security controls and service account permissions

1. Collection: Fluentd tails container logs from /var/log/containers/ on each node
2. Enrichment: Kubernetes metadata filter adds pod, namespace, and container information
3. Pattern Detection: Identifies and parses custom log patterns with structured field extraction
4. Routing: Logs are routed to appropriate Elasticsearch indices based on configuration
5. Storage: Indexed in Elasticsearch with configurable retention policies

• Native Kubernetes DaemonSet log collection - No sidecar containers needed, efficient resource usage
• Flexible index strategies - Per namespace, centralized, or time-series indexing patterns
• Built-in support for custom log pattern detection - Automatically parse structured logs and alerts
• Multi-namespace support - Collect from specific namespaces or cluster-wide with RBAC controls
• Production-ready configurations - Multiple deployment scenarios with security best practices
• Lightweight compared to ELK - Fluentd uses less memory than Logstash for similar functionality

• Kubernetes cluster (v1.16+)
• kubectl configured
• Elasticsearch cluster or instance

1. Deploy Elasticsearch configuration:

   Important: First, edit fluentd-solution/configs/elasticsearch-config-and-secret.yaml to update the Elasticsearch host IP and credentials:

   # Replace with your actual Elasticsearch host IP or hostname
   elasticsearch-host: "192.168.1.100" # Change this!
   # Update credentials if needed
   password: Y2hhbmdlbWU= # base64 encoded, change if needed

   Then deploy:

   kubectl apply -f fluentd-solution/configs/elasticsearch-config-and-secret.yaml

2. Choose and deploy a Fluentd configuration:

   Note: Review the Configuration Options section below to choose the right config for your use case.

   # For single namespace deployment
   kubectl apply -f fluentd-solution/configs/fluentd-single-namespace-alert-config.yaml
   
   # For cluster-wide deployment (excludes system namespaces)
   kubectl apply -f fluentd-solution/configs/fluentd-all-except-system-config.yaml
   
   # For multi-namespace with centralized alerts
   kubectl apply -f fluentd-solution/configs/fluentd-multi-namespace-centralized-alert-config.yaml

3. Configure and deploy the Fluentd DaemonSet:

   Important: Edit the DaemonSet configuration to reference your chosen ConfigMap:

   # Edit fluentd-solution/configs/fluentd-daemonset.yaml
   # Update line 121 to match your chosen ConfigMap name:
   # configMap:
   # name: fluentd-single-namespace-alert-config # Replace with your choice

   Then deploy the DaemonSet:

   kubectl apply -f fluentd-solution/configs/fluentd-daemonset.yaml

4. Verify the deployment:

   # Check if pods are running
   kubectl get pods -l k8s-app=fluentd-logging -n logging
   
   # Check logs for any issues
   kubectl logs -l k8s-app=fluentd-logging -n logging

Configure these environment variables in the Elasticsearch ConfigMap:

Set the password in the Elasticsearch Secret:

The system can be configured to automatically detect and parse custom log patterns with structured field extraction. You can define any pattern string relevant to your application.

Configure the system to detect and parse logs with patterns like "ALERT":

Extracted fields:

• User: Username who triggered the alert
• Action: Action that was performed
• Status: Operation status
• Field: Field that was modified
• Original: Previous value
• Updated: New value

You can customize the pattern detection by modifying the Fluentd configuration to handle any log format specific to your application needs.

• Single namespace: alerts_logging-test
• Per-namespace: alerts_{namespace}
• Centralized: centralized-alert-logs
• Cluster-wide: alerts_cluster
• Time-series: {service}_{app}-alert-YYYY-MM

• Namespace-app format: {namespace}_{app}
• Service-app format: {service}_{app}
• Dynamic: {target_index}
• Generic fallback: multi-namespace-generic

1. Pod not starting: Check ConfigMap name in DaemonSet matches deployed ConfigMap
2. No logs in Elasticsearch: Verify Elasticsearch connection and credentials
3. ALERT parsing not working: Ensure log format matches expected pattern
4. Missing indices: Check namespace filtering and log routing configuration

1. Create new ConfigMap in fluentd-solution/configs/
2. Follow naming convention: fluentd-{purpose}-config.yaml
3. Update DaemonSet to reference new ConfigMap
4. Validate configuration functionality
5. Test configuration and validate functionality

• Credentials: Store Elasticsearch credentials in Kubernetes Secrets
• RBAC: Use provided service accounts and cluster roles
• Network Security: Configure appropriate network policies
• SSL/TLS: Enable SSL verification for production Elasticsearch clusters
• Access Control: Implement appropriate Elasticsearch access controls

• Helm Chart: Complete Helm chart for easy deployment and management
• Advanced Monitoring: Prometheus metrics integration
• Log Retention Policies: Automated index lifecycle management
• Multi-cluster Support: Cross-cluster log aggregation

We welcome contributions! Please see our Contributing Guidelines for details on how to submit pull requests, report issues, and contribute to the project.

1. Fork the repository
2. Create a feature branch
3. Validate configuration changes
4. Update documentation
5. Submit a pull request

This project is licensed under the MIT License - see the LICENSE file for details.