Log Aggregation Pipeline with ELK/EFK Stack

## ROLE
You are a logging infrastructure engineer who has built and operated centralized logging systems processing terabytes of log data daily. You have deep expertise in the ELK stack (Elasticsearch, Logstash, Kibana) and EFK stack (Elasticsearch, Fluentd, Kibana), as well as modern alternatives like Loki and OpenSearch. You understand the trade-offs between different log pipeline architectures, how to design schemas that enable fast searches, and how to manage the storage costs that come with retaining logs at scale. You have built logging systems that support both operational debugging and compliance audit requirements.

## OBJECTIVE
Design a centralized log aggregation pipeline for [ORGANIZATION NAME] collecting logs from [LOG SOURCES: e.g., Kubernetes pods, EC2 instances, Lambda functions, application services, load balancers, databases]. The pipeline must handle [DAILY LOG VOLUME] of log data per day, support real-time search and analysis, comply with [RETENTION REQUIREMENTS], and remain cost-effective. The system should serve [NUMBER OF USERS] engineers for debugging and [COMPLIANCE TEAMS] for audit purposes.

## TASK

### Section 1: Pipeline Architecture Design
- Design the end-to-end log pipeline: collection → processing → enrichment → storage → visualization
- Select and justify the technology stack:
  - Collector: Fluent Bit (lightweight, Kubernetes-native) vs. Fluentd (plugin-rich) vs. Filebeat vs. Vector
  - Processor: Logstash vs. Fluentd vs. Vector for transformation and enrichment
  - Storage: Elasticsearch vs. OpenSearch vs. Loki — with cost and capability comparison
  - Visualization: Kibana vs. OpenSearch Dashboards vs. Grafana
- Design the deployment topology: collector per node (DaemonSet), aggregator layer, and storage cluster
- Plan for high availability: collector redundancy, buffer management, storage replication
- Implement backpressure handling: what happens when the pipeline is overwhelmed
- Design a dead letter queue for failed log processing

### Section 2: Log Collection & Structured Logging
- Create a structured logging standard for applications:
  - JSON log format with mandatory fields: timestamp, level, service, trace_id, span_id, message
  - Recommended additional fields: user_id, request_id, duration_ms, error_code, environment
  - Language-specific implementation guides for [TECH STACK: e.g., Node.js/Winston, Python/structlog, Go/zap, Java/Logback]
- Configure collector agents for different log sources:
  - Kubernetes: container logs via Fluent Bit DaemonSet, metadata enrichment (pod, namespace, labels)
  - System logs: syslog, journald, audit logs
  - Cloud provider logs: CloudTrail, VPC Flow Logs, ALB access logs
  - Application logs: stdout/stderr, file-based, socket-based
- Implement multiline log handling for stack traces and complex log entries
- Design a log sampling strategy for high-volume, low-value logs (health checks, debug logs)

### Section 3: Log Processing & Enrichment
- Design the processing pipeline:
  - Parsing: extract structured fields from unstructured logs (grok patterns, regex, JSON parsing)
  - Enrichment: add metadata (GeoIP, hostname resolution, service ownership, environment tags)
  - Transformation: field renaming, type casting, timestamp normalization
  - Filtering: drop noisy logs, redact sensitive data (PII, credentials, tokens)
  - Routing: send different log types to different indices/streams based on rules
- Implement PII redaction and data masking rules for compliance
- Create a sensitive data detection pipeline (credit card numbers, SSNs, API keys, passwords)
- Design a log classification system: access logs, application logs, error logs, audit logs, security logs
- Build custom parsing rules for non-standard log formats from third-party services

### Section 4: Storage Architecture & Index Management
- Design the Elasticsearch/OpenSearch cluster:
  - Node roles: master, data, ingest, coordinating, ML (if applicable)
  - Cluster sizing: node count, instance types, storage capacity based on [DAILY LOG VOLUME]
  - Shard strategy: shard count per index, shard size targets (30-50GB optimal)
- Implement Index Lifecycle Management (ILM):
  - Hot phase: active writing and searching (SSD storage)
  - Warm phase: reduced replicas, read-optimized (HDD or lower-cost SSD)
  - Cold phase: frozen indices, searchable snapshots
  - Delete phase: automatic cleanup after retention period
- Design index templates with optimized mappings:
  - Field type selection for search performance (keyword vs. text, date formats, numeric types)
  - Disable indexing for fields that don't need searching
  - Configure doc_values and fielddata based on aggregation needs
- Plan snapshot and backup strategy: automated snapshots to S3/GCS, cross-region backup

### Section 5: Search, Visualization & Alerting
- Design Kibana/OpenSearch Dashboards:
  - Operations dashboard: log volume trends, error rates by service, top error messages
  - Security dashboard: failed authentication attempts, unusual access patterns, audit trail
  - Application debugging dashboard: request tracing, error analysis, performance correlation
  - Infrastructure dashboard: system logs, resource warnings, certificate/expiry alerts
- Create saved searches and index patterns for common debugging workflows
- Implement log-based alerting:
  - Error rate threshold alerts per service
  - Security event detection (brute force, privilege escalation, data exfiltration patterns)
  - Application health indicators from log patterns
  - Compliance alerts (audit log gaps, unauthorized access attempts)
- Design search optimization: query templates, field-level security, and role-based access

### Section 6: Operations, Cost Management & Scaling
- Create operational runbooks for common tasks: cluster scaling, index recovery, node replacement
- Design cost optimization strategies:
  - Log volume reduction: filtering, sampling, compression
  - Storage tiering: hot/warm/cold with automatic transitions
  - Index rollover policies based on size and time
  - Cost per GB analysis and budget alerting
- Plan for scaling: horizontal scaling triggers, capacity planning formulas, and growth projections
- Implement monitoring for the logging infrastructure itself: ingestion rate, search latency, disk usage, cluster health
- Design a self-service onboarding process for new services
- Create a log quality scoring system to ensure teams maintain logging standards

## OUTPUT FORMAT
Provide complete configuration files: Fluent Bit/Fluentd configuration, Elasticsearch index templates and ILM policies, Kibana dashboard exports, and deployment manifests (Kubernetes or Docker Compose). Include a structured logging implementation guide with code examples. Provide an architecture diagram described in text and an operational handbook. Include cost projections based on log volume.

## CONSTRAINTS
- PII and sensitive data must never reach storage unredacted — implement fail-safe redaction
- Log retention must comply with [RETENTION REQUIREMENTS] — both minimum retention and maximum retention limits
- Search latency must remain under 5 seconds for the last 24 hours of data
- Pipeline must handle 3x burst capacity without data loss (use buffering and backpressure)
- All configurations must be deployable via IaC and version-controlled
- Include RBAC: developers see their services' logs, security team sees audit logs, compliance sees everything