GitHub

Fluent Bit

About

Fluent Bit is a fast, open-source Log, Metrics, and Traces Processor and Forwarder. It's designed for collecting, processing, and forwarding data from various sources to multiple destinations. It is part of the Fluentd ecosystem and a CNCF sub-project.

What problem does it solve?

Fluent Bit addresses the challenge of collecting, processing and routing large volumes of logs, metrics, and traces data in diverse environments, including Linux, Windows, Embedded Linux, MacOS, and BSD systems. It simplifies the process of gathering data from different sources and sending it to various storage and analysis platforms.

Features

  • High Performance: Optimized for low CPU and memory usage.
  • Data Parsing: Supports parsing of unstructured data formats like JSON, Regex, LTSV, and Logfmt.
  • Reliability and Data Integrity: Features like backpressure handling and data buffering (in memory and file system) ensure data is not lost.
  • Networking: Built-in TLS/SSL support and asynchronous I/O.
  • Pluggable Architecture: Highly extensible with over 70 built-in plugins for inputs, filters, and outputs. Custom plugins can be written in C, Lua (filters), or Golang (outputs).
  • Monitoring: Exposes internal metrics via HTTP in JSON and Prometheus formats.
  • Stream Processing: Enables data selection, transformation, and analysis using SQL queries.
  • Portable: Runs on various operating systems, including Linux, MacOS, Windows, and BSD.

Technologies Used

  • C (core language)
  • CMake (build system)
  • Flex & Bison (for parsing)
  • YAML, OpenSSL (libraries)
  • Lua (for filter plugins)
  • Golang (for output plugins)
  • SQL (Stream Processing)

Benefits

  • Efficiency: Low resource consumption makes it suitable for resource-constrained environments.
  • Flexibility: The pluggable architecture and support for various data formats and destinations provide high flexibility.
  • Scalability: Designed to handle high volumes of data.
  • Reliability: Mechanisms to prevent data loss.
  • Observability: Built-in monitoring capabilities.
  • Vendor Neutral: Integrates with a wide range of services and platforms.

Use Cases

  • Log Aggregation: Collecting logs from various applications and systems and forwarding them to centralized logging solutions (e.g., Elasticsearch, Splunk, CloudWatch).
  • Metrics Collection: Gathering system and application metrics for monitoring and performance analysis (e.g., Prometheus, Datadog, InfluxDB).
  • Security Monitoring: Collecting security-related logs and events for analysis and threat detection.
  • Cloud-Native Environments: Collecting data from containerized applications and Kubernetes clusters.
  • IoT and Embedded Systems: Its small footprint makes it suitable for collecting data from resource-constrained devices.
  • Data Transformation: Filtering, enriching, and transforming data before sending it to its destination.
fluent-bit screenshot