What is the project about?

Quickwit is a cloud-native search engine designed for observability use cases, specifically for logs and traces (with metrics support planned). It's positioned as an open-source alternative to commercial solutions like Datadog, Elasticsearch, Loki, and Tempo.

What problem does it solve?

Quickwit addresses the challenges of efficiently searching and analyzing large volumes of observability data (logs, traces) stored in cloud object storage (e.g., AWS S3, Azure Blob Storage, Google Cloud Storage). It aims to provide a cost-effective and performant solution compared to traditional search engines or other observability platforms. It solves the problem of expensive and slow search on large datasets.

What are the features of the project?

• Full-text search and aggregation queries.
• Elasticsearch-compatible API.
• Jaeger-native integration for distributed tracing.
• OTEL-native support for logs and traces.
• Schemaless or strict schema indexing.
• Sub-second search on cloud storage.
• Decoupled compute and storage.
• Grafana data source plugin.
• Kubernetes deployment support (Helm chart).
• RESTful API.
• Enterprise features: multiple data sources (Kafka, Kinesis, Pulsar), multi-tenancy, retention policies, delete tasks (GDPR), distributed and highly available architecture.

What are the technologies used in the project?

• Rust (implied, as it's a high-performance systems language common for search engines).
• Cloud object storage (AWS S3, Azure Blob Storage, Google Cloud Storage).
• Kafka, Kinesis, Pulsar (for data ingestion).
• Kubernetes (for deployment).
• Grafana, Jaeger (for visualization and tracing).
• OTEL (OpenTelemetry)
• Elasticsearch/Opensearch API

What are the benefits of the project?

• Cost-effectiveness: Designed to be significantly cheaper than solutions like Elastic, especially when dealing with large datasets on cloud storage.
• Performance: Optimized for fast search and retrieval on cloud storage.
• Scalability: Decoupled architecture allows for independent scaling of compute and storage.
• Open-source: Provides transparency and community-driven development (AGPLv3 license).
• Flexibility: Supports both schemaless and strict schema indexing.
• Compatibility: Works with existing tools and workflows through Elasticsearch API compatibility and integrations.

What are the use cases of the project?

• Log management: Storing, searching, and analyzing application and infrastructure logs.
• Distributed tracing: Monitoring and troubleshooting distributed systems by tracing requests across services.
• Observability: Providing a unified platform for logs and traces (and eventually metrics) to gain insights into system behavior.
• Security Information and Event Management (SIEM) Potentially, given its search capabilities, although not explicitly stated.
• GDPR Compliance.