Broadway
Concurrent and multi-stage data ingestion and processing with Elixir
Broadway is an open-source Elixir library for building concurrent and multi-stage data ingestion and processing pipelines. It leverages the Erlang VM's supervision trees for robust failure handling and graceful shutdowns, with built-in integrations for popular message queues.
3K
GitHub Stars
medium
Learning Curve
4.0
DX Score
Preços
Model
free
Plano Gratuito
Completely free and open-source.
Recursos
- ✓ Concurrent data ingestion with batching
- ✓ RabbitMQ integration
- ✓ Amazon SQS integration
- ✓ Google PubSub integration
- ✓ Apache Kafka integration
- ✓ Back pressure via GenStage
- ✓ Built-in rate limiting
- ✓ Broadway Dashboard for monitoring
- ✓ Graceful shutdown handling
Prós
- + Robust failure handling through Erlang VM
- + Production-proven by Discord, Change.org
- + Extensive documentation
- + Built-in monitoring dashboard
- + Prevents system overload with back pressure
Contras
- - Limited to Elixir ecosystem
- - Requires understanding of GenStage concepts
- - Smaller community than Kafka Streams/Flink
- - Learning curve for non-Elixir developers
Melhor Para
indie hacker startup
elixir erlang beam data-pipeline streaming queue kafka sqs rabbitmq