PuffinDB embeds a powerful engine for the execution of incremental and observable data pipelines:
- Execution of multi-step data transformation pipelines orchestred by Redis
- Sequential steps executed on multiple table partitions in parallel across 10,000 serverless functions or more
- Multi-threaded execution of individual steps
- Blocking and non-blocking step execution
- Direct function-to-function communication through NAT hole punching
- Pipelines defined using JSON or YAML syntax including Python and TypeScript scripting
- Steps defined with SQL or PRQL extended with user-defined functions powered by Python, TypeScript, or WebAssembly
- Steps optionally defined with 20+ perfectly map-reducible transforms for super-high performance
- Support for steps invoking curl commands or any Airbyte connector
- Integration with lakehouses like Iceberg, Delta Lake, and Hudi
- Incremental pipeline execution
- Real-time observability