LogBus

A "Swiss Army" knife for handling events with a focus on processing system & application logs. See the docs for more information.

Vision

LogBus operates at Layer 1: Signal Conditioning — the digital layer that cleans, normalizes, and shapes raw event streams before they can be understood. Think of it as DSP for log data: filtering noise, normalizing formats, aggregating samples, applying deterministic transforms.

The conditioned signal it produces is the foundation for Layer 2: Signal Intelligence — pattern detection, anomaly identification, cross-source correlation, meaning extraction. Whether L2 lives inside LogBus or in a downstream system is an open question, but L1 has to be right first.

See docs/vision.md for more.

Hacking

See just help for list of helpful tasks.

Unit Testing

just unit-test will test plugins. Plugins should have their spec files colocated beside them. Plugins should strive to be simple functions, but here are some ways they are not:

• If a plugin needs to manage state, then that can persist in its closure and managed however the plugin sees fit. For example, the write-opensearch output plugin batches events into an array until it is ready to perform a bulk insert.

• Some plugins may require special start & stop handling.

End-to-End Testing

just e2e-test will test more real-world like scenarios and exercise the main engine bits, but limited to simpler inputs & outputs (files).

How Events Flow Between Stages (Rust port)

Each stage in the pipeline is an async task. Stages are connected by bounded MPSC channels (capacity 4096). The engine wires them up based on the inputs: declarations in the YAML config.

[Source] --[4096]--> [fan_out] --[4096]--> [Transform] --[4096]--> [fan_out] --[4096]--> [Sink]

Fan-in (multiple upstream stages → one stage) is handled natively by MPSC: each upstream fan-out task holds a clone of the downstream stage's sender. The downstream's receiver closes only when all upstream senders are dropped — i.e., when every upstream stage has finished.

Fan-out (one stage → multiple downstream stages) is handled by a lightweight fan_out task spawned per stage. It reads from the stage's output channel and clones each event (cheap — events are Arc<Value>) to each downstream's input channel.

Backpressure

Backpressure is automatic and requires no plugin code. Because mpsc::Sender::send().await suspends when the channel buffer is full, a slow stage naturally stalls everything upstream:

1. The sink's input buffer fills up (disk is slow)
2. The upstream fan_out's send().await suspends
3. The transform's output buffer fills up
4. The transform's send().await suspends
5. The transform stops reading its input buffer
6. The source's send().await suspends
7. The source stops reading from I/O (file, stdin, Kafka)

The maximum event accumulation before the source stalls is capacity × number of channels in the chain (default capacity: 4096, tunable via --event-buffer). No events are dropped; the source simply pauses until the sink catches up. Tokio parks suspended tasks with zero CPU overhead.

The UI stats interceptor is the one deliberate exception: it receives events via try_send (non-blocking) so a slow browser connection can never cause backpressure into the data pipeline.

Known Issues, Limitations, Warts

• There should be an example demonstrating how to "commit" events. For example, when consuming from a kafka topic, should only commit offsets once the final pipeline stage has confirmed that the event was received successfully. Not yet sure if there needs to be 1st class support for such a mechanism or if can achieve by wiring up the latter stage's output to the earlier stage's input.

• Would be nice if plugins could communicate the type (and maybe shape as well) of the events they process. Then, a user could be warned if a pipeline configuration contains stages wired up erroneously.

• Some plugins require pipeline-defined functions. For simple functions, inlined in yaml is not cumbersome. For more complicated functions, would be nice to define those in a proper package for a better dev & test story.