Rust eKuiper rewrite holds 5-10 MB under windowed MQTT load where Go engines near 1 GB

What the project is

According to a dev.to post from I-Dacs Labs Engineering, the team built rekuiper, a stream-processing engine written in Rust that reimplements the interface of LF Edge eKuiper rather than its internals. The compatibility surface is deliberately boring: eKuiper's REST API (98 paths and 140 operations, verified black-box against eKuiper's OpenAPI description), its SQL dialect including JSON paths, CASE, array indexing and unnest, its stream option names such as DATASOURCE, FORMAT, CONF_KEY, SCHEMAID and TIMESTAMP, and the kuiper command-line tool. The intent is that existing eKuiper rules, the eKuiper Manager web UI and deployment tooling keep working, so switching engines does not mean rewriting everything around them.

How the benchmark was run

Every engine ran in a container pinned to a single CPU core with 1 GB of memory and no swap, while a separate Mosquitto broker received its own cores and generous queue limits so it would never become the bottleneck. An open-loop Rust load generator called mqttgen, using MQTT 3.1.1 at QoS 0, fed each engine from an identical schedule, and a step only counted when the generator actually stayed on that schedule. Output was checked message by message, because the authors argue that measuring ingest acknowledgements or counting output records can hide loss and duplication — an engine that silently drops a large share of data can still look fast.

Four engines were compared: rekuiper v0.425-beta, eKuiper 2.4.1, Telegraf 1.40.0 and Redpanda Connect 4.109.0, the project formerly known as Benthos. Apache Flink was deliberately left out: according to the post, neither Flink 2.x nor Apache Bahir ships an MQTT connector, so testing Flink would have required a custom source or a Kafka bridge and changed the very ingest path under measurement.

Five MQTT workloads were shaped like real deployments: a stateless telemetry filter across 1,000 devices; per-device 10-second tumbling windows with count, avg and max; 10,000 plain-text ESPHome topics consumed via a wildcard subscription, with a binary format option and the topic carried through as metadata; 10,000 per-VIN vehicle topics with tumbling windows, described as the hardest memory test; and 2,000 EV charger topics using session windows, a construct the authors say neither Telegraf nor Redpanda Connect can express at all.

The results

rekuiper produced complete, correct output at 100,000 messages per second on one core in every workload — the top of the tested range, so its ceiling was never reached. The number the authors care about more is memory: on the windowed workloads, rekuiper stayed between 5 and 10 MB while the Go-based engines climbed to between half a gigabyte and a full gigabyte, or failed. They attribute the gap to design rather than to Rust itself. The exercise also exposed a correctness bug in rekuiper that a throughput-only benchmark would have rewarded as speed.

Three design choices behind the flat memory

The internal stream bus uses bounded per-subscriber queues of 4,096 records with reserve-then-commit batch admission, so a batch is either delivered to every subscriber or rejected outright, and a slow rule pushes back pressure onto its source instead of quietly dropping data. Sinks drain through bounded queues of 10,000 records served by dedicated workers. The MQTT source, built on the rumqttc client, admits everything surfaced by a single network read as one batch of up to 1,024 records, which the post says cuts CPU per message to roughly half that of the Go engines on simple workloads.

Windowed GROUP BY aggregation is incremental: one accumulator is kept per group per aggregate and the rows themselves are never stored, making window memory a function of the number of devices rather than the number of messages. Statements that genuinely need the rows, such as collect(), joins and some HAVING clauses, fall back to a buffered evaluator, and a unit test checks that both evaluators produce identical output on mixed data.

Finally, sinks can enable an offline cache using eKuiper's own options: failed sends queue FIFO in memory up to a threshold, then in disk pages, and only when the disk budget is exhausted are the oldest records dropped — and counted rather than silently lost. The authors note this cache is covered by an integration test but excluded from the performance figures.

Why it matters

Edge gateways typically get one or two cores and a few hundred megabytes of spare memory, and their traffic is bursty in the worst way: fleets reconnect together, chargers start sessions simultaneously, and devices flush buffered readings all at once after an outage. In that environment, memory that grows with message rate is what kills a pipeline, not throughput. A drop-in engine compatible with eKuiper that keeps memory bounded under load could let existing edge deployments run on smaller, cheaper hardware — though the numbers come from the rekuiper team's own benchmark of a beta release, so independent replication is the natural next step. Just as valuable is the methodology: exact output verification under deliberately cramped resource limits is a template for evaluating stream processors honestly.