The kstreams-ts library is a new Node.js implementation of the Kafka Streams model, built on top of the @platformatic/kafka client. It enables stateful processing with exactly-once delivery guarantees, allowing state to persist across consumer failures and rebalances.
The library uses local SQLite databases for state storage, with each record duplicated into a compacted Changelog topic. It leverages Kafka transactions to ensure that output records, Changelog entries, and offsets are committed atomically. The project supports DSL operations like filter, map, groupByKey, and count, as well as a Processor API.
To prevent zombie workers from committing data, the system implements a two-level fencing mechanism. It uses KIP-447 to validate consumer group generations at the broker level and tracks task ownership epochs within the worker. If an epoch changes, the transaction is aborted.
The developer implemented a crash suite for testing that uses SIGKILL to terminate applications during data processing. This suite identified a bug in the @platformatic/kafka client where aborted transactions caused valid records to be filtered out. A fix was submitted in PR #470.
The current beta version (0.1) requires Node.js 24+ and Kafka 3.6+. It currently lacks support for joins, repartition topics, and specific windowing types like hopping or session windows. The developer notes that compatibility with Java Kafka Streams API or formats is not provided.
