Back to docsArchitecture

After a from-scratch rewrite, why stay compatible with RocketMQ

A rewrite was about redrawing boundaries, not abandoning the ecosystem. Why compatibility is the migration path, and what it costs inside the code.

Site summary · Clean-RocketMQ engineering note

The Next-Gen RocketMQ positioning came out of a choice that was not easy. We spent 1,200+ days working with RocketMQ, then 400 days rewriting the core from scratch. That order decides what Clean-RocketMQ is trying to solve.

The reason to rewrite sits in maintenance cost

The RocketMQ core is under 50,000 lines. Judged by line count alone, adding features or tuning performance on top of it is not that hard. Our judgement was more direct: the real challenge is keeping the code simple and maintainable over time.

A message queue that has run for years accumulates compatibility branches, scenario patches, and blurry responsibilities between the domain layer and the storage layer. Features still work, but the boundaries get harder to recognise. Incremental change solves the immediate problem and tends to lock the old coupling in place. The rewrite was about drawing the boundaries again.

Layering behind Clean at its core: external contracts stay stable while domain and storage are redrawn
Layering behind Clean at its core: external contracts stay stable while domain and storage are redrawn

Clean at its core. Build for evolution.

Clean means the core is clean enough: a micro-framework and DDD layering separate domain from storage so each module's job can be explained on its own. Build for evolution means the structure leaves room for later change, and change can happen where boundaries are clear.

The rewrite bill is not just the development cycle

Starting from scratch means walking through the pits RocketMQ already fell into. Message model, client conventions, transaction semantics and message scheduling each have to be aligned again. The roadmap reflects that pace: publish/subscribe, ordered and delayed messages, transactions, registry and RPC are done, and CommitLog sharding has landed; master-slave and storage-compute separation are still in progress, while Raft and cloud native are not finished.

Code can be written in 400 days; trust takes longer. Existing production systems do not migrate because of a new architecture diagram. Users first check two things: do clients and operations tooling have to change, and does troubleshooting follow the same path.

Compatibility addresses migration cost

The value of a message queue lives beyond the broker, in clients, monitoring, alerting and operational habits. A separate ecosystem would ask every user of those facilities to replace them at once. The performance gain happens at one point; the migration cost is carried by the whole chain.

Clean-RocketMQ chooses RocketMQ Compatible: the same message model and client conventions, so it can replace native RocketMQ fully or partially. That makes migration decisions small. A team replaces one non-critical cluster first, keeps the existing clients and operations path, validates throughput and transaction behaviour under real load, and then decides whether to widen the scope.

Compatibility splits migration into full and partial replacement while peripheral tooling stays untouched
Compatibility splits migration into full and partial replacement while peripheral tooling stays untouched

Compatibility also constrains the rewrite. External contracts cannot move, which is what gives the internal implementation room to be reorganised. CommitLog sharding is a result of that constraint: RocketMQ has multiple queues share one append log, while Clean-RocketMQ splits the write path across shards. In the comparison the project publishes, throughput reaches more than 5x RocketMQ, or 5x+ throughput. That figure comes from the project's own comparison; before migrating, re-measure with your message size, flush policy and hardware.

CommitLog sharding splits the write path, with throughput marked as the project's own comparison
CommitLog sharding splits the write path, with throughput marked as the project's own comparison

Simplicity has to support the next change

The hard part of a transactional message queue is consistency. Clean-RocketMQ uses TCC distributed transactions to coordinate message sending and scheduling, and is filling in delay, expiry, periodic and cyclic scheduling. Architecturally it pushes storage-compute separation down into the storage layer, leaving an interface for later cloud-native deployment.

All of this points at one goal: simplicity that survives beyond the first version. Line count grows with features; clear boundaries keep that growth controllable. The project is open source under Apache-2.0, readable and changeable, so users can judge for themselves whether it fits their system.

A from-scratch rewrite is the means, compatibility is the path, and long-term maintainability is the goal. Keep the complexity inside, bring the migration cost down, and the code has a chance to be read, changed, and used in real business for many years.

Related module

docs/ · architecture/

Positioning note: why the rewrite happened, and how compatibility constrains it.

Start here in the repository.