Back to docsStorage

Inside CommitLog sharding: where 5x throughput comes from

A walk through the write path: the global lock, how shards and offset encoding work, what dispatch decoupling changes, and when the 5x figure applies (project self-test).

Site summary · Clean-RocketMQ engineering note

RocketMQ appends messages in order to a single CommitLog. Sequential writes plus page cache have long been treated as the standard answer to throughput. In real benchmarks the bottleneck is often not the disk: every write thread contends for the same putMessageLock, and encoding, offset allocation, appending and waiting for flush all happen inside it; the dispatch thread then parses the log serially to build ConsumeQueue and IndexFile. The more cores and the faster the disk, the more visible that lock and that dispatch path become. CommitLog Sharding in Clean-RocketMQ (Next-Gen RocketMQ) changes exactly this contention structure.

The serial point of one log

Apache RocketMQ 4.9.1 once moved encoding and MessageId generation outside the write lock, lifting small-message production TPS by roughly 28%. That optimisation shows lock work does affect throughput, but the lock itself is still global. The single-CommitLog structure forces writes to queue, and the disk advantage of sequential writes cannot turn into parallel writes across cores. The dispatch side is the same: one ReputMessageService reads the log in order, so index build speed decides when messages become visible to consumers.

The global lock and single dispatch thread on a single CommitLog write path
The global lock and single dispatch thread on a single CommitLog write path

Splitting one lock into N

A shard is chosen before writing. Each shard owns its own CommitLog instance, MappedFileQueue and write lock; once a message enters a shard it is still appended in order. The default path picks a shard per message at random, spreading traffic evenly across several logs; shard selection is wrapped in the routing layer, so it can later be replaced by a per-thread or per-topic policy. Lock contention drops from global to per-shard, and write concurrency rises with the shard count. The disk sees N sequential write streams, which is where NVMe multi-queue capability finally pays off.

Shard count is configurable, but the capacity ceiling is set by the encoding radix, covered below.

Messages routed to multiple independent CommitLogs, each holding its own write lock
Messages routed to multiple independent CommitLogs, each holding its own write lock

Offset encoding keeps compatibility

If each shard kept its own offset, consumer queues, client checkpoints and replication would lose a shared coordinate. Clean-RocketMQ encodes it instead: the global offset equals the in-shard offset multiplied by the maximum shard count, plus the shard id. Reading takes the modulo to get the shard and integer division to get the position inside it. One 64-bit integer still locates any message, and ConsumeQueue entries, checkpoints and the replication path all keep the same coordinate; RocketMQ compatibility stays at the protocol layer, and clients need no change.

The cost is a sparse offset space: adjacent messages no longer have adjacent numbers. Tools that rely on offset continuity for monitoring, migration or range scans have to interpret offsets under the new rule. The maximum shard count is part of the encoding, so raising it after writes have landed makes old data undecodable; fix the ceiling at deployment time.

Global offset encoded from in-shard offset and shard id, decoded on read
Global offset encoded from in-shard offset and shard id, decoded on read

Dispatch leaves the write path

Each shard has a CommitListener that reads the log in in-shard offset order and pushes events into a shared DispatchQueue. A fixed pool of Dispatcher threads drains the queue and builds ConsumeQueue and Index. Write threads return ACK without waiting for index construction, and dispatch is no longer limited to a single thread. The default is 5 dispatch threads, adjustable against backlog.

The cost is that dispatch delay becomes its own metric. When writes succeed but consumers see nothing, check queue backlog and dispatch threads before blaming disk utilisation.

Write path and dispatch path decoupled, index construction done asynchronously by a dispatcher pool
Write path and dispatch path decoupled, index construction done asynchronously by a dispatcher pool

When 5x holds

Our published comparison is about 5x throughput. Read that as an indication of the ceiling once the serial point is removed, not as a constant any configuration can reproduce. The gain needs several preconditions: multiple cores, an SSD that can handle parallel queues, enough concurrent producers, mostly small and medium messages, and asynchronous flush. With a single HDD, a single producer or mostly synchronous flush, the bottleneck falls back to the device and fsync, and sharding only adds complexity.

Load profileShard gainWhy
High-concurrency small-message writesHighLock contention is the main bottleneck
Many topics, many queue event streamsHighBoth write and dispatch can run in parallel
Strict global FIFONeeds reviewNo global write order across shards
Large messages, bandwidth-boundLowNetwork and disk bandwidth dominate
Low-traffic single nodeLowNot enough concurrency; sharding adds overhead
Mostly synchronous flushLow to mediumfsync serialises at the device layer

An architect's checklist

Write-heavy transactional message loads that can accept partitioned ordering fit this design: order state transitions, trade messages, logs and telemetry, IoT telemetry, feed distribution. What they share is a write volume far above the cost of handling one message, with ordering usually scoped to a business key.

Strict-ordering workloads need their boundary checked first. The default random per-message shard choice scatters messages of one consumer queue across several logs, so cross-shard write order no longer equals send order. Topics that need strict FIFO should use fixed shard routing or verify queue order after dispatch; do not assume global order survives sharding.

When benchmarking, align at least four settings: shard count, message size, flush mode and replica mode. A 5x+ throughput number only means something when reliability settings match.

Sharding does not make the disk faster; it splits the part that used to be serial so that cores and queues can actually take part in writing. The change is concentrated in the contention structure, while offset encoding and the dispatch interface stay simple, and the RocketMQ message model needs no relearning. That is why we insist on Clean at its core in the storage layer. Whether it is worth adopting depends on whether the global lock is your bottleneck today.

Related module

store/ · commitlog/

CommitLog instances, shard routing, offset encoding and the dispatch pipeline.

Start here in the repository.