• 1B rows (14 GB): 204.6s → 13.5s
• 100M rows (1.4 GB): 21.0s → 0.5s
• Code: lehoangtran289/1brc-java

0. The challenge

The input is a text file with one measurement per line, in the format <station name>;<temperature>. The temperature always has exactly one fractional digit:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Istanbul;23.0

The program has to compute the min, mean and max temperature for each station and print them sorted by station name:

{Abha=-23.0/18.0/59.2, Abidjan=-16.2/26.0/67.3, Abéché=-10.0/29.4/69.0, ...}

The rules that matter for the tricks below: station names are 1 to 100 bytes of UTF-8, there are at most 10,000 unique stations (the generated data has about 413), and temperatures range from -99.9 to 99.9.

1. Setup

The 1B times don’t scale 10× from 100M. Pass 6 took 1.7s on 100M, which would predict 17s for 1B, but the 1B run took 43.9s. The 14 GB file likely doesn’t stay in page cache next to the JVM and the IDE on 16 GB of RAM, so the 1B runs include disk reads.

2. Results

Each pass below starts from what the previous profile showed, then describes the idea it led to and the result. The flame graph at the end of a pass is the evidence for the next one.

Pass 1: Baseline (21.0s)

The baseline is the most obvious Java you could write: read each line with Files.lines, split it on ;, parse the temperature with Double.parseDouble and aggregate into a HashMap. A TreeMap sorts the output at the end.

Map<String, Stats> stats = new HashMap<>();

try (Stream<String> lines = Files.lines(Path.of(FILE))) {
    lines.forEach(line -> {
        String[] parts = line.split(";");
        String station = parts[0];
        double measurement = Double.parseDouble(parts[1]);

        stats.computeIfAbsent(station, k -> new Stats()).add(measurement);
    });
}

System.out.println(new TreeMap<>(stats));

In the profile, readLine takes 36%, String.split about 29%, parseDouble about 21% and the HashMap about 11%.

Pass 2: mmap + parse bytes by hand (21.0s → 7.0s)

The three biggest frames in Pass 1, readLine, split and parseDouble, all do the same kind of work. They turn text into objects: a String for every line, then substrings and a String[], then text into a double. The aggregation itself, the HashMap, is only about 11%.

I removed those object-creating steps one at a time and profiled again after each one.

Step 1: replace readLine with mmap. The file is mapped into memory as a MemorySegment (the Foreign Memory API, final in Java 22, preview in 21). The code now scans the mapped bytes for \n itself, but it still does new String for every line. That brought the time down to 19.4s.

try (FileChannel channel = FileChannel.open(Path.of(FILE), StandardOpenOption.READ)) {
    MemorySegment file = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size(), Arena.global());
    // ...
}

Step 2: find ; in the same byte scan instead of calling split. The name and the temperature were each still turned into a String. That got it to 12.6s.

Step 3: replace parseDouble. Since every temperature has exactly one fractional digit, I parse the digits straight into an int of tenths, so -12.3 becomes -123. The stats stay in integers until printing. That reached 7.0s.

// "-12.3" -> -123
int sign = 1;
currentByte = file.get(JAVA_BYTE, offset++);    // first char
if (currentByte == MINUS) {
    sign = -1;
    currentByte = file.get(JAVA_BYTE, offset++);
}
int temp = toDigit(currentByte);                // first digit
currentByte = file.get(JAVA_BYTE, offset++);
if (currentByte != DOT) {
    temp = temp * 10 + toDigit(currentByte);    // second digit
    offset++;                                   // skip '.'
}
currentByte = file.get(JAVA_BYTE, offset);
temp = temp * 10 + toDigit(currentByte);        // fraction digit
offset += 2;                                    // skip fraction digit & '\n'
int measurement = sign * temp;

Why mmap saved only 2s

The mmap step saved only about 2s, even though readLine had been 7.5s of the profile. The reason is that readLine did three jobs: getting the bytes, finding the \n and building the String. The mmap version still does the last two, which now show up as MemorySegment.get (about 2.6s) and String.<init> (about 2.1s). Copying the file was never the cost. Here is each piece measured in isolation:

The lesson: to estimate a saving, you have to subtract all the new frames that still do the old job, and then confirm end to end with hyperfine, not with a single profiling run.

Pass 3: byte[] key instead of String (7.0s → 2.98s)

In Pass 2’s profile, HashMap.computeIfAbsent is about 24% (String.hashCode, equals) and String.<init> about 16%. Both costs exist only because the program still builds a String for every line to use as a map key, which then gets hashed and compared.

I switched to using the name bytes as the key. One reusable Key, holding a scratch buffer, the length and a hash computed while scanning, does every lookup. The bytes get copied into a new Key only the first time a station shows up, which happens about 413 times in 100M lines.

Pass 4: Custom hash table (2.98s → 2.65s)

With the String gone, the map itself is the next cost. In Pass 3’s profile, HashMap.get is 27%, spent in getNode → Key.equals → Arrays.equals. Every lookup goes through several layers (virtual hashCode/equals, node objects), all for a table of only ~413 keys.

I replaced it with my own open-addressing table: 2^14 slots with linear probing, indexed by the precomputed hash. On a probe, it compares the hash and the length first and looks at the bytes only if both match. One Entry holds both the name and the stats, so a lookup returns the thing to update directly.

public Entry getOrInsert(byte[] buffer, int length, int hash) {
    int index = (hash ^ (hash >>> 14)) & MASK;

    while (true) {
        Entry entry = entries[index];

        if (entry == null) { // new station: copy the name only here
            entries[index] = new Entry(Arrays.copyOf(buffer, length), hash);
            return entries[index];
        }

        // cheap checks first, bytes last
        if (entry.hash == hash
                && entry.nameBuffer.length == length
                && Arrays.equals(entry.nameBuffer, 0, length, buffer, 0, length)) {
            return entry;
        }

        index = (index + 1) & MASK; // collision: linear probing
    }
}

With 2^14 slots and ~413 keys, the table is under 3% full, so probe chains stay short.

Pass 5: 8 bytes at a time (2.65s → 1.8s)

In Pass 4’s profile, about 47% is MemorySegment.get → getByte, which reads one byte per call and checks bounds and liveness on every call. Another 23% is Arrays.equals, comparing names byte by byte.

Both come down to handling one byte at a time, so the fix is to read 8 bytes at once as a little-endian long. To find the ; inside that long without looping over its bytes, I used a SWAR (SIMD Within A Register) bit trick:

private static final long SEMI_PATTERN = 0x3B3B3B3B3B3B3B3BL; // ';' in every byte
private static final long ONES         = 0x0101010101010101L;
private static final long HIGH_BITS    = 0x8080808080808080L;

private static long findSemicolon(long word) {
    long input = word ^ SEMI_PATTERN;            // the ';' byte becomes 0x00
    return (input - ONES) & ~input & HIGH_BITS;  // marks the first 0x00 byte
}

1. XOR with ;;;;;;;; turns the ; byte into 00 and every other byte into something non-zero.
2. (x - 0x0101…) & ~x & 0x8080… is the classic “find a zero byte” expression. It sets the high bit of the first 00 byte.
3. Because the long is little-endian, the first byte in the file is the lowest byte of the long, so Long.numberOfTrailingZeros(match) >>> 3 gives the index of ; (0 to 7).

For example, Hamburg; fits in one long with ; as byte 7, so match is 0x8000000000000000, which has 63 trailing zeros, and 63 >>> 3 = 7.

The name is kept as those long words too: they are XOR’d into the hash as they’re read and compared as a long[] in the table. A typical name now takes 1 or 2 compares instead of about 10.

while (true) {
    word = file.get(LONG_LE, offset);   // read 8 bytes
    match = findSemicolon(word);
    if (match != 0) break;              // ';' is in this word

    words[wordsLength++] = word;
    hash ^= word;
    offset += 8;
}
// keep only the bytes before ';' in the last word
long nameBytes = word & ((match ^ (match - 1)) >>> 8);

Pass 6: Branch-free temperature (1.8s → 1.7s)

About 14% of Pass 5’s profile is still MemorySegment.get → getByte. That is the temperature, which was still parsed byte by byte, with branches on the sign and the digit count. Both depend on random data, so the branch predictor can’t learn them. On top of that, getOrInsert (about 11%) runs a wordsEqual loop on every lookup.

I made two changes.

The temperature is now read as one long. The 4 possible shapes are X.X, -X.X, XX.X and -XX.X. A bit trick finds the .: ASCII digits (0x30–0x39) have bit 4 set and . (0x2E) doesn’t. Shifts and one multiply then compute the value with no branches. This one is not mine: credit goes to merykitty and royvanrijn from the original challenge.

private static final long DOT_BITS = 0x10101000;

// bit 4 of bytes 1, 2, 3: the one that is 0 is the '.'
// returns 12, 20 or 28
private static int dotPos(long word) {
    return Long.numberOfTrailingZeros(~word & DOT_BITS);
}

The first two name words are now fields on Entry. In the 100M file, about 98% of lines have a station name of 15 bytes or less, so the name and its ; fit in the first two words. Those lookups are two long compares with no loop. Only longer names fall back to comparing the rest of the array.

if (entry.hash == hash
        && entry.word0 == words[0]
        && entry.word1 == words[1]
        && (wordLength <= 2 || wordTailEquals(entry.words, words, wordLength))) {
    return entry;
}

Pass 7: Drop the per-read bound check (~1.7s, no change)

In Pass 6’s profile, about 62% of the samples are in readLong → MemorySegment.get → getLongUnaligned, and readLong checks offset + 7 < fileSize on every read.

A line reads at most 108 bytes past its start, so every line that starts more than 128 bytes before EOF can use a plain file.get. The last ≤128 bytes are copied into a zero-padded buffer and parsed by the same code:

long tailLength = fileSize - tailStart;
MemorySegment tail = Arena.global().allocate(tailLength + 128); // zero-filled
MemorySegment.copy(file, tailStart, tail, 0, tailLength);
processFileRange(tail, 0, tailLength, stationTable);

It made no difference on 100M. On 1B the time went from 43.9s to 42.2s, which is likely run-to-run noise.

The reason is that the check was always true, so the branch predictor always got it right and it cost almost nothing. The profiler puts the time on the load because that’s where the CPU waits. Each line’s reads depend on where the previous line ended, and new pages cause page faults.

The lesson: a hot load means “waiting for data”, not “the read code is slow”. Removing predictable instructions that aren’t on that chain of dependent steps doesn’t save time.

Pass 8: Parallel chunks (1.7s → 0.5s)

Every profile so far shows one thread, main, mostly waiting on that chain of dependent loads, while 7 of the 8 cores sit idle.

So I split the file into line-aligned chunks and parsed them in a parallel IntStream. Each chunk gets its own StationTable, so there’s no sharing and no locks, and the tables are merged by station name at the end.

private static final int CHUNK_COUNT = Runtime.getRuntime().availableProcessors() * 16;

StationTable[] stationTables = new StationTable[CHUNK_COUNT];
long[] boundaries = splitFileIntoChunks(file, tailStart); // each boundary is a '\n'

IntStream.range(0, CHUNK_COUNT)
        .parallel()
        .forEach(i -> {
            long start = (i == 0) ? 0 : boundaries[i - 1] + 1;
            stationTables[i] = new StationTable();
            ChunkProcessor.processFileRange(file, start, boundaries[i] + 1, stationTables[i]);
        });

// merge
TreeMap<String, Entry> results = new TreeMap<>();
for (StationTable table : stationTables) {
    for (Entry entry : table.entries) {
        if (entry == null) continue;
        results.merge(new String(entry.nameBuffer, StandardCharsets.UTF_8), entry, Entry::merge);
    }
}

A few details:

• There are 128 chunks (8 cores × 16), not 8. The M1 Pro has 6 performance and 2 efficiency cores, so equal-sized chunks per core would leave the fast cores waiting on the slow ones. With many small chunks, the fast cores just take more of them.
• Reads may cross a chunk’s end by a few bytes. That’s harmless, because the mapping is read-only and the bytes past the end belong to the next line anyway.
• The merge walks 128 small tables of ~413 stations each, which is tiny next to the parse.

The work is now spread across ForkJoinWorkerThreads, and each one is still dominated by MemorySegment.get. On 1B the run took 13.5s, far from 10 × 0.5s, which fits the page-cache note in Setup.

Possible next directions

The official leaderboard is topped by 1.535s, measured on 8 cores of an AMD EPYC 7502P server with the file on a RAM disk. The README’s FAQ points out that wall-clock times are only comparable on the same machine, so my 13.5s on a laptop says little about where this solution would rank. The README does list ideas, though, and these are the ones I haven’t tried yet:

• Measure 1B without the disk. The official runs read the file “from a RAM disk”, so I/O isn’t part of the result. My 1B runs likely include disk reads (see Setup). A machine with enough RAM to keep the 14 GB file cached would show how much of the 13.5s is actually parsing.
• Unsafe instead of MemorySegment. Every entry in the top 10 is marked “uses Unsafe”, and MemorySegment.get has been the biggest frame since Pass 4. Pass 7 is a warning, though: if that frame is the CPU waiting on memory, cheaper reads won’t save much.
• Start-up time: GraalVM native image, AppCDS, CRaC. The README lists these for “speeding up the application start-up”, and 8 of the top 10 entries run as a GraalVM native binary. At 0.5s for 100M rows, JVM start-up and JIT warm-up are likely a noticeable share of the time.
• The Vector API. The README suggests “using the (incubating) Vector API”. Pass 5 looks for ; 8 bytes at a time with SWAR, while SIMD registers can check 16 or more bytes per instruction.
• Choosing and tuning the GC. This is also on the README’s list. After Pass 3 the hot loop allocates almost nothing, so I’d expect little from it.
• 10,000 stations. The generated data has 413 stations, but the rules allow up to 10,000, and the README’s 10K key set results note that entries “may perform very differently” there. My 2^14-slot table would be about 61% full instead of under 3%, so probe chains get longer. That’s worth measuring with the repo’s create_measurements3.sh.
• Read the top entries. Each leaderboard row links to its source. Quan Anh Mai (merykitty), whose temperature trick Pass 6 uses, is one of the three authors of the winning entry. The README’s “1BRC on the Web” list also has write-ups, such as Marko Topolnik’s step-by-step from 71s to 1.7s.

The full code and the benchmark scripts are on GitHub. If you spot a mistake or have an idea for the next pass, feel free to leave a comment or contact me directly.

I wrote this post based on my own conceptual understanding of the topic, if you notice any mistakes or have any suggestions for improvement, feel free to leave a comment or contact me directly. I will be happy to update the post accordingly.

0. Introduction

The Isolation level (I in ACID) is the setting that fine-tunes the balance/trade-off between performance and reliability, consistency when multiple transactions are performing queries simultaneously.

From highest amount of consistency and protection to the least, the isolation levels supported by MySQL InnoDB are: SERIALIZABLE, REPEATABLE READ (default), READ COMMITTED, and READ UNCOMMITTED.

1. READ UNCOMMITTED

In this mode, a transaction is allowed to READ UNCOMMITTED data from other transactions. This is the lowest isolation level, and it can lead to DIRTY READS (visible uncommitted changes) and other anomalies that we will cover in the next sections.

2. READ COMMITTED

A transaction can only READ COMMITTED data from other transactions. This prevents DIRTY READS, but it can still lead to:

• NON-REPEATABLE READS (data can change between reads in the same transaction).
• PHANTOM READS (#rows changed between 2 reads in the same transaction).

This example illustrates the NON-REPEATABLE READ phenomenon, but similar logic can be applied to phantom reads:

How MySQL implements READ COMMITTED?

MySQL takes a “snapshot” (latest committed data) of the database’s current state for each query in a transaction.

When a transaction reads data, it accesses the snapshot instead of the source table. A snapshot is refreshed and synced with committed data for subsequent queries within the same transaction. For example Tx1 COMMIT could refresh the snapshot for Tx2, and Tx2 will see the changes made by Tx1, just like the diagram above.

3. REPEATABLE READ

TLDR, REPEATABLE READ:

• Prevents NON-REPEATABLE READs by reusing the snapshot created by the transaction’s first plain SELECT. Rows committed by other transactions afterward remain invisible to this transaction.
• Prevents phantom reads by using gap locks and next-key locks for range scans.
• But there are edge cases

Compare to READ COMMITTED, there are 2 main differences at this level: (1) when does snapshot refresh? (2) locking strategy.

(1) Snapshot refresh

In this mode, a snapshot is taken at the transaction’s first read.

Transaction only sees 1 snapshot at its first read, and that snapshot is used for all subsequent reads within that transaction. ==> This ensures consistent subsequent READs throughout the transaction’s execution and avoids NON-REPEATABLE READS.

(2) Locking strategy

Under READ COMMITTED, InnoDB uses record locks for locking: it locks index records but generally does not lock the surrounding gaps. Because gaps remain unlocked, another transaction can insert a new row into the searched range, allowing phantom reads.

Meanwhile, REPEATABLE READ adds gap locks and next-key locks for range scans. I covered these in detail here MySQL locks post.

Basically, when a transaction executes a locking range operation (e.g., SELECT ... FOR UPDATE, UPDATE, or DELETE), InnoDB locks the scanned index range using next-key or gap locks. This prevents other transactions from inserting new rows and therefore prevents phantom reads.

-- Tx1
BEGIN;
SELECT * FROM users WHERE age BETWEEN 20 AND 30 FOR UPDATE;

-- Tx2: waits if age = 25 falls inside Tx1's locked index range
INSERT INTO users(age) VALUES (25);

Edge Case: Mixing Snapshot Reads and Current Reads

Under REPEATABLE READ, plain SELECT statements use a fixed MVCC snapshot, while write operation like UPDATE and DELETE operate on the latest committed version of the data directly.

-- Tx1
BEGIN;
SELECT * FROM users WHERE age = 25; -- Returns 0 rows; snapshot is established.

-- Tx2
INSERT INTO users(age) VALUES (25);
COMMIT;

-- Tx1
UPDATE users SET status = 'active' WHERE age = 25; -- Affects 1 row.
SELECT * FROM users WHERE age = 25; -- Returns 1 row; Tx1 can now see its own modification.

Although Tx1’s snapshot cannot see the row inserted by Tx2, the UPDATE uses the latest committed data (bypassing its snapshot), so it can find and modify the newly committed row. Note that the UPDATE statement does not refresh the snapshot, but Tx1 can see its own modification afterward.

Lost Update problem

This problem occurs when applications use a read-modify-write pattern instead of performing the modification atomically in the database.

For example, 2 transactions can read the same data, update it in its own memory, and write back to the database. The 2nd transaction wins and overwrites the first transaction’s update, resulting in a LOST UPDATE.

So how to prevent this? There are 2 approaches:

• Pessimistic locking: use SELECT ... FOR UPDATE to lock the row before reading it. This prevents other transactions from modifying the row until the transaction is committed or rolled back.
• Optimistic locking: use a version column to detect if the row has been modified by another transaction before updating it. If the version number has changed, the transaction can abort or retry.
  • In implementation, the version should be compared between client’s data vs the database’s data (etags or if-match HTTP pattern).

4. Serializable

This level is like REPEATABLE READ, but InnoDB implicitly converts all plain SELECT statements to SELECT ... FOR SHARE (shared locks). The exact locks depend on the query:

• UNIQUE index lookup: acquires a shared record lock on the found row.
• Range or NON-UNIQUE index scan: acquires shared next-key locks over the scanned index range.

Shared locks allow concurrent reads, but conflicting writes must wait. Next-key locks also prevent new rows from being inserted into the protected gaps.

As a result, SERIALIZABLE prevents dirty reads, non-REPEATABLE READs, phantoms, and typical lost-update patterns.

The trade-off is lower concurrency and potential deadlocks. Therefore in practice, applications often remain on REPEATABLE READ and use targeted SELECT ... FOR UPDATE queries where serialization is required to reduce locking scope.

Summary

Key takeaways:

• Isolation level is fundamentally a trade-off between consistency and performance ⇒ higher isolation means more locking, less throughput.

• READ UNCOMMITTED
  • best performance, lowest isolation level. It can lead to DIRTY READS and other anomalies.
• READ COMMITTED
  • Use snapshot mechanism: prevents DIRTY READS, but it can still lead to NON-REPEATABLE READS and PHANTOM READS.
• REPEATABLE READ (InnoDB default)
  • Blocks the most common anomalies (dirty read, non-REPEATABLE READ, phantom read on locking scans) while still being reasonably concurrent.
  • Beware for Mixing Snapshot Reads and Current Reads: plain SELECT reads the snapshot, UPDATE/DELETE/SELECT ... FOR UPDATE read the latest committed data.
  • Lost updates problem: Use pessimistic or optimistic locking to prevent this.
• SERIALIZABLE
  • Gives strongest data consistency, but trades off performance, hence reach for targeted SELECT ... FOR UPDATE on REPEATABLE READ instead.

References

• MySQL ACID
• A straightforward guide for isolation levels
• MySQL locks: A quick guide

Exclusive Locks (X) and Shared Locks (S)

Exclusive Locks (X)

As the name suggests, exclusive lock is a type of lock that prevents any other transaction from locking the same row.

It means that no other transaction can READ or WRITE to that resource until the lock is released (but this depends on the isolation level). InnoDB’s REPEATABLE READ allows tx to read rows with X locks (consistent read technique).

Shared Locks (S)

Shared Lock (S) allows other transactions to READ the locked object and acquire shared locks on it, but not to write to it.

It means that multiple transactions can hold shared locks on the same resource at the same time, allowing them to read the data concurrently.

Compatibility

Row lock can be either shared or exclusive. The compatibility of locks is as follows:

• Shared (S) locks are compatible with other shared locks, but not with exclusive locks (X). 1 row can have multiple shared locks at the same time.
• Exclusive (X) locks are not compatible with any other (X and S) locks. 1 row can only have one exclusive lock at a time.

Intention Locks (IS and IX)

Intention Lock is a table-level lock that indicates a transaction’s intention to acquire row-level locks on a table (either shared or exclusive). So there are 2 types of intention locks:

• Intention Shared Lock (IS): a transaction intends to acquire shared locks (S) on rows in the table. E.g., SELECT ... FOR SHARE

• Intention Exclusive Lock (IX): a transaction intends to acquire exclusive locks (X) on rows in the table. E.g., SELECT ... FOR UPDATE

Intention locks do not lock the whole table, but they mark that there exists a transaction that wants to lock rows in this table. This allows further transactions to know that they cannot acquire conflicting locks on the table without having to check every row, which improves efficiency.

Why Intention Locks are Needed

Simply put, an intention lock isn’t used to lock the new row; it is used to ask permission to enter the neighborhood and to provide a structured way to wait in line if the neighborhood is currently blocked.

For example, when a transaction wants to insert the number 5 into a gap between 4 and 7: The row 5 does not exist yet. Since an exclusive lock (X) is a record lock (meaning it must be physically attached to and existing slot or key in the index tree), there is no physical data structure for an exclusive lock to latch onto yet. Therefore, MySQL must lock the gap (4, 7) instead.

==> Hence The insert intention lock acts as the placeholder on the gap until the physical row is created. And it allows other transactions to insert into the same gap concurrently, as long as they are not trying to insert the same value.

Lock Compatibility

Record Locks

Record lock is a lock on an index record (not actual data row, but its index record). E.g, SELECT ... FOR UPDATE or SELECT ... FOR SHARE will acquire record locks on the index records of the selected rows.

Note: record locks always lock index records, even if table has no index. InnoDB will create a hidden clustered index and use it for locking.

InnoDB Index

• Clustered Index: associated with the primary key (PK) of the table. The data rows are stored in the leaf nodes of the clustered index.
  • If there is no PK, InnoDB will choose first UNIQUE NOT NULL column as the clustered index.
  • If there is no PK + UNIQUE column, InnoDB will create a hidden clustered index.
• Secondary Index: any Indexes other than the clustered index. Each record of a secondary index contains the indexed column values and its PK.

Gap Locks

Gap lock is a lock on the gap between index records. Its main GOAL is to prevent insertion of new rows into the gap range -> preventing PHANTOM READ.

E.g, SELECT ... WHERE id BETWEEN 10 AND 20 FOR UPDATE will acquire gap locks on the range (10, 20) and prevent insertions of new rows into that range.

NOTE: Gap lock can coexist even with conflicting types. For example, transaction A can hold a shared gap lock (gap S-lock) on a gap while transaction B holds an exclusive gap lock (gap X-lock) on the same gap. Because gap locks’ only purpose is to prevent insertions, and they do not conflict with each other in terms of reading or writing existing rows.

==> There is no difference between shared (S) and exclusive (X) gap locks.

Drawback of Gap Locks

Gap locks can lead to deadlocks in certain scenarios. For example:

• Tx A holds a gap lock on (10, 20), Tx B holds a gap lock on (10, 20)
• Now Tx A wants to insert a new row with id=15, it must wait for tx B to release its gap lock.
• But Tx B also wants to insert a new row with id=12, it must wait for tx A to release its gap lock. ==> This creates a deadlock

How to avoid deadlocks caused by gap locks:

• Use lower isolation levels: consider using READ COMMITTED instead of REPEATABLE READ to reduce the use of gap locks.
• Use consistent locking order: always lock rows in the same order across transactions.
• Use short transactions: keep transactions as short as possible to reduce the chance of deadlocks.

Next-Key Locks

Next key lock is a combination of Record Lock + Gap Lock before the index record. That is, a next-key lock is an index-record lock plus a gap lock on the gap preceding the index record.

MySQL performs row-level locking on index records it scans through.

Example: given index contains 5, 9, 10, and 20. Possible next-key locks cover the following ranges: (-inf, 5], (5, 9], (9, 10], (10, 20], and (20, +inf). Query SELECT ... WHERE id > 10 FOR UPDATE will acquire next-key locks on (10, 20] and (20, +inf).

Examples

Example 1: Unique Index vs Non-Unique Index

Assume an InnoDB index (REPEATABLE READ) contains: 10, 20, 30, 50

• Query: SELECT ... WHERE id = 20 FOR UPDATE;
  • On an UNIQUE index, InnoDB will acquire a record lock only on 20. No gap lock is needed because a unique index guarantees that no additional row with id = 20 can be inserted.
  • On a non-unique index, InnoDB will acquire a next-key lock on (10, 20] and a record lock on 20 to prevent insertion of new rows with id = 20.
• Query: SELECT ... WHERE id BETWEEN 15 AND 35 FOR UPDATE;

  • InnoDB will acquire next-key locks on the ranges (10, 20] and (20, 30], which also locks the rows 20 and 30 (record locks).

  • Using next-key lock, the resulting behavior is as follows:

    • Block Insert inside gap: INSERT ... VALUES (25);
    • Block Update locked record: UPDATE ... WHERE age = 30;
    • Block Move into gap: UPDATE ... SET age = 25 WHERE age = 10;

Conditions for using next-key locks:

• The table uses InnoDB and isolation level is REPEATABLE READ.
• The statement is a locking operation such as SELECT ... FOR UPDATE, UPDATE, or DELETE.
• The query performs a range or nonunique-index scan.
• The execution plan uses the relevant index.

A plain SELECT normally uses an MVCC snapshot (I will cover this in another topic about MySQL’s isolation level) and does not acquire these locks. An exact lookup using a complete unique index usually acquires only a record lock.

Example 2: Full Table Scan and Locks

Assume a table with index on column address but not nickname. The table definition is as follows:

• Query: UPDATE member SET age = 30 WHERE address='HN' AND nickname='T';
  • Since there is no index on nickname, InnoDB will perform a full table scan and acquire next-key locks on all rows that match the condition address='HN'. In this case, all write operations with address='HN' will be blocked until the transaction is committed or rolled back. ==> This can impact concurrency performance.

What happens if the table does not have an index? In this case, the UPDATE operation is performed by full scanning the table, and locks are set on all table records.

==> So index could affect not only query performance but also concurrency performance, and must be designed well to avoid unnecessary locks.

Example 3: How INSERT works with locks

Assume application performs multiple INSERT operations simultaneously with the same PK value. For example, two transactions try to insert a row with id=10 at the same time:

--- TX1
INSERT INTO users(id) VALUES (10);

--- TX2
INSERT INTO users(id) VALUES (10);

Locks Acquired by INSERT:

• an insert-intention lock on the target index gap ==> allows other transactions to insert into the same gap concurrently
• an exclusive record lock (X) on the inserted row

So, without loss of generality, assume T1 inserts the row and holds an exclusive lock (X) on id = 10. Hence T2 must wait, and when T1 commits, T2 will try to insert the same row but fails because of duplicate key error.

References

• MySQL 8.4 Reference Manual: Locking
• Straightforward Guide for MySQL Locks
• CSDN Article on MySQL Locks
• Tistory Article on MySQL Locks

1. Introduction to Container Orchestration

The challenge of Managing containerized applications: creating a container image and managing from your local laptop is easy. But managing a cluster of containerized applications across multiple distributed nodes in productions is much harder.

But once you have dozens (or hundreds) of conta`iners running across multiple machines, new problems appear:

• How do you deploy applications with horizontal scalability across multiple servers?
• How do you handle network connection between nodes and among containers?
• How do you handle monitoring, observability to ensure containers and nodes are resilient?
• How do you roll out new versions without service interruption?
• How do you group related containers?

This is where container orchestration comes in. It automates the deployment, scaling and management of containerized applications across a cluster of machines.

Popular container orchestrators includes: Kubernetes (K8s), Google GKE, AWS EKS, Azure AKS, Docker swarm. But we will focus on K8s in this post.

2. Kubernetes Architecture Overview

A Kubernetes cluster contains 2 main parts: Control Plane nodes and Worker nodes.

The control plane makes decisions, while Worker nodes actually run your containers.

2.1. Control Plane

The control plane is responsible for managing the entire K8s cluster. It coordinates all activities and communications with worker nodes to ensure desired state of the whole cluster.

The control plane core components:

• kube-apiserver:
  • Exposes K8s API
  • Acts as the entry point and gatekeeper for the cluster
  • Every interaction (from kubectl, helm chart, automation, …) goes through here. It validates and persists objects to etcd.
• kube-scheduler
  • Watches for newly created Pods that haven’t been assigned to a node yet, then picks the best node based on resource availability, taints, affinity rules, etc.
• etcd
  • A distributed key-value store.
  • This is where entire cluster state (configurations, metadata) lives: nodes, pods, configs, secrets. If etcd goes down, the cluster can’t function.
• kube-controller-manager
  • Runs background controller that continuously ensure the actual cluster state matches the desired state. E.g: node health, replica counts, job completion.
    • Node controller - notices when nodes go offline
    • ReplicaSet controller - ensures the right number of pod replicas are running
    • Endpoints controller - populates Service endpoints

2.2. Worker Nodes

Each worker node runs the following components:

• kubelet
  • An agent that runs on every node.
  • It talks to the API server, receives Pod specs, and ensures containers are running and healthy via the container runtime.
• kube-proxy
  • Handles networking and service communication
  • Enables communication between Pods and Services across the cluster. It implements the virtual IP abstraction for Services, forwarding traffic to the correct Pod endpoints.
• Container runtime
  • The software that actually runs containers (e.g., Docker, containerd, …).

Note: kubelet and kube-proxy also run on control-plan nodes, but primary role is managing workloads on worker nodes. (since control-plane components themselves)

2.3. Interacting with K8s cluster

Developer interacts with API Server running on control plane node for administration and application management.

The primary tool for interacting with a Kubernetes cluster is kubectl, the CLI client (basically curl).

Users access the deployed applications running on worker nodes via kube-proxy.

kubectl reads from a kubeconfig file (~/.kube/config by default) which contains cluster endpoints and credentials. You can manage multiple clusters using contexts:

# List available contexts
kubectl config get-contexts

# Switch context
kubectl config use-context my-cluster

• Common Q/A:
  • What happens if control plan crashes?
    • You the administrator ofcourse :D you should have set up multiple control plane nodes for HA. If one goes down, the others take over.
    • Also, if a control plane crashes, cluster management halts: existing pods keep running, but no new pods can be scheduled, updated, or deleted, and API requests fail. While worker nodes continue running, the system cannot self-heal or scale, and recovery requires restoring the API server and etcd database to resume operations.

2.4. How K8s runs your application

Kubernetes deploys workloads by placing containers into Pods

• Pods is smallest deployment unit, pods are scheduled to run on Nodes
• Kubernetes continuously monitors: Application health, Node health, Desired vs actual state
• Kubernetes is declarative, you tell it what you want, not how to do it and the control plane makes it happen.

At a high level, here’s the sequence of events when you run kubectl apply -f deployment.yaml:

  kubectl apply -f deployment.yaml
        │
        ▼
  kube-apiserver   ← validates manifest, stores in etcd
        │
        ▼
  kube-controller-manager  ← ReplicaSet controller notices new Pods are needed
        │
        ▼
  kube-scheduler   ← assigns each Pod to a suitable Node
        │
        ▼
  kubelet (on Node) ← pulls image, starts container via container runtime
        │
        ▼
  Container is running 🚀

1. You write a YAML manifest describing the desired state (e.g., “run 3 replicas of my-app using image my-app:1.2.0”).
2. You submit it to the API server, which authenticates, validates & persists desired state in etcd.
   • etcd maintains the source of truth for the cluster state.
3. The scheduler spots the unscheduled Pods in etcd and selects best worker nodes based on constraints, returns the scheduling decision to the API Server.
4. API Server persists the scheduling decision back to etcd, triggers kubelet on the assigned node.
5. The kubelet on assigned node pulls image and runs workload inside a pod.
6. The cluster (kube-controller-manager) continuously monitors the actual state and reconciles any drift - if a container crashes, it’s restarted automatically.

3. Kubernetes Objects

Kubernetes objects represent the resources that define and run applications inside a cluster. They are defined in YAML manifests and managed via the API server.

Every object has:

• apiVersion - Which version of the Kubernetes API you’re using to create this object
• kind - the type of object (e.g: Deployment, Service, …)
• metadata - data that helps uniquely identify the object (e.g: name, namespace, labels, annotations)
  • name - unique name for the object within its namespace
  • namespace - specifies which namespace the object belongs to (default is default)
  • labels - key-value pairs used for grouping and selecting objects (e.g: app: my-app, env: production)
• spec - the desired state you’re declaring, technical details, including image, applications, ports, etc.

3.1. Pod

• A Pod is the smallest deployable unit in Kubernetes, an abstraction over a container.
• It wraps one or more tightly coupled containers that share the same network namespace (same IP address) and storage.
• Kubernetes manages Pods, not individual containers.
• Pods are ephemeral - it can crash for any reason, kubelet coordinates with API Server to start a new pod if it is managed by ReplicaSet or Deployment.

• New pod = new cluster IP address

• E.g: In practice, most Pods run a single container. Multi-container Pods are used for sidecar patterns (e.g., a logging agent alongside your app).

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  labels:
    app: my-app
    env: production
spec:
  containers:
    - name: my-app
      image: my-app:1.2.0
      ports:
        - containerPort: 8080
      resources:
        requests:
          cpu: "100m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"

3.2. ReplicaSet

A ReplicaSet ensures how many replicas of a pod should be running (spec.replicas).

The selector.matchLabels field is how the ReplicaSet knows which Pods it owns. It will create or delete Pods until the count matches replicas. Example:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: frontend
  labels: # labels on the ReplicaSet itself, not the pods it manages
    app: guestbook
    tier: frontend
spec:
  # modify replicas according to your case
  replicas: 3
  selector:
    matchLabels: # which existing Pods belong to this ReplicaSet
      tier: frontend
  template: # When the ReplicaSet needs more Pods, it creates them using this template.
    metadata:
      # new pods created by this ReplicaSet will have these labels
      # must match the selector.matchLabels above
      labels: 
        tier: frontend
    spec:
      containers:
      - name: php-redis
        image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5

• ReplicaSet checks all Pods with label tier=frontend
• If there are less than 3, it creates new Pods using the template spec.template
• Those Pods get the same label: tier=frontend
• Now they match the selector → ReplicaSet manages them.

In practice, you rarely create ReplicaSets directly. You use a Deployment, which manages ReplicaSets for you and adds rolling-update capabilities.

3.3. Deployment

A Deployment manages ReplicaSets and provides a level of abstraction above ReplicaSet, which manages pods.

Creating deployment = creating ReplicaSets + pods

Deployment runs pods/containers, but it won’t expose any IP address or port#. You need to create a Service for that.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment # name of the deployment object
  labels:
    app: nginx
spec:
  replicas: 3
  # .spec.selector must match .spec.template.metadata.labels
  selector: # specifies a label selector for the Pods targeted by this Deployment
    matchLabels:
      app: nginx
  template: # pod template 
    metadata:
      labels: # labels for the pods created by this deployment, must match the selector above
        app: nginx
    spec:
      containers: # list of containers in the pod
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Note:

• The selector field is required in a Deployment. It tells the Deployment which Pods to manage. The template.metadata.labels must match the selector.matchLabels for the Deployment to work correctly.
• You should not create other Pods whose labels match this selector, either directly, by creating another Deployment -> controllers will fight and won’t behave correctly.

3.4. Service

Pods are ephemeral - their IP addresses change when they restart. This is where Services come in.

Service defines a stable DNS name and stable IP address to communicate with a group of pod instances. It acts as a virtual load balancer that distributes traffic to the healthy pods matching its selector.

Service and Pod life cycle are not connected. If a pod dies, the Service and its IP address still stay. So we can use a Service to expose workload inside or outside the cluster.

Within the cluster, Services are discoverable by DNS. A Service named my-app-service in namespace default is reachable at my-app-service.default.svc.cluster.local (or just my-app-service from within the same namespace).

Service types:

3.4.1. ClusterIP

It gives you a service inside your cluster that other apps inside your cluster can access. There is no external access. However, you can access it from outside the cluster using kubectl proxy --port=xxxx for debugging internal pods and development.

apiVersion: v1
kind: Service
metadata:  
  name: my-internal-service
spec:
  selector:    
    app: my-app # select pods with label app=my-app
  type: ClusterIP
  ports:  
  - name: http
    port: 80 # the port that the service will listen on
    targetPort: 80 # the port on the pod that the service will forward to
    protocol: TCP

3.4.2. NodePort

Opens a specific port on all Nodes, and any traffic that is sent to this port is forwarded to the cluster IP service. Accessible from outside the cluster at <NodeIP>:<NodePort>.

Cons: you can only have 1 service per NodePort, range is limited (30000-32767), and you need to manage load balancing across nodes yourself.

apiVersion: v1
kind: Service
metadata:  
  name: my-nodeport-service
spec:
  selector:    
    app: my-app # select pods with label app=my-app
  type: NodePort
  ports:  
  - name: http
    port: 80 # the port that the service will listen on
    targetPort: 80 # the port on the pod that the service will forward to
    nodePort: 30036 # the port on each node to expose (must be in range 30000-32767)
    protocol: TCP

3.4.3. LoadBalancer

Provisions an external load balancer (e.g., AWS ELB, GCP Load Balancer). The standard way to expose a service externally in cloud environments.

If your cluster is in a cloud, then Kubernetes first creates ClusterIP service, then opens high port on each node, and then reaches out to cloud provider’s API and configures a cloud load balancer with the host and port. Adding/removing nodes automatically updates load balancer config

Traffic -> load balancer -> NodePort -> clusterIP service, -> routed to the Pod

Cons: each exposed service with a LoadBalancer will get its own IP address, and you have to pay for a LoadBalancer per exposed service, which can get expensive!

Note: LoadBalancer > NodePort > ClusterIP. When you create NodePort -> it also creates ClusterIP. When you create LoadBalancer, it also creates NodePort and ClusterIP.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: MyApp # select pods with this label
  ports: # list of ports to expose
    - protocol: TCP
      port: 80 # the port that the service will listen on
      targetPort: 9376 # port on the pod to forward to
  clusterIP: 10.0.171.239
status:
  loadBalancer: # contains the status of the load balancer provisioned by the cloud provider
    ingress:
    - ip: 192.0.2.127

3.5. Ingress

Services handle east-west (internal) and basic north-south traffic. But for production HTTP/HTTPS routing from the internet, you use an Ingress.

An Ingress resource defines routing rules - path-based or host-based - to route external traffic to the right Services. It requires an Ingress Controller (e.g., NGINX Ingress Controller, Traefik) to be installed in the cluster to enforce those rules.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls-secret

The flow looks like this:

Internet → LoadBalancer → Ingress Controller Pod → Service → Pod

Key advantages of Ingress over a plain LoadBalancer Service:

• One LoadBalancer IP can serve many hostnames/paths (cost saving)
• TLS termination in one place
• Annotations for rate limiting, authentication, etc.

3.6. K8s Volumes

Containers have ephemeral filesystems - anything written inside a container is lost when it restarts. Volumes solve this.

A volume is mounted into a Pod’s container at a specified path. There are many volume types:

emptyDir - Temporary storage that lives for the lifetime of the Pod. Useful for sharing files between containers in the same Pod.

spec:
  volumes:
    - name: shared-data
      emptyDir: {}
  containers:
    - name: app
      volumeMounts:
        - name: shared-data
          mountPath: /data

hostPath - Mounts a directory from the host node’s filesystem. Useful for development but generally avoid in production (breaks portability).

PersistentVolume (PV) and PersistentVolumeClaim (PVC) - The standard pattern for durable storage.

• A PersistentVolume is a piece of storage provisioned in the cluster (manually or dynamically by a StorageClass).
• A PersistentVolumeClaim is a request for storage. Pods reference PVCs, not PVs directly.

# PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data-pvc
spec:
  accessModes:
    - ReadWriteOnce   # only one node can mount this at a time
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
---
# Reference PVC in a Pod
spec:
  volumes:
    - name: my-data
      persistentVolumeClaim:
        claimName: my-data-pvc
  containers:
    - name: app
      volumeMounts:
        - name: my-data
          mountPath: /var/lib/data

Access modes:

• ReadWriteOnce - mounted as read-write by a single node
• ReadOnlyMany - mounted read-only by many nodes
• ReadWriteMany - mounted read-write by many nodes (requires a distributed filesystem like NFS or EFS)

3.7. ConfigMap and Secret

Hardcoding configuration into container images is bad practice. Kubernetes provides two objects for externalizing config.

ConfigMap - stores non-sensitive configuration as key-value pairs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  APP_ENV: production
  LOG_LEVEL: info
  DB_HOST: postgres-service
  config.yaml: |
    server:
      port: 8080
      timeout: 30s

Secret - stores sensitive data (passwords, tokens, certificates). Values are base64-encoded (not encrypted by default - enable encryption at rest in production).

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  DB_PASSWORD: cGFzc3dvcmQxMjM=  # base64("password123")
  API_KEY: c3VwZXJzZWNyZXQ=

You can consume ConfigMaps and Secrets in Pods as environment variables or mounted files:

spec:
  containers:
    - name: app
      image: my-app:1.2.0
      # Option 1: inject all keys as env vars
      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets
      # Option 2: inject specific keys
      env:
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL
      # Option 3: mount as files
      volumeMounts:
        - name: config-volume
          mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: app-config

A few important notes:

• Never commit Secrets directly to version control. Use a secrets management tool like HashiCorp Vault, Sealed Secrets, or cloud KMS integration.
• Mounting as files is useful when your app expects config files rather than env vars.
• ConfigMap updates don’t automatically propagate to running Pods when consumed as env vars - you need to restart the Pod. When mounted as a file, updates do propagate (with a short delay).

3.8. StatefulSet

Deployments are great for stateless applications. For stateful workloads - databases, message queues, distributed systems - you use a StatefulSet.

StatefulSets provide:

• Stable, unique Pod names - my-db-0, my-db-1, my-db-2 (not random hashes)
• Stable network identities - each Pod gets a predictable DNS entry: my-db-0.my-db-service.default.svc.cluster.local
• Ordered, graceful deployment and scaling - Pods are created/deleted one at a time in order
• Per-Pod PVCs via volumeClaimTemplates - each Pod gets its own persistent volume

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-db
spec:
  serviceName: my-db-service   # headless service for DNS
  replicas: 3
  selector:
    matchLabels:
      app: my-db
  template:
    metadata:
      labels:
        app: my-db
    spec:
      containers:
        - name: my-db
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secrets
                  key: DB_PASSWORD
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

StatefulSets require a headless Service (a Service with clusterIP: None) to provide the stable DNS for each Pod:

apiVersion: v1
kind: Service
metadata:
  name: my-db-service
spec:
  clusterIP: None        # headless
  selector:
    app: my-db
  ports:
    - port: 5432

When to use StatefulSet vs Deployment:

Note that: This article is AI-generated based on the latest Spring Kafka documentation and best practices from several references.

1. Understanding Error Handlers

1.1. The Basics

Spring Kafka provides container-level error handlers that catch exceptions thrown by your @KafkaListener methods. These handlers determine what happens when message processing fails.

Starting with version 2.8, Spring Kafka introduced CommonErrorHandler to replace the legacy ErrorHandler and BatchErrorHandler interfaces. This unified approach handles both record and batch listeners with a single configuration.

Key points:

• The error handler runs after your listener throws an exception
• By default, DefaultErrorHandler is used with 10 retry attempts
• Error handlers can seek back to failed offsets or skip records entirely
• Works for both transactional and non-transactional listeners

1.2. DefaultErrorHandler Overview

The DefaultErrorHandler replaced SeekToCurrentErrorHandler and RecoveringBatchErrorHandler. It provides retry logic with configurable backoff strategies.

Default behavior:

• 10 retry attempts with no delay between attempts
• After retries exhausted → logs the failed record at ERROR level
• By default, seeks back to the failed offset for redelivery
• Can be configured with a custom recoverer (like publishing to DLT)

Basic configuration example:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
        ConsumerFactory<String, Order> consumerFactory) {
    
    ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory);
    
    // Configure error handler with 3 retry attempts, 1 second between retries
    DefaultErrorHandler errorHandler = new DefaultErrorHandler(
        new FixedBackOff(1000L, 2L)  // 1s delay, 2 retries = 3 total attempts
    );
    
    factory.setCommonErrorHandler(errorHandler);
    return factory;
}

Fatal exceptions — these skip retries entirely:

• DeserializationException
• MessageConversionException
• ConversionException
• MethodArgumentResolutionException
• NoSuchMethodException
• ClassCastException

These are considered fatal because retrying won’t fix the issue. The recoverer is invoked immediately.

2. Offset Commit Strategies

2.1. Understanding AckMode

Spring Kafka provides different acknowledgment modes that control when offsets are committed. This is critical for understanding message delivery guarantees.

Important: By default, Spring Kafka sets enable.auto.commit=false. The framework manages offset commits, not Kafka itself.

Available AckMode values:

• BATCH (default) — Commits offset when all records from the poll are processed
• RECORD — Commits offset after each record is processed
• MANUAL — Application calls Acknowledgment.acknowledge(), commit happens when the whole batch completes
• MANUAL_IMMEDIATE — Application calls Acknowledgment.acknowledge(), commit happens immediately

2.2. BATCH Mode (Default)

With BATCH mode, Spring Kafka processes messages one by one but only commits the offset after the entire batch from the poll completes.

Use case:

• Best for throughput
• Acceptable to reprocess some messages after failures
• Handles most standard scenarios

Properties:

spring:
  kafka:
    listener:
      ack-mode: BATCH  # This is the default

Behavior:

Poll returns: [msg1, msg2, msg3, msg4, msg5]
Process msg1 → success
Process msg2 → success
Process msg3 → FAILURE

What happens:

1. Container commits offsets for msg1 and msg2
2. Error handler seeks back to offset for msg3
3. Next poll returns [msg3, msg4, msg5] again
4. If app crashes before commit → msg1 and msg2 will be reprocessed

Trade-off: Higher throughput, but potential for duplicate processing if the consumer crashes between processing and commit.

2.3. RECORD Mode

With RECORD mode, the offset is committed immediately after each message is processed successfully.

Use case:

• Minimize duplicate processing after restarts
• Each message is expensive to process
• You can tolerate lower throughput

Properties:

spring:
  kafka:
    listener:
      ack-mode: RECORD

Behavior:

Process msg1 → commit offset for msg1
Process msg2 → commit offset for msg2
Process msg3 → FAILURE → error handling

Trade-off: Lower throughput (more commits), but fewer duplicates after failures.

2.4. MANUAL_IMMEDIATE Mode

This gives you full control. You decide exactly when to commit offsets by calling Acknowledgment.acknowledge().

Use case:

• Processing messages asynchronously with custom thread pools
• Need precise control over commit timing
• Building exactly-once semantics with external state

Properties:

spring:
  kafka:
    listener:
      ack-mode: MANUAL_IMMEDIATE

Example code:

@KafkaListener(topics = "orders", groupId = "order-processor")
public void listen(@Payload Order order,
                   Acknowledgment acknowledgment) {
    try {
        processOrder(order);
        // Only commit if processing succeeds
        acknowledgment.acknowledge();
    } catch (Exception e) {
        log.error("Failed to process order: {}", order, e);
        // Don't acknowledge → message will be redelivered
    }
}

Async processing example:

@Service
public class AsyncListener {

    private final ExecutorService executorService = 
        Executors.newFixedThreadPool(20);
    
    @Autowired
    private OrderProcessor processor;
    
    @KafkaListener(topics = "orders")
    public void listenAsync(@Payload Order order,
                           Acknowledgment acknowledgment) {
        // Submit to thread pool
        executorService.submit(() -> {
            try {
                processor.process(order);
                // Acknowledge after async processing completes
                acknowledgment.acknowledge();
            } catch (Exception e) {
                log.error("Async processing failed", e);
                // Don't acknowledge
            }
        });
    }
}

2.5. Async Acknowledgments (async-acks)

By default, out-of-order acknowledgments are disabled. If you’re using MANUAL or MANUAL_IMMEDIATE with async processing, you might want to enable this.

Properties:

spring:
  kafka:
    listener:
      ack-mode: MANUAL_IMMEDIATE
      async-acks: true  # Allow out-of-order commits

When to use:

• Processing messages with a custom thread pool
• Messages can complete in different order than received
• High-throughput scenarios where ordering isn’t critical

Warning: Enabling async-acks means offsets might be committed out of order. If message 5 finishes before message 3, offset 5 gets committed first. If the app crashes, message 3 and 4 might not be redelivered.

3. Dead Letter Topics (DLT)

3.1. Publishing Failed Records to DLT

After retries are exhausted, you typically want to send failed messages somewhere for later analysis rather than just logging them.

DeadLetterPublishingRecoverer publishes failed records to a dead letter topic.

Basic setup:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
        ConsumerFactory<String, Order> consumerFactory,
        KafkaTemplate<String, Order> kafkaTemplate) {
    
    ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory);
    
    // Create recoverer that sends to DLT
    DeadLetterPublishingRecoverer recoverer = 
        new DeadLetterPublishingRecoverer(kafkaTemplate);
    
    // Create error handler with recoverer
    DefaultErrorHandler errorHandler = new DefaultErrorHandler(
        recoverer,
        new FixedBackOff(1000L, 2L)  // 3 attempts: initial + 2 retries
    );
    
    factory.setCommonErrorHandler(errorHandler);
    return factory;
}

Default behavior:

• Failed records are sent to <originalTopic>.DLT
• Same partition as original record
• DLT topic must have at least as many partitions as original topic

3.2. Custom DLT Routing

You can customize which DLT to use based on the exception type or record content.

Example:

DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(
    kafkaTemplate,
    (record, exception) -> {
        if (exception.getCause() instanceof ValidationException) {
            return new TopicPartition(record.topic() + ".validation.DLT", record.partition());
        } else if (exception.getCause() instanceof DatabaseException) {
            return new TopicPartition(record.topic() + ".database.DLT", record.partition());
        } else {
            return new TopicPartition(record.topic() + ".DLT", record.partition());
        }
    }
);

3.3. DLT Headers

The recoverer adds several headers to the DLT record for debugging:

• kafka_dlt-exception-fqcn — Exception class name
• kafka_dlt-exception-cause-fqcn — Root cause class name
• kafka_dlt-exception-message — Exception message
• kafka_dlt-exception-stacktrace — Full stack trace
• kafka_dlt-original-topic — Original topic name
• kafka_dlt-original-partition — Original partition
• kafka_dlt-original-offset — Original offset
• kafka_dlt-original-timestamp — Original timestamp
• kafka_dlt-original-consumer-group — Consumer group

These headers help you:

• Debug what went wrong
• Know where the message came from
• Implement retry logic from the DLT later

3.4. Skipping Retries for Specific Exceptions

The DefaultErrorHandler considers certain exceptions fatal by default. You can add your own.

Example:

@Bean
public DefaultErrorHandler errorHandler(
        KafkaTemplate<String, Order> kafkaTemplate) {
    
    DeadLetterPublishingRecoverer recoverer = 
        new DeadLetterPublishingRecoverer(kafkaTemplate);
    
    DefaultErrorHandler handler = new DefaultErrorHandler(
        recoverer,
        new FixedBackOff(1000L, 2L)
    );
    
    // Add custom non-retryable exceptions
    handler.addNotRetryableExceptions(
        IllegalArgumentException.class,
        ValidationException.class,
        BadRequestException.class
    );
    
    return handler;
}

Now these exceptions skip retries and go straight to the DLT on first failure.

4. Batch Listener Error Handling

4.1. Using BatchListenerFailedException

For batch listeners, you must throw BatchListenerFailedException to indicate which specific record in the batch failed.

Basic batch listener:

@KafkaListener(
    id = "order-batch-processor",
    topics = "orders",
    containerFactory = "batchFactory"
)
public void processBatch(List<ConsumerRecord<String, Order>> records) {
    for (ConsumerRecord<String, Order> record : records) {
        try {
            processOrder(record.value());
        } catch (Exception e) {
            // Identify the failed record
            throw new BatchListenerFailedException("Failed to process", e, record);
        }
    }
}

For POJO batch listeners (no ConsumerRecord):

@KafkaListener(topics = "orders", containerFactory = "batchFactory")
public void processBatch(List<Order> orders) {
    for (int i = 0; i < orders.size(); i++) {
        try {
            processOrder(orders.get(i));
        } catch (Exception e) {
            // Use index instead of record
            throw new BatchListenerFailedException("Failed to process", e, i);
        }
    }
}

4.2. Batch Error Handler Configuration

Configure the batch factory with error handling:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Order> batchFactory(
        ConsumerFactory<String, Order> consumerFactory,
        KafkaTemplate<String, Order> kafkaTemplate) {
    
    ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory);
    factory.setBatchListener(true);  // Enable batch mode
    
    DeadLetterPublishingRecoverer recoverer = 
        new DeadLetterPublishingRecoverer(kafkaTemplate);
    
    DefaultErrorHandler errorHandler = new DefaultErrorHandler(
        recoverer,
        new FixedBackOff(1000L, 2L)
    );
    
    factory.setCommonErrorHandler(errorHandler);
    return factory;
}

4.3. How Batch Error Handling Works

When a record in a batch fails:

1. Offsets for records before the failed record are committed
2. Failed record (and remaining records) are retried according to BackOff
3. After retries exhausted → only the failed record goes to DLT
4. Failed record’s offset is committed
5. Remaining records are redelivered for processing

Example flow:

Batch: [msg1, msg2, msg3, msg4, msg5]
Process msg1 → success
Process msg2 → success
Process msg3 → FAILURE

What happens:

1. Commit offsets for msg1, msg2
2. Retry attempt 1: [msg3, msg4, msg5] → still fails on msg3
3. Retry attempt 2: [msg3, msg4, msg5] → still fails on msg3
4. Retry attempt 3: [msg3, msg4, msg5] → still fails on msg3
5. Send msg3 to DLT, commit offset for msg3
6. Continue with [msg4, msg5]

4.4. Batch Processing with Thread Pools

Processing batch messages asynchronously with a thread pool is a common pattern for maximizing throughput. However, it requires careful error handling to ensure no message loss.

4.4.1. The Challenge

When you submit batch records to a thread pool, the @KafkaListener method returns immediately. If you’re not careful, offsets get committed before async processing completes → messages can be lost if processing fails.

Basic setup (naive approach):

@Configuration
public class BatchAsyncConfig {
    
    @Bean
    public ExecutorService kafkaProcessorPool() {
        return new ThreadPoolExecutor(
            10,  // core threads
            50,  // max threads
            60L, TimeUnit.SECONDS,
            new LinkedBlockingQueue<>(1000)
        );
    }
    
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Order> batchFactory(
            ConsumerFactory<String, Order> consumerFactory) {
        
        ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
            new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.setBatchListener(true);
        
        // CRITICAL: Use MANUAL_IMMEDIATE for async processing
        factory.getContainerProperties().setAckMode(
            ContainerProperties.AckMode.MANUAL_IMMEDIATE
        );
        
        return factory;
    }
}

4.4.2. Reliable Async Batch Processing

Use CompletableFuture to track all async tasks and only acknowledge after all complete successfully.

Complete implementation:

@Service
public class AsyncBatchListener {
    
    private static final Logger log = LoggerFactory.getLogger(AsyncBatchListener.class);
    
    @Autowired
    private ExecutorService kafkaProcessorPool;
    
    @Autowired
    private OrderProcessor orderProcessor;
    
    @KafkaListener(
        topics = "orders",
        containerFactory = "batchAsyncFactory"
    )
    public void processBatch(
            List<ConsumerRecord<String, Order>> records,
            Acknowledgment acknowledgment) {
        
        log.info("Received batch of {} records", records.size());
        
        // Create a future for each record
        List<CompletableFuture<Void>> futures = new ArrayList<>();
        
        for (int i = 0; i < records.size(); i++) {
            final int index = i;
            final ConsumerRecord<String, Order> record = records.get(i);
            
            CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
                try {
                    orderProcessor.process(record.value());
                    log.debug("Successfully processed record at index {}", index);
                } catch (Exception e) {
                    log.error("Failed to process record at index {}", index, e);
                    throw new CompletionException(e);
                }
            }, kafkaProcessorPool);
            
            futures.add(future);
        }
        
        // Wait for all futures to complete
        try {
            CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
                .join();  // Blocks until all complete
            
            // All succeeded → commit offset
            acknowledgment.acknowledge();
            log.info("Batch processed successfully, offset committed");
            
        } catch (CompletionException e) {
            log.error("Batch processing failed, offset NOT committed", e);
            // Don't acknowledge → entire batch will be redelivered
            // This ensures no message loss but can cause duplicates
        }
    }
}

Key points:

• Uses MANUAL_IMMEDIATE ack mode
• CompletableFuture.allOf() waits for all tasks to complete
• Only calls acknowledge() if all tasks succeed
• If any task fails → batch is redelivered

Trade-off: This approach guarantees no message loss but means the entire batch is reprocessed if any single message fails.

4.4.3. Per-Record Acknowledgment with Async Processing

For finer-grained control, acknowledge successful records individually while handling failures separately.

Advanced implementation:

@Service
public class FinegrainedAsyncBatchListener {
    
    private static final Logger log = LoggerFactory.getLogger(FinegrainedAsyncBatchListener.class);
    
    @Autowired
    private ExecutorService kafkaProcessorPool;
    
    @Autowired
    private OrderProcessor orderProcessor;
    
    @Autowired
    private KafkaTemplate<String, Order> kafkaTemplate;
    
    @KafkaListener(
        topics = "orders",
        containerFactory = "batchAsyncFactory"
    )
    public void processBatch(
            List<ConsumerRecord<String, Order>> records,
            Acknowledgment acknowledgment) {
        
        log.info("Processing batch of {} records", records.size());
        
        // Track failed records
        ConcurrentHashMap<Integer, ConsumerRecord<String, Order>> failedRecords = 
            new ConcurrentHashMap<>();
        
        CountDownLatch latch = new CountDownLatch(records.size());
        
        for (int i = 0; i < records.size(); i++) {
            final int index = i;
            final ConsumerRecord<String, Order> record = records.get(i);
            
            kafkaProcessorPool.submit(() -> {
                try {
                    orderProcessor.process(record.value());
                    log.debug("Successfully processed record at index {}", index);
                } catch (Exception e) {
                    log.error("Failed to process record at index {}: {}", 
                        index, e.getMessage());
                    failedRecords.put(index, record);
                } finally {
                    latch.countDown();
                }
            });
        }
        
        // Wait for all tasks to complete
        try {
            if (!latch.await(60, TimeUnit.SECONDS)) {
                log.error("Batch processing timed out after 60 seconds");
                // Don't acknowledge → entire batch redelivered
                return;
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            log.error("Batch processing interrupted", e);
            return;
        }
        
        // Handle failures
        if (!failedRecords.isEmpty()) {
            log.warn("Batch had {} failures", failedRecords.size());
            
            // Send failed records to DLT
            failedRecords.values().forEach(record -> {
                try {
                    kafkaTemplate.send(
                        record.topic() + ".DLT",
                        record.key(),
                        record.value()
                    );
                } catch (Exception e) {
                    log.error("Failed to send to DLT: {}", record, e);
                }
            });
        }
        
        // Acknowledge the batch (successful records committed, failed sent to DLT)
        acknowledgment.acknowledge();
        log.info("Batch processing complete. Success: {}, Failed: {}", 
            records.size() - failedRecords.size(), failedRecords.size());
    }
}

This approach:

• Processes all records concurrently
• Tracks failures individually
• Sends failed records to DLT
• Acknowledges the batch once all processing attempts complete

Trade-off: More complex but provides better throughput. Failed messages go to DLT instead of reprocessing entire batch.

4.4.4. Retry Strategy for Async Batch Processing

Implement retry logic before sending to DLT to handle transient failures.

With retry mechanism:

@Service
public class RetryingAsyncBatchListener {
    
    private static final Logger log = LoggerFactory.getLogger(RetryingAsyncBatchListener.class);
    private static final int MAX_RETRY_ATTEMPTS = 3;
    
    @Autowired
    private ExecutorService kafkaProcessorPool;
    
    @Autowired
    private OrderProcessor orderProcessor;
    
    @Autowired
    private KafkaTemplate<String, Order> kafkaTemplate;
    
    @KafkaListener(
        topics = "orders",
        containerFactory = "batchAsyncFactory"
    )
    public void processBatch(
            List<ConsumerRecord<String, Order>> records,
            Acknowledgment acknowledgment) {
        
        log.info("Processing batch of {} records", records.size());
        
        List<CompletableFuture<RecordResult>> futures = records.stream()
            .map(record -> processWithRetry(record))
            .collect(Collectors.toList());
        
        // Wait for all to complete
        CompletableFuture<Void> allOf = CompletableFuture.allOf(
            futures.toArray(new CompletableFuture[0])
        );
        
        try {
            allOf.get(120, TimeUnit.SECONDS);
        } catch (Exception e) {
            log.error("Batch processing failed", e);
            // Don't acknowledge → batch redelivered
            return;
        }
        
        // Process results
        List<RecordResult> results = futures.stream()
            .map(CompletableFuture::join)
            .collect(Collectors.toList());
        
        // Send failed records to DLT after retries exhausted
        results.stream()
            .filter(result -> !result.isSuccess())
            .forEach(result -> sendToDlt(result.getRecord(), result.getException()));
        
        // Acknowledge batch
        acknowledgment.acknowledge();
        
        long successCount = results.stream().filter(RecordResult::isSuccess).count();
        long failureCount = results.size() - successCount;
        log.info("Batch complete. Success: {}, Failed: {}", successCount, failureCount);
    }
    
    private CompletableFuture<RecordResult> processWithRetry(
            ConsumerRecord<String, Order> record) {
        
        return CompletableFuture.supplyAsync(() -> {
            Exception lastException = null;
            
            for (int attempt = 1; attempt <= MAX_RETRY_ATTEMPTS; attempt++) {
                try {
                    orderProcessor.process(record.value());
                    return new RecordResult(record, true, null);
                } catch (Exception e) {
                    lastException = e;
                    log.warn("Attempt {}/{} failed for record offset {}: {}", 
                        attempt, MAX_RETRY_ATTEMPTS, record.offset(), e.getMessage());
                    
                    if (attempt < MAX_RETRY_ATTEMPTS) {
                        // Exponential backoff
                        try {
                            Thread.sleep((long) Math.pow(2, attempt) * 1000);
                        } catch (InterruptedException ie) {
                            Thread.currentThread().interrupt();
                            return new RecordResult(record, false, ie);
                        }
                    }
                }
            }
            
            // All retries exhausted
            return new RecordResult(record, false, lastException);
            
        }, kafkaProcessorPool);
    }
    
    private void sendToDlt(ConsumerRecord<String, Order> record, Exception exception) {
        try {
            ProducerRecord<String, Order> dltRecord = new ProducerRecord<>(
                record.topic() + ".DLT",
                record.partition(),
                record.key(),
                record.value()
            );
            
            // Add error headers
            dltRecord.headers().add("original-topic", record.topic().getBytes());
            dltRecord.headers().add("original-offset", 
                String.valueOf(record.offset()).getBytes());
            dltRecord.headers().add("error-message", 
                exception.getMessage().getBytes());
            dltRecord.headers().add("error-class", 
                exception.getClass().getName().getBytes());
            
            kafkaTemplate.send(dltRecord);
            log.info("Sent to DLT: offset={}", record.offset());
            
        } catch (Exception e) {
            log.error("Failed to send to DLT", e);
        }
    }
    
    @Data
    @AllArgsConstructor
    private static class RecordResult {
        private ConsumerRecord<String, Order> record;
        private boolean success;
        private Exception exception;
    }
}

This implementation:

• Retries each record up to 3 times with exponential backoff
• Processes all records concurrently in thread pool
• Only sends to DLT after all retries exhausted
• Acknowledges batch after all processing complete

4.4.5. Trade-offs Analysis

Approach 1: All-or-Nothing (Section 4.4.2)

✅ Pros:

• Simple to implement
• No message loss guaranteed
• Maintains batch ordering

❌ Cons:

• One failure reprocesses entire batch
• Lower throughput with frequent failures
• Can cause duplicate processing of successful messages

Best for: Low failure rates, strict ordering requirements

Approach 2: Finegrained with DLT (Section 4.4.3)

✅ Pros:

• Higher throughput (no reprocessing successful messages)
• Failed messages isolated to DLT
• Better resource utilization

❌ Cons:

• More complex code
• Potential ordering issues if replaying from DLT
• Need to monitor and replay DLT messages

Best for: High throughput requirements, acceptable to lose ordering for failed messages

Approach 3: Retry with DLT (Section 4.4.4)

✅ Pros:

• Handles transient failures automatically
• Best throughput/reliability balance
• Reduces DLT volume

❌ Cons:

• Most complex implementation
• Increased processing latency during retries
• Thread pool can get saturated during failures

Best for: Production systems with transient failures (database timeouts, rate limits)

Throughput Comparison:

Configuration recommendations:

# For high throughput with async processing
spring:
  kafka:
    consumer:
      max-poll-records: 500  # Larger batches
      fetch-min-size: 1024   # Wait for more data
      fetch-max-wait: 500    # But don't wait too long
    listener:
      ack-mode: manual_immediate
      concurrency: 3  # Multiple container threads

# Thread pool sizing
kafka:
  processor:
    core-threads: 20
    max-threads: 100
    queue-capacity: 2000

Thread pool sizing guidelines:

• Core threads: 2-3x number of Kafka consumer threads
• Max threads: Based on downstream service capacity
• Queue capacity: max-poll-records × concurrency × 2

Example: If max-poll-records=500, concurrency=3, queue should be ~3000.

4.4.6. Monitoring and Observability

When using async batch processing, add metrics to track:

@Service
public class MonitoredAsyncBatchListener {
    
    private final MeterRegistry meterRegistry;
    private final Counter processedCounter;
    private final Counter failedCounter;
    private final Timer processingTimer;
    
    public MonitoredAsyncBatchListener(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.processedCounter = Counter.builder("kafka.messages.processed")
            .tag("type", "batch-async")
            .register(meterRegistry);
        this.failedCounter = Counter.builder("kafka.messages.failed")
            .tag("type", "batch-async")
            .register(meterRegistry);
        this.processingTimer = Timer.builder("kafka.batch.processing.time")
            .register(meterRegistry);
    }
    
    @KafkaListener(topics = "orders", containerFactory = "batchAsyncFactory")
    public void processBatch(
            List<ConsumerRecord<String, Order>> records,
            Acknowledgment acknowledgment) {
        
        processingTimer.record(() -> {
            // Your processing logic here
            processBatchInternal(records, acknowledgment);
        });
    }
    
    private void processBatchInternal(
            List<ConsumerRecord<String, Order>> records,
            Acknowledgment acknowledgment) {
        // Track batch size
        meterRegistry.gauge("kafka.batch.size", records.size());
        
        // Process batch...
        // Update counters based on results
        processedCounter.increment(successCount);
        failedCounter.increment(failureCount);
    }
}

Key metrics to monitor:

• Batch processing time
• Success/failure rates
• Thread pool utilization
• DLT message count
• Consumer lag

5. Deserialization Error Handling

5.1. Using ErrorHandlingDeserializer

Deserialization errors are tricky because they happen before your @KafkaListener method is invoked. Without special handling, a single bad message stops the entire consumer.

Solution: Wrap your deserializer with ErrorHandlingDeserializer.

Configuration:

@Bean
public ConsumerFactory<String, Order> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    
    // Wrap the actual deserializer
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
    
    return new DefaultKafkaConsumerFactory<>(props);
}

Or with properties:

spring:
  kafka:
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.trusted.packages: "*"

5.2. Handling Deserialization Failures

With ErrorHandlingDeserializer, failed records arrive with null value and exception details in headers.

Record listener:

@KafkaListener(topics = "orders")
public void listen(ConsumerRecord<String, Order> record) {
    if (record.value() == null) {
        // Deserialization failed
        throw new DeserializationException("Cannot deserialize", null, null);
    }
    
    processOrder(record.value());
}

Batch listener:

@KafkaListener(topics = "orders", containerFactory = "batchFactory")
public void listenBatch(List<ConsumerRecord<String, Order>> records) {
    for (ConsumerRecord<String, Order> record : records) {
        if (record.value() == null) {
            // Deserialization failed → send to DLT
            throw new BatchListenerFailedException(
                "Deserialization failed", 
                record
            );
        }
        processOrder(record.value());
    }
}

Important: For batch listeners, you must manually check for null values. There’s no automatic detection like with record listeners.

5.3. Conversion Errors

Starting with version 2.8, batch listeners can handle conversion errors when using a MessageConverter.

Example:

@KafkaListener(topics = "orders")
public void listen(
        List<Order> orders,
        @Header(KafkaHeaders.CONVERSION_FAILURES) List<ConversionException> exceptions) {
    
    for (int i = 0; i < orders.size(); i++) {
        if (orders.get(i) == null && exceptions.get(i) != null) {
            throw new BatchListenerFailedException(
                "Conversion error", 
                exceptions.get(i), 
                i
            );
        }
        processOrder(orders.get(i));
    }
}

6. Backoff Strategies

6.1. Fixed Backoff

Retries at fixed intervals.

// 3 retries with 1 second between each
new FixedBackOff(1000L, 2L)

6.2. Exponential Backoff

Retries with exponentially increasing delays.

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<String, Order> template) {
    ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(6);
    backOff.setInitialInterval(1000L);    // Start with 1 second
    backOff.setMultiplier(2.0);           // Double each time
    backOff.setMaxInterval(10000L);       // Cap at 10 seconds
    
    // Retry after: 1s, 2s, 4s, 8s, 10s, 10s
    
    DeadLetterPublishingRecoverer recoverer = 
        new DeadLetterPublishingRecoverer(template);
    
    return new DefaultErrorHandler(recoverer, backOff);
}

6.3. Custom Backoff Based on Exception

Different backoff strategies for different errors:

DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, defaultBackOff);

handler.setBackOffFunction((record, exception) -> {
    if (exception.getCause() instanceof TransientException) {
        // Retry quickly for transient errors
        return new FixedBackOff(500L, 5L);
    } else if (exception.getCause() instanceof RateLimitException) {
        // Wait longer for rate limit errors
        return new FixedBackOff(5000L, 3L);
    }
    // Use default backoff for other cases
    return null;
});

7. Common Pitfalls and Best Practices

7.1. Losing Messages with Async Processing

Problem: When processing messages asynchronously with a custom thread pool, the @KafkaListener method returns immediately. If using BATCH or RECORD ack mode, the offset is committed before async processing completes.

// DON'T DO THIS
@KafkaListener(topics = "orders")
public void listen(Order order) {
    executorService.submit(() -> {
        // Process asynchronously
        processOrder(order);  // If this fails, message is lost
    });
    // Method returns → offset committed before processing!
}

Solution: Use MANUAL_IMMEDIATE ack mode:

@KafkaListener(
    topics = "orders",
    containerFactory = "manualFactory"
)
public void listen(Order order, Acknowledgment ack) {
    executorService.submit(() -> {
        try {
            processOrder(order);
            ack.acknowledge();  // Commit only after success
        } catch (Exception e) {
            log.error("Failed", e);
            // Don't acknowledge → message will be redelivered
        }
    });
}

7.2. Message Duplicates After Restart

Problem: Consumer crashes after processing but before committing offset → message gets reprocessed.

This is normal with at-least-once delivery. Your processing logic should be idempotent.

Solution — Idempotent processing:

@KafkaListener(topics = "orders")
public void listen(Order order) {
    // Use database constraints or unique checks
    try {
        orderRepository.insertIfNotExists(order);
    } catch (DuplicateKeyException e) {
        log.info("Order {} already processed, skipping", order.getId());
        // Don't throw exception → acknowledge and move on
    }
}

Or track processed message IDs in a separate table:

@Transactional
public void processOrder(Order order) {
    if (processedMessageRepository.exists(order.getMessageId())) {
        log.info("Already processed message {}", order.getMessageId());
        return;
    }
    
    // Process the order
    orderService.createOrder(order);
    
    // Mark as processed
    processedMessageRepository.save(new ProcessedMessage(order.getMessageId()));
}

7.3. Forgetting to Configure DLT Topics

Problem: DefaultErrorHandler tries to send to DLT, but topic doesn’t exist → error on error.

Solution: Create DLT topics ahead of time with appropriate partition count:

# Original topic has 10 partitions
kafka-topics.sh --create \
  --topic orders.DLT \
  --partitions 10 \
  --replication-factor 3

Or configure to create dynamically:

@Bean
public NewTopic ordersDlt() {
    return TopicBuilder
        .name("orders.DLT")
        .partitions(10)
        .replicas(3)
        .build();
}

7.4. Not Setting Immediate Stop for Long Processing

Problem: Graceful shutdown waits for entire batch to complete. If each message takes 30 seconds, shutdown can take minutes.

Solution:

spring:
  kafka:
    listener:
      ack-mode: RECORD
      immediate-stop: true  # Stop after current record

7.5. Batch Listener without BatchListenerFailedException

Problem: Throwing generic exception from batch listener retries the entire batch, can’t identify which record failed.

Bad:

@KafkaListener(topics = "orders", containerFactory = "batchFactory")
public void processBatch(List<Order> orders) {
    for (Order order : orders) {
        processOrder(order);  // Exception loses context
    }
}

Good:

@KafkaListener(topics = "orders", containerFactory = "batchFactory")
public void processBatch(List<Order> orders) {
    for (int i = 0; i < orders.size(); i++) {
        try {
            processOrder(orders.get(i));
        } catch (Exception e) {
            throw new BatchListenerFailedException("Failed", e, i);
        }
    }
}

8. Production Configuration Example

Here’s a solid production-ready configuration combining everything:

@Configuration
public class KafkaErrorHandlingConfig {
    
    @Bean
    public ConsumerFactory<String, Order> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        
        // Wrap deserializer for error handling
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
        props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);
        props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.example.model");
        
        // Disable auto-commit (Spring Kafka manages it)
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        
        return new DefaultKafkaConsumerFactory<>(props);
    }
    
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
            ConsumerFactory<String, Order> consumerFactory,
            KafkaTemplate<String, Order> kafkaTemplate) {
        
        ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
            new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        
        // Commit after each record for minimal duplicates
        factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD);
        
        // Stop immediately on shutdown
        factory.getContainerProperties().setStopImmediate(true);
        
        // Configure DLT error handler
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(
            kafkaTemplate,
            (record, ex) -> {
                // Custom DLT routing based on exception
                if (ex.getCause() instanceof ValidationException) {
                    return new TopicPartition(record.topic() + ".validation.DLT", -1);
                }
                return new TopicPartition(record.topic() + ".DLT", record.partition());
            }
        );
        
        // Exponential backoff with max retries
        ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(5);
        backOff.setInitialInterval(1000L);
        backOff.setMultiplier(2.0);
        backOff.setMaxInterval(30000L);
        
        DefaultErrorHandler errorHandler = new DefaultErrorHandler(recoverer, backOff);
        
        // Don't retry validation errors
        errorHandler.addNotRetryableExceptions(ValidationException.class);
        
        // Reset retry state if exception type changes
        errorHandler.setResetStateOnExceptionChange(true);
        
        factory.setCommonErrorHandler(errorHandler);
        
        return factory;
    }
    
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Order> batchFactory(
            ConsumerFactory<String, Order> consumerFactory,
            KafkaTemplate<String, Order> kafkaTemplate) {
        
        ConcurrentKafkaListenerContainerFactory<String, Order> factory = 
            new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.setBatchListener(true);
        
        // Similar error handler for batch processing
        DeadLetterPublishingRecoverer recoverer = 
            new DeadLetterPublishingRecoverer(kafkaTemplate);
        
        DefaultErrorHandler errorHandler = new DefaultErrorHandler(
            recoverer,
            new FixedBackOff(1000L, 3L)
        );
        
        factory.setCommonErrorHandler(errorHandler);
        
        return factory;
    }
    
    // Create DLT topics automatically
    @Bean
    public NewTopic ordersDlt() {
        return TopicBuilder
            .name("orders.DLT")
            .partitions(10)
            .replicas(3)
            .build();
    }
    
    @Bean
    public NewTopic ordersValidationDlt() {
        return TopicBuilder
            .name("orders.validation.DLT")
            .partitions(10)
            .replicas(3)
            .build();
    }
}

application.yml:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-service
      enable-auto-commit: false
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.trusted.packages: com.example.model
    listener:
      ack-mode: record
      immediate-stop: true

logging:
  level:
    org.springframework.kafka: DEBUG

9. Final Thoughts

Error handling in Spring Kafka requires understanding how offsets, acknowledgments, retries, and dead letter topics work together.

Key takeaways:

• Use RECORD ack mode for minimal duplicates, BATCH for higher throughput
• Always configure DeadLetterPublishingRecoverer for production
• Make your processing logic idempotent to handle duplicates
• Use ErrorHandlingDeserializer to prevent deserialization errors from blocking consumers
• For batch listeners, always throw BatchListenerFailedException with the failed record
• Use MANUAL_IMMEDIATE ack mode for async processing with custom thread pools
• For maximum throughput, process batches asynchronously with CompletableFuture — track all tasks before acknowledging
• Choose async batch processing strategy based on requirements: all-or-nothing (simple), finegrained (high throughput), or retry-based (best balance)
• Configure appropriate backoff strategies based on your error types
• Create DLT topics ahead of time with correct partition counts
• Size thread pools based on consumer concurrency: core threads = 2-3x consumer threads, queue = max-poll-records × concurrency × 2
• Monitor async processing metrics: processing time, success/failure rates, thread pool utilization, consumer lag

Spring Kafka’s error handling is flexible enough to handle most real-world scenarios. The key is understanding the trade-offs between throughput, latency, and message delivery guarantees. When pushing for maximum throughput with async batch processing, carefully balance implementation complexity against reliability requirements.

10. References

• https://docs.spring.io/spring-kafka/reference/kafka/annotation-error-handling.html
• https://piotrminkowski.com/2024/03/11/kafka-offset-with-spring-boot/
• https://runebook.dev/en/docs/spring_boot/application-properties/application-properties.integration.spring.kafka.listener.async-acks

At that point, every rule change becomes a deployment. A product manager asks to add a region, marketing wants to test a different eligibility path, sales needs an enterprise override, and engineering ends up threading feature flags through a method that was supposed to be a clean predicate. The code still runs, but it no longer represents a stable algorithm. It represents mutable business policy trapped inside compiled Java.

A Boolean expression evaluator moves that policy into data. The application still controls which variables are available and how those variables are computed, but the rule itself can be stored, validated, versioned, cached, and evaluated at runtime.

This post builds a small Boolean expression evaluator in Java. It supports variables, AND, OR, NOT, and parentheses. The implementation uses a tokenizer, a recursive descent parser, an abstract syntax tree, and an evaluator with real short-circuit behavior.

1. The Problem with Hardcoded Rules

A simple eligibility check is a good fit for normal Java code.

public boolean checkEligibility(User user) {
    return user.isActive() && user.getAge() >= 18;
}

This is readable, fast, and hard to misuse. The rule is close to the domain model, the compiler checks the method, and tests can cover the few branches that exist.

Now consider a more realistic rule.

public boolean checkEligibility(User user) {
    return user.isActive()
            && ((user.getAge() >= 18 && user.hasVerifiedEmail())
                || (user.getAge() >= 13 && user.hasParentConsent()))
            && ("US".equals(user.getRegion())
                || "CA".equals(user.getRegion())
                || "UK".equals(user.getRegion()))
            && !user.isBlacklisted()
            && (user.isPremium()
                || (user.isTrial() && user.getDaysActive() <= 30));
}

The method is still correct, but it now combines several concerns:

• extracting facts from User
• encoding business policy
• controlling Boolean operator precedence
• handling future rule variants through code changes

The immediate cost is readability. The larger cost is operational. Adding a new region, changing the trial window, removing email verification for an experiment, or introducing a tenant-specific override requires a code change, a review, a test cycle, and a deployment. If the system serves multiple tenants, this pattern usually turns into duplicated methods, nested feature flags, or rule-specific branches scattered across services.

A runtime evaluator separates the stable part from the volatile part. Java code computes named facts, while a stored expression decides how those facts are combined.

2. Rules as Runtime Data

Instead of compiling the whole rule into Java, store the Boolean expression as a string.

String eligibilityRule =
        "isActive AND "
      + "((age_gte_18 AND hasVerifiedEmail) OR (age_gte_13 AND hasParentConsent)) AND "
      + "(region_US OR region_CA OR region_UK) AND "
      + "NOT isBlacklisted AND "
      + "(isPremium OR (isTrial AND daysActive_lte_30))";

The application builds a context map from the domain object.

public Map<String, Boolean> buildContext(User user) {
    Map<String, Boolean> context = new HashMap<>();

    context.put("isActive", user.isActive());
    context.put("age_gte_18", user.getAge() >= 18);
    context.put("age_gte_13", user.getAge() >= 13);
    context.put("hasVerifiedEmail", user.hasVerifiedEmail());
    context.put("hasParentConsent", user.hasParentConsent());
    context.put("region_US", "US".equals(user.getRegion()));
    context.put("region_CA", "CA".equals(user.getRegion()));
    context.put("region_UK", "UK".equals(user.getRegion()));
    context.put("isBlacklisted", user.isBlacklisted());
    context.put("isPremium", user.isPremium());
    context.put("isTrial", user.isTrial());
    context.put("daysActive_lte_30", user.getDaysActive() <= 30);

    return context;
}

The service method becomes small again.

public boolean checkEligibility(User user, String expression) {
    Map<String, Boolean> context = buildContext(user);
    return BooleanEvaluator.evaluate(expression, context);
}

This design makes one boundary explicit: the evaluator does not know about User, regions, subscriptions, trials, or blacklists. It only knows how to evaluate a Boolean expression against named Boolean values. That constraint keeps the evaluator reusable and reduces the risk of turning it into an unbounded scripting layer.

The trade-off is also clear. You lose compile-time validation for the expression string, so you must replace it with save-time validation, good error messages, safe rollout, and monitoring. Runtime rules are useful when policy changes frequently, but they need guardrails because bad rules fail at runtime unless you catch them earlier.

3. Use Cases

Boolean expression evaluators are useful when a system needs configurable decision logic but does not need a full rule engine.

Feature flags are a common example.

String rule = "isEmployee AND country_US AND NOT blockedUser";

if (BooleanEvaluator.evaluate(rule, context)) {
    showFeature();
}

Authorization checks are another natural fit, especially when policies differ between tenants or environments.

String rule = "hasRole_admin OR (hasRole_manager AND department_engineering)";

if (BooleanEvaluator.evaluate(rule, context)) {
    allowAccess();
}

Notification systems can use the same pattern to decide whether an event should trigger a message.

String rule = "order_total_gt_1000 AND (vipCustomer OR firstPurchase)";

if (BooleanEvaluator.evaluate(rule, context)) {
    sendOffer();
}

The evaluator should remain narrow. If the business needs numeric comparison, string matching, list membership, date arithmetic, or custom functions, those features can be added, but each one expands the language surface and increases the cost of validation, testing, and abuse prevention.

4. Evaluator Architecture

The evaluator has three stages.

Input:
  "NOT (A AND B) OR C"

Tokenizer:
  [NOT, LPAREN, A, AND, B, RPAREN, OR, C, EOF]

Parser:
  builds an abstract syntax tree

AST:
        OR
       /  \
     NOT   C
      |
     AND
    /   \
   A     B

Evaluator:
  walks the tree and reads values from the context map

The tokenizer turns raw characters into tokens. The parser turns tokens into an abstract syntax tree, usually called an AST. The evaluator walks that tree and computes the final Boolean result.

This architecture gives each component one job. Tokenization errors are about invalid characters. Parser errors are about invalid grammar. Evaluation errors are about missing variables or unsupported operators. That separation matters in production because different failures need different responses: invalid syntax should block rule activation, while missing variables usually indicate a mismatch between the rule catalog and the application’s context builder.

The grammar is small.

expression    := orExpression
orExpression  := andExpression ("OR" andExpression)*
andExpression := notExpression ("AND" notExpression)*
notExpression := "NOT" notExpression | primary
primary       := VARIABLE | "(" expression ")"

This gives NOT the highest precedence, followed by AND, then OR. Parentheses override that order.

5. Implementation

The implementation below uses a hand-written recursive descent parser. For this grammar, that is simpler than introducing a parser generator, and it makes precedence easy to inspect in code.

5.1. Token Model

Each token has a type, value, and source position. Position is not needed for evaluation, but it is useful for error messages.

enum TokenType {
    VARIABLE,
    AND,
    OR,
    NOT,
    LPAREN,
    RPAREN,
    EOF
}

final class Token {
    final TokenType type;
    final String value;
    final int position;

    Token(TokenType type, String value, int position) {
        this.type = type;
        this.value = value;
        this.position = position;
    }

    @Override
    public String toString() {
        return value == null ? type.name() : type + "(" + value + ")";
    }
}

The language reserves AND, OR, and NOT. Variable names are case-sensitive in this implementation, while keywords are case-insensitive.

5.2. Parse Exception

A runtime expression language needs precise errors. Returning only Invalid expression is cheap for the implementation, but expensive for whoever has to debug a rejected rule.

final class ExpressionParseException extends RuntimeException {
    ExpressionParseException(String message, int position, String expression) {
        super(format(message, position, expression));
    }

    private static String format(String message, int position, String expression) {
        return message + " at position " + position
                + System.lineSeparator()
                + expression
                + System.lineSeparator()
                + " ".repeat(Math.max(0, position)) + "^";
    }
}

For example, A AND (B OR) can point directly to the token position where another operand was expected.

5.3. Tokenizer

The tokenizer scans the input once. It skips whitespace, emits parentheses, reads identifiers, and maps reserved keywords to operator tokens.

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Map;

final class Tokenizer {
    private static final Map<String, TokenType> KEYWORDS = Map.of(
            "AND", TokenType.AND,
            "OR", TokenType.OR,
            "NOT", TokenType.NOT
    );

    private Tokenizer() {
    }

    static List<Token> tokenize(String input) {
        if (input == null) {
            throw new IllegalArgumentException("Expression must not be null");
        }

        List<Token> tokens = new ArrayList<>();
        int pos = 0;

        while (pos < input.length()) {
            char ch = input.charAt(pos);

            if (Character.isWhitespace(ch)) {
                pos++;
                continue;
            }

            if (ch == '(') {
                tokens.add(new Token(TokenType.LPAREN, "(", pos));
                pos++;
                continue;
            }

            if (ch == ')') {
                tokens.add(new Token(TokenType.RPAREN, ")", pos));
                pos++;
                continue;
            }

            if (Character.isLetter(ch)) {
                int start = pos;
                StringBuilder builder = new StringBuilder();

                while (pos < input.length()) {
                    char current = input.charAt(pos);

                    if (Character.isLetterOrDigit(current) || current == '_') {
                        builder.append(current);
                        pos++;
                    } else {
                        break;
                    }
                }

                String rawValue = builder.toString();
                String upperValue = rawValue.toUpperCase(Locale.ROOT);
                TokenType type = KEYWORDS.getOrDefault(upperValue, TokenType.VARIABLE);
                String tokenValue = type == TokenType.VARIABLE ? rawValue : upperValue;

                tokens.add(new Token(type, tokenValue, start));
                continue;
            }

            throw new ExpressionParseException(
                    "Unexpected character '" + ch + "'",
                    pos,
                    input
            );
        }

        tokens.add(new Token(TokenType.EOF, null, input.length()));
        return tokens;
    }
}