Apache Kafka 4.0 was announced on the eighteenth of March. KRaft is now the only way to manage metadata; ZooKeeper support has been removed entirely.

I have written about this twice already — when KRaft was marked production ready for new clusters in December 2022, and when the bridge release arrived in August 2023. So this is the end of a plan I have been reading release notes about for two and a half years, and the thing worth writing down is not the feature. It is that the plan finished.

The schedule, as promised and as delivered

The 2022 KIP said KRaft would be production ready for new clusters, that a bridge release would follow, that ZooKeeper would be deprecated, and that removal would happen in 4.0.

That is what happened. Production ready in 3.3. Migration in early access in 3.5, alongside the deprecation. General availability of the migration in 3.6. 3.9 as the last version that can perform it. Removal in 4.0.

A cluster still on ZooKeeper today cannot upgrade to 4.0 directly — it has to migrate on 3.9 first, and then upgrade. That is a real constraint with a real deadline attached, and it has been visible for years.

I want to dwell on this because the normal outcome is different. Most deprecations are announced, then quietly extended, then extended again, then carried indefinitely because removing them would break someone who never moved. The compatibility shim becomes permanent, and the codebase keeps two paths forever — which was the original argument for removing dockershim, and the reason that removal also had to actually happen rather than being promised.

Kafka published a schedule with version numbers, kept it, and shipped the removal. That is unusual enough to be worth noticing on its own.

What it took to be able to do it

Three things, and they are the transferable part.

The replacement was ready first and separately. KRaft was production ready two and a half years before ZooKeeper was removed. Nobody was ever asked to migrate to something unfinished.

The migration was reversible while it ran. The dual-write mode kept ZooKeeper current during the transition, so an operator could abandon a migration in progress rather than having to be certain in advance. Most of the reason people do not migrate is not effort — it is that the door only opens one way.

And the timeline was long and published. Two and a half years from "you can choose this" to "you have no choice" is enough for an organisation to fit the work into a normal quarter rather than an emergency.

Take any of the three away and this ends the way deprecations usually end.

What is actually different now

For anyone running Kafka, the operational surface is one system instead of two. No ZooKeeper ensemble with its own quorum arithmetic, its own version matrix, its own monitoring, its own place in the runbook.

The scaling property from the original change still seems to me the most important one. Controller failover used to require loading metadata out of ZooKeeper, so recovery time grew with partition count — the more you used Kafka, the worse it recovered. Metadata as a replicated log with offsets means a new controller starts from what the quorum already has.

4.0 also drops Java 8 and includes early access to queue semantics on top of topics, which is a different discussion and one I will leave alone until it is not early access.

The bit I keep coming back to

I still do not run Kafka in production. I have followed this for two and a half years because it was a well-documented example of a thing I do have to do: remove a dependency from a system that cannot stop.

The reframing at the centre of it was the good part — Kafka's metadata problem looked like state synchronisation until someone modelled it as a log, at which point it became something the project already knew how to do well. The execution around it is the part I would actually copy. Ship the replacement early, keep the old path alive while people move, give them a way back, publish the date, and then honour it.

None of that is technically difficult. It is mostly a willingness to carry two implementations for two years, on purpose, so that the removal at the end is uneventful. Which is exactly why most deprecations never finish.