Why pods cannot reach Docker Compose services after migrating K3s from Flannel to Cilium

What this document claims, and what it does not

It claims: we ran a version combination that Cilium's published compatibility documentation did not list at the time of this deployment, and we found no published account of this exact combination at the time of writing — our own search being dated and necessarily limited. We are reporting what happened, on a production installation. The combination is:

It does not claim: that we are first at anything, that this combination is supported, that anyone should copy it, or that Cilium works on Kubernetes 1.36 in general. Cilium 1.19.6 is not upstream-tested against 1.36. What follows is local evidence from one estate, not certification and not a recommendation. We ran it because the alternative — a release candidate CNI in production — was worse, and the risk was accepted explicitly and in writing.

The part worth reading is not the migration. Flannel-to-Cilium on K3s is thoroughly documented and we learned nothing new about it. The part worth reading is a failure mode that only exists when these three pieces meet, whose symptom points nowhere near its cause, and which we could not find connected anywhere in the documentation of either side.

The estate

A single node runs two worlds that must talk to each other:

• Docker Compose owns the stateful services — several PostgreSQL instances, a cache, a message broker. Referred to below by role labels: <compose-db-catalog>, <compose-db-communications>, <compose-cache>, <compose-broker>.
• Kubernetes (K3s) owns a growing set of stateless workloads. They reach the Compose services through Kubernetes Service objects whose EndpointSlice points at the Compose container addresses — a deliberate migration bridge, so that workloads speak Kubernetes names on both sides of the move.

Docker's FORWARD policy is DROP, so the bridge was made explicit long ago: a single canonical, versioned, idempotent script owns a narrow allowlist in DOCKER-USER — pod CIDR to one backend address, one port, per dependency. That script had been the boundary for months and was believed to be the whole boundary.

It was not.

The symptom

After the CNI swap, every workload came up and died the same way: timeouts creating its database connection pool. From inside a pod:

So: Service translation works from the host and from Compose, DNS works, routing works — and only pod-originated traffic to a Compose-backed Service fails. Nothing logged an error. The CNI reported no drops. The rules were all present and correct: the service chains carried the right translation, the allowlist carried the right exceptions, the route to the Compose bridge resolved correctly for a forwarded packet from the pod address.

A packet capture showed the SYN leaving the pod interface, retransmitting, and never arriving at the Compose bridge. Counters on the allowlist rule and on the CNI's masquerade rule stayed flat. The packet was disappearing between two places where everything looked right.

The mechanism

Two facts, each unremarkable alone.

Fact one — Docker Engine 28/29 protects unpublished container addresses in the raw table. For every container port that is not published, Docker installs a rule of the shape:

-t raw -A PREROUTING -d <container-address>/32 ! -i <compose-bridge> -j DROP

On the relevant IPv4 ingress path, the raw/PREROUTING hook is evaluated before connection tracking, before nat, and therefore long before DOCKER-USER, which lives in filter/FORWARD. The intent is sound — a container's address should not be reachable by routing around the bridge — and this is documented behaviour.

Fact two — where the Service translation happens depends on the CNI.

With an iptables kube-proxy and a CNI that does not translate services itself, a packet from a pod to a ClusterIP is still addressed to the ClusterIP when it reaches raw/PREROUTING. Docker's rule matches on the container address. It does not match. The packet proceeds, gets DNAT'd in nat/PREROUTING, traverses filter/FORWARD where the allowlist accepts it, and is masqueraded on the way out. The boundary really is DOCKER-USER.

With Cilium, even with kubeProxyReplacement: false, ClusterIP translation for pod-originated traffic happens in the eBPF datapath, before the packet enters netfilter. So the packet arrives at raw/PREROUTING already addressed to the Compose container, on a lxc* interface — which is not the Compose bridge.

That is precisely the shape Docker's rule exists to drop. And it drops it, in the first table, before any rule anyone wrote for this estate is consulted.

The allowlist was never wrong. It was simply never reached.

kubeProxyReplacement: false is not a datapath guarantee

It is tempting to read kubeProxyReplacement: false as "service translation still happens in iptables, therefore netfilter sees the ClusterIP". That inference is wrong, and it is exactly the inference that kept us looking in the wrong table.

kubeProxyReplacement governs whether Cilium takes over kube-proxy's full role. It does not promise that every service translation goes through netfilter when it is off: Cilium documents that, even with kubeProxyReplacement: false, in-cluster ClusterIP traffic from pods is load-balanced per-packet in the eBPF datapath. Socket-level load balancing (translation at connect() time) is a distinct mode with its own configuration — verify which one is active rather than attributing the behaviour to it by default. What we observed here is the consequence that matters for the boundary: with kubeProxyReplacement: false, pod-originated ClusterIP traffic arrived at raw/PREROUTING already carrying the backend address.

The operational rule we took away: do not reason from the flag; verify where translation happens. Two checks suffice:

• cilium status --verbose and cilium config view | grep -i -e socketlb -e lb show which service paths are handled in BPF;
• the counter method below shows, per probe, whether netfilter ever saw the ClusterIP at all. If the service chains stay flat while a connection is attempted, translation happened before netfilter — whatever the flags suggest.

Why it had been invisible

This failure mode cannot occur while the old CNI is in place, because the precondition — a packet already carrying the backend address at raw time — never happens. It also cannot be found by reading configuration: every rule involved is correct, in the right table, in the right order, doing exactly what it says. The interaction only exists at the seam.

How we proved it

The diagnostic that settled it is worth more than the fix, because it is general: compare counters on the two rules that represent the competing explanations, across a single probe.

# before
iptables -t raw  -L PREROUTING -v -n -x | grep 'DROP.*<compose-db-communications>'
iptables -t nat  -L <service-chain>     -v -n -x | grep KUBE-SEP

# one connection attempt from inside the pod
kubectl exec -n <ns> <pod> -- bash -c 'timeout 6 bash -c "echo > /dev/tcp/<clusterip>/<port>"'

# after

Result: the Docker raw DROP counter moved by +8; the service-endpoint counter moved by +0. One measurement, two numbers, no interpretation required. Everything else we had been looking at — routes, masquerade rules, policy routing tables, the CNI's own drop feed — was consistent with both explanations and therefore discriminated nothing.

Three practical notes, each of which cost us time:

• Probe with bash, not sh. /dev/tcp is a bash builtin; under sh the probe reports failure for every destination regardless of connectivity.
• Watch out for terminating ACCEPTs when reading counters. An ACCEPT in a user-defined chain ends traversal of that hook, so a downstream rule's counter staying flat does not mean the packet was dropped there.
• Do not read an exit code through a pipe. cmd | sed reports sed's status. We produced one wrong measurement this way and had to redo it.

Reading a disappearing packet from the counters

The pattern that makes this failure disorienting is: the connection times out, the service-endpoint counters (KUBE-SVC/KUBE-SEP) never move, and the CNI's drop feed is silent. Every tool is telling the truth; they watch different layers.

• KUBE-SEP flat does not mean "no traffic". It means netfilter's nat hook never saw a packet still addressed to the ClusterIP — which is exactly what happens when translation already occurred in the eBPF datapath.
• A silent CNI drop feed does not mean "no drop". Cilium reports drops taken by its own datapath; a packet discarded by a netfilter raw rule is invisible to it.
• A missing conntrack entry narrows it further. raw runs before connection tracking, so a drop there leaves no conntrack trace at all; a drop in filter would leave one.
• raw is the hook that runs before connection tracking (hook priority raw −300, conntrack −200, mangle −150 — so mangle runs after conntrack and cannot explain a missing conntrack entry). When every later stage shows nothing, count in raw first.

The procedure stays the one above: pick the two rules that represent the two competing explanations, read both counters, run one probe, read again. One attempt, two numbers. In our case the raw DROP counter advanced by the retransmission count while KUBE-SEP stayed at zero — the packet was being judged one table earlier than the boundary we owned.

The fix

Extend the canonical firewall owner to the table where the decision is actually taken. Concretely: a governed chain in raw, jumped from raw/PREROUTING after the CNI's own feeder and before Docker's DROP rules, containing exactly the same narrow exceptions the filter allowlist already expressed — pod CIDR to one backend address, one literal port, per dependency.

Properties that matter more than the rules themselves:

• Addresses are derived at run time from the container runtime, never written down. A literal address in a firewall rule outlives the container it was copied from and silently becomes an exception for whoever inherits it.
• Position is verified, not assumed. An allow placed after the DROP allows nothing — and every probe would still pass on an existing connection-tracking entry, so the check must compare the index of our jump against the index of the first Docker DROP for our backends, not merely assert that both exist. Docker inserts new rules at the head when containers start, so a jump that was correctly placed yesterday can be overtaken today.
• Idempotent purge of only what we own, by ownership tag, so stale rules built with old addresses cannot accumulate.
• Negative tests are part of the contract: an unlisted port and an unlisted destination must remain denied after the fix, and we verify that they do.

This is not a bypass of Docker's isolation. It is the same explicit exception the estate already granted in filter, expressed in the table where the packet is actually judged. Per-workload authorisation is then narrowed further by NetworkPolicy.

Production status after the migration

This was a production migration, not an isolated lab exercise. Cilium 1.19.6 and Hubble now run on a Kubernetes 1.36.2 node that continues to coexist with a Docker Compose estate hosting stateful dependencies.

At the latest post-migration verification, the node was Ready; Cilium reported 49 healthy controllers and one reachable cluster node; nine Kubernetes pods were running, with seven ready Cilium endpoints. A service-to-service health probe across the Compose-to-Kubernetes boundary returned HTTP 200, the public HTTP endpoints remained available, and the Compose estate reported 66 running containers with no unhealthy container.

The migration corrected three real transition defects: CNI paths had to follow the container runtime rather than an assumed K3s layout; a residual Flannel VXLAN device had to be removed safely before Cilium could own the tunnel; and Cilium's earlier eBPF service translation exposed Compose backend addresses to Docker's raw-table policy before DOCKER-USER could see the flow.

The final bridge is deliberately narrow: a host-level raw-table chain permits only declared pod-to-Compose dependencies, by backend role and TCP port. Addresses are derived from the live containers, owned rules are rebuilt idempotently, rule ordering is checked against Docker's drops, and negative probes verify that undeclared destinations and ports remain unavailable.

The rollback path was rehearsed in dry-run and remains deliberately manual. It supports policy-only recovery and a full Cilium-to-Flannel restoration with vendor state cleanup, CNI/eBPF/CRD removal and mandatory post-recovery checks. It was not invoked in production because the Cilium target state recovered and remained healthy; this report therefore makes no claim about a measured production rollback duration.

Production deviations

The first production failure was our own assumption

The first Cilium rollout did not fail because of an undocumented product bug. It failed because we configured CNI paths from an assumption about the K3s layout instead of deriving them from the container runtime that would consume them after Flannel was disabled.

The result was a cluster without a usable pod datapath: workloads could start, but could not obtain the network configuration they needed. The correction was not another hard-coded path. We changed the preflight and deployment contract so that the expected paths are derived from the runtime that actually decides them.

This was an important distinction. Mutation tests had shown that the old gate enforced its assertion. They had not shown that the assertion's premise was true. A test can be alive, strict and still protect the wrong model.

Further deviations recorded during the window

• A residual Flannel VXLAN interface still held the tunnel UDP port required by Cilium. We removed it only after verifying that it was not carrying forwarding state, rather than deleting a live network device blindly.
• ClusterIP translation occurred in Cilium's eBPF datapath before netfilter. Docker therefore evaluated the translated Compose backend in raw/PREROUTING, before the existing DOCKER-USER allowances could see the flow.
• A pod can remain Running while still belonging to obsolete network state. The final convergence criterion was therefore set equality between all non-host-network pods and Cilium endpoints, not a remembered list of workloads.

What is not novel here

Stated plainly, so nobody has to discover it in a comment thread:

• Migrating K3s from Flannel to Cilium is well documented, and we learned nothing new about it.
• The K3s CNI directory layout, and Flannel's tunnel device surviving flannel-backend: none while holding the tunnel UDP port, are both known. Our contribution is not having discovered them: it is having verified them against the runtime that actually decides, and folded them into the operating contract. The account of what went wrong is in Production deviations above, and is not repeated here.
• Docker's raw protection is documented as a Docker behaviour. What we found undocumented is its interaction with a CNI that translates services before netfilter.

The claim is narrow on purpose: this specific interaction, at these versions, in a hybrid Compose/Kubernetes estate. If it is documented elsewhere and we missed it, we would rather be corrected than quoted.

Reproduction

Minimal shape, no product involved:

1. One host running Docker Engine 29 and a Kubernetes distribution, with at least one Compose service that publishes no ports.
2. A Kubernetes Service whose EndpointSlice points at that container's address.
3. With an iptables kube-proxy and a non-pre-translating CNI, plus an allowlist in DOCKER-USER: a pod reaches the Service.
4. Install a CNI that translates ClusterIPs in eBPF ahead of netfilter.
5. The same pod now times out, while the host and other Compose containers still reach the Service. The Docker raw DROP counter for that container address advances; the service-endpoint counter does not.

Validation status at the time of writing

Completed and verifiable at observation time:

• the live Flannel-to-Cilium migration on this node;
• diagnosis of the pod-to-Compose failure, by the counter method above;
• the narrowly scoped raw-table boundary, with positive and negative checks performed immediately after the fix;
• a datapath convergence gate that compares pods and endpoints as sets, in both directions, with an adversarial test suite for the gate itself.

Explicitly not completed at observation time:

• per-workload NetworkPolicy authorisation;
• the vendor connectivity suite;
• a measured burn-in window, and any claim of prolonged operational stability.

This document does not claim them, and will not, until they are true and measured.

Caveats

• Single node, single estate, one observation window. We have not tested multi-node, other CNIs with pre-netfilter translation, or Docker 28 versus 29 separately.
• Cilium 1.19.6 on Kubernetes 1.36 is outside the upstream support matrix. Nothing here should be read as evidence that the pairing is safe in general; it is evidence that it ran here, under our own proofs, during one window.
• The fix widens a host-level exception. It is acceptable in this estate because the exceptions are narrow, derived, position-checked and covered by negative tests. Copy the properties, not the rules.

---

Published 2026-07-30 as a field report on work performed 2026-07-29. Every statement above is backed by a recorded evidence entry; the traceability register is internal by design. This page will receive dated addenda as further validation completes (vendor connectivity suite, measured burn-in, per-workload NetworkPolicy) — new evidence, not corrections.