K3s Distributions Compared: Flannel, Calico, Cilium — CNI Performance and NetworkPolicy Reality

Software Testing That Earns Its Keep: A Field Report from Three Codebases

Evidence-based testing practice from production codebases — property-based invariants, seeded fault injection, contract tests, benchmark-verified performance, and quality gates that are enforced, not aspirational.

k3s Disaster-Recovery-Testing und Edge-Patterns

Umfassende DR-Test- und Edge-spezifische Patterns für k3s — Failover-Validierung, geplante Snapshots, Multi-Cluster-Recovery und RPO/RTO-Planung.

K3s is the most popular lightweight Kubernetes distribution for edge, IoT, ARM, and resource-constrained environments — but "k3s" covers a broad ecosystem of installer tools, CNI plugins, and deployment patterns that behave very differently in production.

This article compares the landscape across four dimensions: distribution options, CNI plugin performance and NetworkPolicy support, ingress and storage, and production considerations like HA, scaling limits, and known pitfalls.

If you found out the hard way that NetworkPolicies are silently ignored with Flannel, this article is for you.

K3s Distribution Options

There are four major ways to run k3s. Each targets a different use case:

k3s Official (Rancher / SUSE)

The reference distribution: a single <100 MB binary that bundles everything. Server nodes run k3s server (control-plane + datastore), agent nodes run k3s agent without control-plane overhead.

curl -sfL https://get.k3s.io | sh -

Single-server mode uses embedded SQLite; HA uses embedded etcd (3+ servers) or an external datastore (PostgreSQL, MySQL, external etcd). Bundled defaults include Flannel (VXLAN), Traefik, CoreDNS, ServiceLB (Klipper), Local Path Provisioner, and a kube-router-based NetworkPolicy controller.

k3sup (ketchup)

A client-side SSH tool by Alex Ellis that bootstraps k3s onto any VM from your local machine. Never logs into the remote — uses SSH key-forwarding.

k3sup install --ip 192.168.1.100 --user root

# Join an agent
k3sup join --ip 192.168.1.101 --server-ip 192.168.1.100 --user root

The Pro version ($25+/mo) adds parallel HA deployment from JSON configs, uninstall, exec, get-config. Best for IaC-style provisioning of bare-metal, EC2, or Raspberry Pi nodes.

k3d

k3s in Docker — creates containerized clusters for local development and CI/CD.

k3d cluster create mycluster --servers 3 --agents 2 --port "8080:80@loadbalancer"

Auto-merges kubeconfig, supports --k3s-arg for any k3s flag, port mappings, registry mounts, and GPU passthrough. Best for testing multi-node behavior on a single machine without provisioning real VMs.

AutoK3s

A management platform (CLI + UI) for deploying k3s across cloud providers (AWS, GCP, Alibaba, Tencent). Supports airgap installation and Helm-based add-on management. Best for multi-cloud deployments with a unified dashboard.

CNI Plugin Deep Dive

The choice of Container Network Interface (CNI) plugin is the most impactful decision for your k3s cluster's performance, security isolation, and operational complexity.

Flannel (Default)

Flannel is the default CNI in k3s. It provides simple overlay networking using VXLAN, host-gw, or WireGuard backends. Minimal overhead, minimal features.

Performance (10GbE testbed, no policies):

• Throughput: ~6.8 Gbps
• Latency: ~75 μs
• Memory: ~50 MB per node

NetworkPolicy support: ✅ With k3s's built-in controller, ❌ Flannel alone

This is the critical distinction that catches most users off guard.

Calico

Calico is the most mature CNI for policy-driven networking. Supports iptables, eBPF, and VPP dataplanes. Full Kubernetes NetworkPolicy + global network policy CRDs.

Performance (10GbE testbed):

Memory: ~120-180 MB per node.

Cilium

Cilium is eBPF-native. This is not optional — Cilium requires kernel ≥5.8 and runs entirely in the eBPF subsystem. The result is the best performance with the deepest observability.

Performance (10GbE testbed):

• Throughput: ~9.2 Gbps
• Latency: ~42 μs
• Policy impact: 2-3%

CiliumNetworkPolicy CRDs extend L3/L4 policy to L7 (HTTP, gRPC, Kafka, DNS). Hubble provides eBPF-native flow observability — no sidecars, no iptables, no overhead.

Memory: ~200 MB per node (higher than Calico, justified by eBPF dataplane).

Canal (Flannel + Calico Policy)

Canal combines Flannel's VXLAN data plane with Calico's Felix policy engine. You get Flannel's simple overlay with Calico's NetworkPolicy enforcement — but without Calico's eBPF or BGP capabilities.

Performance mirrors Flannel (~6.8 Gbps, ~75 μs) with a ~15% policy overhead from iptables.

Comparison Matrix

NetworkPolicy Reality Check

Flannel Does NOT Enforce NetworkPolicies

This is the most common k3s gotcha. Flannel is a networking plugin only — it provides overlay connectivity and nothing else. The official Flannel README states:

"Flannel is focused on networking. For network policy, other projects such as Calico can be used."

If you create a NetworkPolicy in a cluster using Flannel without a policy controller, the object is accepted by the API server but never enforced. Your "deny all ingress" policy denies nothing.

What k3s Ships to Fix This

K3s bundles its own NetworkPolicy controller using kube-router's netpol library. This controller is enabled by default and provides basic Kubernetes NetworkPolicy enforcement on top of Flannel.

To verify it's running:

# Check if the kube-router netpol controller is active
kubectl -n kube-system get pods -l k8s-app=kube-router

# Or check k3s startup flags
journalctl -u k3s | grep disable-network-policy

To disable it (needed when using Calico or Cilium, which ship their own policy engines):

curl -sfL https://get.k3s.io | sh -s - --disable-network-policy

Or in the config file:

# /etc/rancher/k3s/config.yaml
disable-network-policy: true

The Three Scenarios

How to Verify NetworkPolicy Is Working

Don't trust configs — verify:

# Deploy a test pod and a deny-all policy
kubectl run test-pod --image=nginx
kubectl run test-client --image=busybox -- sleep 3600

# Apply a policy that denies all ingress
kubectl create networkpolicy deny-all --pod-selector=app=test-pod \
  --policy-type=Ingress

# This should FAIL if NetworkPolicy is enforced
kubectl exec test-client -- wget -qO- http://test-pod:80
# ❌ If connection succeeds: NetworkPolicy is NOT working
# ✅ If connection times out: NetworkPolicy is enforced

Performance Impact of NetworkPolicies

When policies are enforced, performance depends on the CNI data plane:

• Cilium (eBPF): 2-3% throughput reduction — negligible
• Calico (eBPF): 3-5% — very good
• Calico (iptables): 15-20% — noticeable; scales linearly with rule count
• Flannel + kube-router netpol: modest overhead for basic policies

Changing CNIs in k3s

K3s makes it straightforward to swap CNIs. From the official docs:

Start K3s with --flannel-backend=none and install your CNI of choice. Most CNI plugins come with their own network policy engine, so it is recommended to set --disable-network-policy as well to avoid conflicts.

Switching to Calico

# 1. Install k3s without flannel and built-in netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Calico
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml

For k3s, Calico needs containerIPForwarding: Enabled:

# calico-installation.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    containerIPForwarding: Enabled

Switching to Cilium

# 1. Install k3s without flannel and netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Cilium via Helm
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system

Important: Before running k3s-killall.sh, manually remove Cilium virtual interfaces or you risk losing network connectivity:

ip link delete cilium_host
ip link delete cilium_net
ip link delete cilium_vxlan

Ingress Options

Traefik (k3s Default)

Traefik is deployed automatically by k3s on ports 80/443 as a LoadBalancer service. Customizable via HelmChartConfig:

# /var/lib/rancher/k3s/server/manifests/k3s-traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      type: LoadBalancer
    ports:
      web:
        port: 80
    logs:
      general:
        level: INFO

Traefik v3 supports Gateway API, has built-in ACME (Let's Encrypt) integration, and a diagnostic dashboard. Memory: ~120 MB for 3 replicas.

To disable Traefik during install: --disable=traefik.

Nginx Ingress

The community ingress-nginx reached EOL in March 2026. F5 maintains the successor: NGINX Ingress Controller (NIC) and NGINX Gateway Fabric (Gateway API-native). RKE2 v1.36 flipped to Traefik as default; v1.37 removes ingress-nginx entirely.

If you need Nginx, use the F5-maintained NIC, not the archived community project.

Ingress Comparison

Storage Options with k3s

Local Path Provisioner (Default)

Provisioner: rancher.io/local-path. Access mode: ReadWriteOnce only. Volume binding: WaitForFirstConsumer. No replication, no shared access, node affinity required. Best for dev, single-node, or ephemeral data.

Longhorn

CNCF Incubating (graduated in 2026). Distributed block storage with synchronous replication, snapshots, S3 backups, and a built-in web UI. Memory: ~300 MB per storage node. Minimum 3 nodes for HA. Best for production block storage and databases.

Rook-Ceph

CNCF Graduated. Block (RBD), Filesystem (CephFS), and Object (RGW) in one. On k3s, requires overriding the kubelet path:

helm install rook-ceph rook-release/rook-ceph \
  --set csi.kubeletDirPath=/var/lib/rancher/k3s/agent/kubelet

Memory: 1.5-3 GB per storage node (significantly higher than Longhorn). Minimum 3 nodes with dedicated raw disks. Best for enterprise storage with RWX requirements.

Storage Decision Guide

Production Considerations

HA: Embedded etcd vs External Datastore

Resource Profiling

Official k3s benchmarks (Intel 8375C):

On Raspberry Pi 4: server + workload uses ~30% core, ~1600 MB RAM. Agent only: ~10% core, ~268 MB.

Datastore IOPS requirements are modest — SQLite needs 10 IOPS / <10ms latency; embedded etcd needs 50 IOPS / <5ms latency.

Known Issues & Pitfalls

1. etcd startup loop at scale (k3s#14130): At ~2500 nodes, clearAlarms() gets rate-limited by etcd under write pressure (>83 writes/sec), causing infinite restart loops.

2. kubelet sandbox regression (k3s#14078): On v1.36.0, RunPodSandbox times out due to a gRPC module version bump. Workaround: pin to v1.35.x.

3. Crash loop on no default route (k3s#13895): If k3s starts before a default network route exists, it enters a 5-second restart loop at 100% CPU. Mitigation: systemd drop-in with StartLimitBurst=3 and RestartSec=30s.

4. Pod-to-apiserver timeout on same node (k3s#13721): Traffic to 10.43.0.1 (kubernetes service) intermittently times out due to k3s's load-balancer logic — SYN reaches cni0 but no ACK returns.

5. SD card wear on ARM: etcd is write-intensive. Always use an external SSD for Raspberry Pi clusters.

Production Best Practices

1. Minimum 3 server nodes for HA quorum
2. Use fast SSDs (NVMe preferred) for /var/lib/rancher/k3s/server/
3. Isolate datastore IOPS from workload storage
4. Place a load balancer (HAProxy) in front of server nodes
5. Use external datastore for clusters >50 nodes
6. Monitor etcd latency — critical for control plane health
7. Set --disable-network-policy when using Calico or Cilium
8. For Cilium: clean up virtual interfaces before k3s uninstall
9. For Rook-Ceph: always override csi.kubeletDirPath to /var/lib/rancher/k3s/agent/kubelet

Weave Net Is Dead

Weave Net was the fourth major option but is effectively archived. Weaveworks shut down, Kubespray removed Weave support in 2025, and maintenance has been handed to a community fork. Do not use Weave for new clusters.

Decision Framework

Summary

The k3s ecosystem offers choices at every layer. The default setup (Flannel + Traefik + Local Path) works well for single-node and edge deployments, but as soon as you need NetworkPolicy enforcement, HA, or production performance, you'll need to swap components.

Key takeaways:

• Flannel does not enforce NetworkPolicies — k3s ships a kube-router-based controller to fill this gap, but if you disable it, policies are silently ignored
• Cilium offers the best performance and deepest policy support (L3-L7) at the cost of kernel requirements and memory
• Calico is the most battle-tested CNI with both iptables and eBPF data plane options
• Embedded etcd is fine for clusters up to ~50 nodes; beyond that, use an external datastore
• Traefik is adequate for most k3s deployments and avoids maintaining a second ingress controller

The right choice depends on your scale, performance requirements, and how much kernel-level control you need — but in every case, verify NetworkPolicy enforcement explicitly. Don't assume it works because you created a NetworkPolicy object.

Cross-Links

• Kubernetes GitOps with ArgoCD — managing k3s clusters declaratively
• Container Orchestration — comparing orchestration approaches beyond k3s
• Container Building and Hardening Guide — secure images for your cluster workloads
• Kubernetes Gateway API Migration — next-generation ingress for Kubernetes
• CNCF Cloud Native Landscape — where k3s fits in the ecosystem