Edge Computing Patterns with k3s: Multi-Cluster Management, Offline-First Deployments, and Observability at Scale

Software Testing That Earns Its Keep: A Field Report from Three Codebases

Evidence-based testing practice from production codebases — property-based invariants, seeded fault injection, contract tests, benchmark-verified performance, and quality gates that are enforced, not aspirational.

K3s Distributions Compared: Flannel, Calico, Cilium — CNI Performance and NetworkPolicy Reality

A comprehensive comparison of k3s distributions (official, k3sup, k3d, AutoK3s) and CNI plugins (Flannel, Calico, Cilium, Canal) covering NetworkPolicy support, performance benchmarks, ingress, storage, and production deployment considerations.

k3s is optimized for edge computing scenarios where you need to deploy and manage hundreds or thousands of small Kubernetes clusters across distributed sites. This article explores proven patterns for multi-cluster management, offline-first deployments, and observability at scale using k3s.

Multi-Cluster Management at Scale

Managing distributed edge clusters requires hierarchical patterns, GitOps workflows, and centralized control planes that can scale to thousands of clusters.

Rancher Fleet for 10,000+ Clusters

Rancher Fleet is a GitOps multi-cluster management tool designed for Kubernetes at scale. It follows a hierarchical control plane model:

Central Cluster (Rancher + Fleet)
    ↓ GitOps repositories
    ↓
Regional Clusters (Hub clusters)
    ↓ Fleet agents
    ↓
Edge Clusters (Leaf nodes)

Key capabilities:

• Declarative GitOps: Deploy to thousands of clusters via Git repositories
• Hierarchical architecture: Central → Regional → Edge topology
• Targeted deployments: Deploy to subsets using cluster selectors, labels, and groups
• Rollout strategies: Canary, blue-green, and phased rollouts
• Resource optimization: Single agent per cluster (~50MB memory footprint)
• Offline support: Bundle manifests and artifacts for airgap deployments

Fleet CLI basics:

# Register a downstream cluster
fleetctl login https://fleet.example.com
fleetctl cluster register --name store-001 --labels region=us-west,zone=store

# Create a GitRepo resource
fleetctl gitrepo create \
  --name my-app \
  --repo https://github.com/org/manifests \
  --paths ./clusters/production \
  --targets "clusterSelectors.matchLabels.region=us-west"

# Trigger rollout
fleetctl gitrepo update my-app --force

GitOps Patterns for Edge

GitOps is essential for edge deployments because it provides:

1. Audit trail: Every change is recorded in Git
2. Offline resilience: Clusters can continue operating without central connectivity
3. Rollback: Revert by committing a change to Git
4. Multi-tenancy: Different teams manage different manifests

GitRepo resource example:

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: retail-app
  namespace: fleet-default
spec:
  repo: https://github.com/retail/edge-manifests
  branch: main
  paths:
    - ./clusters/production
  targets:
    - clusterSelector:
        matchLabels:
          region: us-west
          site-type: retail-store
  revisionHistoryLimit: 10

Hierarchical Cluster Architecture

For distributed edge sites, use a 3-tier hierarchy:

Regional cluster benefits:

• Caching proxy: Harbor registry caches images for edge sites
• Failover hub: Regional cluster can serve as Fleet hub if central is unavailable
• Bandwidth optimization: Regional clusters filter and aggregate telemetry
• Local observability: Regional Grafana dashboards for quick incident response

Case Study: 500+ Retail Stores

A national retail chain deployed k3s to 500+ stores using Rancher Fleet:

Challenges:

• Deployment took 3-5 days per store (manual on-site visits)
• Configuration drift across sites
• No centralized visibility
• Firmware updates required physical visits

Solution:

Central Rancher (k3s)
  ↓ GitOps manifests
  ↓ Fleet agents
  ↓
500+ Store k3s clusters (Raspberry Pi 4 + industrial chassis)

Results:

• Deployment time: Reduced from days to less than one hour (remote provisioning)
• Error rate: 90% reduction in deployment errors
• Visibility: Real-time dashboards showing all 500+ clusters
• Updates: Firmware and app updates pushed fleet-wide in minutes
• Hardware: Raspberry Pi 4 (8GB) with custom industrial enclosure
• Cost: ~$300 per store vs$2,000+ for previous solution

Fleet manifest for retail stores:

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: retail-store
spec:
  repo: https://github.com/retail/store-manifests
  paths:
    - ./clusters/production
  targets:
    - clusterSelector:
        matchLabels:
          environment: production
          store-type: retail
  helm:
    releaseName: pos-system
    chart: ./charts/pos-system
    valuesFiles:
      - ./clusters/production/values/store-default.yaml

Offline-First Deployments

Edge sites often have unreliable or no connectivity. Offline-first patterns ensure clusters can bootstrap, update, and operate without central access.

Airgap Installation

k3s supports airgap installation via a bundled images archive:

Step 1: Download airgap bundle on connected machine

# Download k3s binary and airgap images
wget https://github.com/k3s-io/k3s/releases/download/v1.30.2/k3s-airgap-images-amd64.tar.gz
wget https://github.com/k3s-io/k3s/releases/download/v1.30.2/k3s

# Copy to airgap media
cp k3s k3s-airgap-images-amd64.tar.gz /media/usb/

Step 2: Transfer to airgap machine and install

# Import images
sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp /media/usb/k3s-airgap-images-amd64.tar.gz /var/lib/rancher/k3s/agent/images/
sudo chmod 644 /var/lib/rancher/k3s/agent/images/*.tar.gz

# Install k3s
sudo install /media/usb/k3s /usr/local/bin/

# Start k3s
sudo INSTALL_K3S_SKIP_DOWNLOAD=true \
  INSTALL_K3S_SKIP_ENABLE=true \
  INSTALL_K3S_EXEC="server --no-deploy traefik" \
  /usr/local/bin/k3s

Step 3: Verify airgap installation

# Check images are imported
sudo crictl images | grep -E "rancher|pause"

# Verify cluster
kubectl get nodes

Private Registry for Airgap

Deploy a private container registry at each edge site to cache images:

Harbor deployment via Helm:

# harbor-values.yaml
expose:
  type: nodePort
  tls:
    enabled: false
persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      storageClass: "local-path"
portal:
  replicas: 1
registry:
  replicas: 1
  storage:
    type: filesystem

# Install Harbor
helm repo add harbor https://helm.goharbor.io
helm install harbor harbor/harbor -f harbor-values.yaml -n harbor --create-namespace

# Push images to local registry
docker tag rancher/local-path-provisioner:v0.0.26 harbor.local/rancher/local-path-provisioner:v0.0.26
docker push harbor.local/rancher/local-path-provisioner:v0.0.26

Configure k3s to use local registry:

# Create registry mirror config
sudo mkdir -p /etc/rancher/k3s
cat <<EOF | sudo tee /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
    - http://harbor.local/v2/registry/mirrors/docker.io
  registry.k8s.io:
    endpoint:
    - http://harbor.local/v2/registry/mirrors/registry.k8s.io
EOF

# Restart k3s
sudo systemctl restart k3s

Embedded Registry Mirror with Spegel

Spegel is a P2P registry mirror that runs on each k3s node:

# spegel-helm-values.yaml
image:
  repository: ghcr.io/xenitab/spegel
  tag: v0.0.18
registry:
  mirror:
    enabled: true
    interval: 10m
  registries:
    - https://registry.k8s.io
    - https://docker.io
podMonitor:
  enabled: true

# Install Spegel
helm repo add spegel https://charts.spegel.io
helm install spegel spegel/spegel -f spegel-helm-values.yaml -n kube-system

How Spegel works:

1. P2P distribution: Images are distributed peer-to-peer across nodes
2. Local cache: Each node caches images locally
3. Registry proxy: Spegel acts as a registry proxy, pulling from peers first
4. Offline resilience: Once images are cached, no external connectivity needed

Benefits:

• No external registry required after initial boot
• Faster image pulls (local LAN vs internet)
• Bandwidth savings (each image downloaded once per cluster)
• Offline operation (clusters continue operating without internet)

Observability at Scale

Observability across thousands of edge clusters requires bandwidth-efficient patterns, hierarchical aggregation, and resilient telemetry collection.

Prometheus + node-exporter + Grafana

Standard monitoring stack adapted for edge:

# prometheus-operator-values.yaml
prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 500m
        memory: 1Gi

nodeExporter:
  enabled: true

grafana:
  enabled: true
  persistence:
    enabled: true
    size: 5Gi
    storageClassName: local-path

Edge Prometheus configurations:

• Reduced retention: 15-30 days (vs 90+ days in central)
• Resource limits: ~500MB RAM, ~500m CPU per cluster
• Local-path storage: No distributed storage required
• Consul federation: Optional federation to regional Prometheus

Loki + Fluent Bit for Log Aggregation

Loki provides cost-effective log aggregation for edge:

# loki-stack-values.yaml
loki:
  persistence:
    enabled: true
    storageClassName: local-path
    size: 10Gi
  config:
    limits_config:
      retention_period: 168h # 7 days
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: inmemory

fluent-bit:
  enabled: true
  config:
    service: |
      [SERVICE]
          Flush        1
          Daemon       Off
          Log_Level    info
          Parsers_File parsers.conf
    inputs: |
      [INPUT]
          Name         tail
          Path         /var/log/containers/*.log
          Parser       docker
          Tag          kube.*
          Mem_Buf_Limit 5MB
          Skip_Long_Lines On
    filters: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Kube_URL            https://kubernetes.default.svc:443
          Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
          Kube_Tag_Prefix     kube.var.log.containers.
          Merge_Log           On
          Merge_Log_Key       log_processed
          K8S-Logging.Parser  On
          K8S-Logging.Exclude On
    outputs: |
      [OUTPUT]
          Name            loki
          Match           *
          Host            loki.monitoring.svc
          Port            3100
          Labels          job=fluent-bit,cluster=store-001,region=us-west
          Line_Format     json

Loki optimization for edge:

• Short retention: 7 days (vs 30+ days in central)
• Local storage: No external database required
• Tail-based sampling: Filter logs before shipping
• Structured logging: JSON format for efficient querying

Thanos/Cortex for Long-Term Storage

For long-term metrics retention and cross-cluster analytics:

# thanos-values.yaml
query:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi

queryFrontend:
  replicas: 1
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 100m
      memory: 128Mi

store:
  replicas: 1
  persistence:
    storageClass: local-path
    size: 50Gi
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 512Mi

objstore:
  type: s3
  config:
    endpoint: minio.storage.svc.cluster.local
    bucket: thanos
    access_key: ${THANOS_S3_ACCESS_KEY}
    secret_key: ${THANOS_S3_SECRET_KEY}

Thanos remote_write from edge Prometheus:

# remote_write configuration
remote_write:
  - url: http://thanos-receive.thanos.svc.cluster.local:19291/api/v1/receive
    queue_config:
      capacity: 10000
      max_shards: 10
      min_shards: 1
      max_samples_per_send: 1000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 100ms
      retry_on_http_429: true
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop # Drop low-value metrics

Bandwidth-Efficient Telemetry

Tail-based sampling (Loki):

# Fluent Bit filter for sampling
[FILTER]
    Name                sample
    Match               kube.*
    Rate                10    # Keep 10% of logs
    Sample_Key          level
    Sample_Values       info,warn,error,critical

Metrics aggregation (Prometheus):

# recording rules for edge aggregation
groups:
  - name: edge_aggregation
    interval: 30s
    rules:
      - record: cluster:node_cpu_usage:avg5m
        expr: avg(rate(node_cpu_seconds_total[5m])) by (cluster)
      - record: cluster:node_memory_usage:avg5m
        expr: avg(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) by (cluster)
      - record: cluster:container_memory_usage_bytes:sum
        expr: sum(container_memory_working_set_bytes) by (cluster, namespace)

Downsampling for long-term storage:

# Thanos downsampling configuration
downsample:
  enable: true
  retention:
    raw: 30d
    five_minutes: 90d
    one_hour: 1y

File-Backed Queues for Outages

Use bbolt for reliable telemetry during outages:

[OUTPUT]
    Name            loki
    Match           *
    Host            loki.monitoring.svc
    Port            3100
    Labels          job=fluent-bit,cluster=store-001
    storage_path    /var/lib/fluent-bit
    storage_sync    normal
    storage_checksum off
    storage_max_backups 3
    storage_max_chunks_up 128
    storage_chunk_size 1M

How bbolt works:

1. Local queue: Logs buffered to local bbolt database
2. Retry on failure: Fluent Bit retries failed sends
3. No data loss: Survives network outages and pod restarts
4. Backpressure: Rate limits when remote endpoint is slow

Production Considerations

Network Topology

Design network topology for edge resilience:

Internet
  |
  ↓
[Central Data Center]
  |                                   |
  ↓ (fiber/VPN)                      ↓ (LTE/5G backup)
[Regional Hub A]                 [Regional Hub B]
  |                                   |
  ↓ (local fiber)                    ↓ (local fiber)
[Edge Stores]                      [Edge Stores]
  (mesh P2P via Spegel)              (mesh P2P via Spegel)

Best practices:

• Dual uplinks: Primary fiber + LTE/5G backup
• VPN tunnels: WireGuard or IPsec for secure communication
• Local DNS: CoreDNS with conditional forwarding for local services
• Time sync: Chrony for accurate time across edge sites

Resource Planning

Recommended hardware for k3s edge nodes:

k3s resource tuning:

# k3s config for edge
controller-args:
  - --disable=traefik
  - --disable=servicelb
  - --disable=metrics-server
  - --kubelet-arg=pod-max-pods=50
  - --kubelet-arg=cgroups-per-qos=true
  - --kubelet-arg=system-reserved=cpu=200m,memory=512Mi
  - --kubelet-arg=kube-reserved=cpu=200m,memory=512Mi

Security at the Edge

Security hardening for edge deployments:

1. Immutable infrastructure: No SSH access, only Fleet agent
2. Network segmentation: VLANs or network policies for isolation
3. Certificate management: Automated cert rotation via cert-manager
4. Image scanning: Trivy admission controller
5. RBAC: Strict role-based access control
6. Audit logging: Kubernetes audit log + centralized SIEM

Admission controller for image scanning:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: trivy-admission-controller
webhooks:
  - name: trivy.kb.io
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    clientConfig:
      service:
        name: trivy
        namespace: trivy-system
    admissionReviewVersions: ["v1"]
    sideEffects: None

Decision Framework

Conclusion

k3s provides a solid foundation for edge computing scenarios, but successful deployments require patterns for:

1. Multi-cluster management: Fleet for GitOps control across thousands of clusters
2. Offline-first operations: Airgap installations, private registries, and P2P mirrors
3. Observability at scale: Bandwidth-efficient telemetry, hierarchical aggregation, and resilient collection

The combination of Fleet's GitOps control plane, Harbor's registry caching, and Spegel's P2P distribution creates a resilient edge computing platform that scales from tens to thousands of clusters while maintaining operational simplicity and offline resilience.