Blog

The Kenough Kubernetes Cluster

The nodes #

I named them ken1 and ken2 because, despite being outdated mini PCs, they are kenough to be Kubernetes clusters.

Quick hardware overview:

The Cilium Mesh Experiment #

Cilium has a Cluster Mesh feature that allows you to extend the Cilium service mesh on multiple clusters together: https://cilium.io/use-cases/cluster-mesh/. I want to test this feature across a real physical network separation by running a Kubernetes cluster each on the Kens and having apps communicate across the Cluster Mesh. A 1Gb link between two clusters of 4 cores should sorta kinda maybe be representative of a 100Gb link between two production clusters of 400 cores.

Talos Linux on Proxmox #

After some brief attempts with KinD and microk8s on Ubuntu I decided to look into this Talos Linux thing and was immediately very happy.

Reference to follow: Talos on Proxmox

Warning: dense setup notes after this point.

Setup I: Template VM #

Generate a Talos image with the guest-agent extension. Create a VM following the minimum specs:

Role Memory Cores Disk
Control plane 2GB 2 10GB
Worker 1GB 1 10GB

Configuration

Setup II: Cluster bootstrap #

Do a full clone of the template and boot it. Apply the config for the first control plane node, then set TALOSCONFIG the generated config file and run talosctl config endpoint <IP> and talosctl config node <IP>.

Initialize etcd with talosctl bootstrap, then get your kubeconfig with talosctl kubeconfig ..

Setup III: Additional nodes #

Adding additional nodes:

I created 3 control plane and 3 worker nodes total for the next step of setting up Cilium.

Cilium on Talos #

Reference: Deploying Cilium CNI on Talos

You need to change the machine config to add cni=none. I reset my VMs and generated a new base config: talosctl gen config talos-proxmox-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out_cilium --install-image factory.talos.dev/installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.7.0 --config-patch @nocni_patch.yaml

nocni_patch.yaml:

cluster:
  network:
    cni:
      name: none
  # Disables kube-proxy
  proxy:
    disabled: true

Once the nodes are partially ready (Pods will not be ready as there is no CNI), use Helm to install cilium.

cilium values.yaml for full CNI+LB+Ingress 
# helm install cilium cilium/cilium --version 1.15.6 --namespace kube-system -f cilium-values.yaml

ipam:
  mode: kubernetes

kubeProxyReplacement: true # not since 1.14?

l2announcements:
  enabled: true

externalIPs:
  enabled: false

securityContext:
  capabilities:
    ciliumAgent: ["CHOWN","KILL","NET_ADMIN","NET_RAW","IPC_LOCK","SYS_ADMIN","SYS_RESOURCE","DAC_OVERRIDE","FOWNER","SETGID","SETUID"]
    cleanCiliumState: ["NET_ADMIN","SYS_ADMIN","SYS_RESOURCE"]

cgroup:
  autoMount:
    enabled: false
  hostRoot: "/sys/fs/cgroup"


k8sServiceHost: localhost
k8sServicePort: 7445
# Cilium ships with a low rate limit by default that can result in strange issues when it gets rate limited
k8sClientRateLimit:
  qps: 50
  burst: 100

hubble:
  relay:
    enabled: true
  ui:
    enabled: true

hostFirewall:
  enabled: true

ingressController:
  enabled: true
  loadbalancerMode: shared
  default: true

rolloutCiliumPods: true

I installed the Cilium CLI and ran cilium status to confirm Cilium was healthy.

When using Cilium LB, some additional steps are needed to get it to assign IPs for LoadBalancer and Ingress objects:

Useful commands #

Metrics and Grafana Cloud #

Talos metrics

Grafana Cloud has a very reasonable free tier that I signed up for. There's a dedicated Kubernetes section that will generate a preconfigured Helm install for you to remote write metrics to Grafana Cloud with premade dashboards available.

Storage without busy looping half the CPU #

I wanted to have dynamically allocated PVs that are accessible from any node. They'd all use the same SSD but in the future it would scale to multiple disks / multiple nodes.

The first few storage options attempted did not work out:

Enter Piraeus Linstor

Here's what it looks like once you have some working pools with storage assigned to PVs:

Piraeus Linstor on Talos Setup Overview #

Full post here:

Talos patch examples #

Patch collection

Allow scheduling on control plane

cluster:
    allowSchedulingOnControlPlanes: true

DRBD patch for Piraeus Linstor

# https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/talos.md
machine:
  kernel:
    modules:
      - name: drbd
        parameters:
          - usermode_helper=disabled
      - name: drbd_transport_tcp

Metrics patch

# https://www.talos.dev/v1.7/kubernetes-guides/configuration/deploy-metrics-server/
# Needed on all nodes
# also,
# kubectl apply -f https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml
# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
machine:
  kubelet:
    extraArgs:
      rotate-server-certificates: true

No CNI patch for installing your own CNI

cluster:
  network:
    cni:
      name: none
  # Disables kube-proxy
  proxy:
    disabled: true

Mayastor patch

- op: add
  path: /machine/sysctls
  value:
    vm.nr_hugepages: "1024"
- op: add
  path: /machine/nodeLabels
  value:
    openebs.io/engine: mayastor
- op: add
  path: /machine/kubelet/extraMounts
  value:
    # for etcd
    - destination: /var/local/mayastor/localpv-hostpath/etcd # Destination is the absolute path where the mount will be placed in the container.
      type: bind # Type specifies the mount kind.
      source: /var/local/mayastor/localpv-hostpath/etcd # Source specifies the source path of the mount.
      # Options are fstab style mount options.
      options:
        - bind
        - rshared
        - rw

    # for loki
    - destination: /var/local/mayastor/localpv-hostpath/loki # Destination is the absolute path where the mount will be placed in the container.
      type: bind # Type specifies the mount kind.
      source: /var/local/mayastor/localpv-hostpath/loki # Source specifies the source path of the mount.
      # Options are fstab style mount options.
      options:
        - bind
        - rshared
        - rw

OpenEBS patch

# talosctl patch --mode=no-reboot machineconfig -n IP --patch @openebs_new_patch.yaml
machine:
  kubelet:
      # # The `extraMounts` field is used to add additional mounts to the kubelet container.
      extraMounts:

      # for etcd
          - destination: /var/local/openebs/localpv-hostpath/etcd # Destination is the absolute path where the mount will be placed in the container.
            type: bind # Type specifies the mount kind.
            source: /var/local/openebs/localpv-hostpath/etcd # Source specifies the source path of the mount.
      #       # Options are fstab style mount options.
            options:
              - bind
              - rshared
              - rw

            # for loki
          - destination: /var/local/openebs/localpv-hostpath/loki # Destination is the absolute path where the mount will be placed in the container.
            type: bind # Type specifies the mount kind.
            source: /var/local/openebs/localpv-hostpath/loki # Source specifies the source path of the mount.
      #       # Options are fstab style mount options.
            options:
              - bind
              - rshared
              - rw

Random notes on various Talos things #

Talos mayastor:

OpenEBS new instructions https://openebs.io/docs/quickstart-guide/installation

Talos Longhorn

Reprovision Talos node with more storage