Xirin

Search

SearchSearch

HomeLab Series - Setting up K3s Cluster with MetalLB

Jan 08, 2024, 4 min read

Hence I added more services into my homelab, I need a more elegant way to manage those server and services.So,Rebuild my environment from bare VM into a k8s cluster driven one.

I am going to post a series to explain how achieved it.

image

Next:: HomeLab Series -Installing Homarr with Helm in Cluster and Exposing it via Cloudflare

Prelude

In this section, I will show how to build a cluster with the following specifications:

  • Quantity of nodes: 3
  • Web UI: Rancher
  • Load Balancer: Metallb
  • Public Access: Cloudflare Tunnel
  • Ingress Controller: Nginx
  • Setup Tool: k3sup
  • Base System: Rocky Linux 9.2
  • Package Management: Helm

Preparation:

Firstly, assume you have three Rocky Linux 9.2 VMs installed in Proxmox. Create an admin user for each and set the IP addresses to static values to avoid IP changes after restarts.

HostNameIP AddressUser
Node110.1.1.101admin
Node210.1.1.102admin
Node310.1.1.103admin

Before installing k3s, some preparations are necessary:

  1. Turn Off the Firewall for Convenience:

    Disabling the firewall can ease the installation process, but it’s crucial to re-enable it later for security

    systemctl disable firewalld

  2. Turn Off SELINUX:

    SELINUX can often interfere with installations and services running on the system.

    bashCopy code

    sudo sed -i -e "s/^SELINUX=enforcing$/SELINUX=disabled/g"    /etc/selinux/config  
# After modification, reboot VM to confirm 
getenforce

  3. Turn Off Swap:

    Kubernetes usually requires swap to be turned off for better performance and stability.

    swapoff -a

  4. Generate SSH Key Pair:

    This step is crucial for enabling SSH access without a password, simplifying node management.

    ssh-keygen

  5. Transfer Public Key to Servers:

    Copy the public key to other servers for secure passwordless SSH connections.

    bashCopy code

    ssh-copy-id [email protected] ssh-copy-id [email protected]

  6. Add Admin User to Auto Sudo in Every VM:

    Granting admin users the ability to execute sudo commands without a password streamlines various administration tasks.

    • Add User to the Wheel Group:

      sudo usermod -aG wheel admin

    • Verify the User’s Group Membership:

      groups admin

    • Ensure that Wheel Group Has sudo Privileges: This step is to ensure that the admin users can execute commands as superusers.

Preparation is now complete.

Installation Begin

Install Node1:

Set up the first node as the master node using k3sup.

export SERVER_IP=10.1.1.101 export USER=admin  k3sup install --ip $SERVER_IP --user $USER --no-extras --ssh-key /home/admin/.ssh/id_rsa

Join Node2 and Node3 to Node1:

Extend the cluster by adding additional nodes.

# For Node2 
export AGENT_IP=10.1.1.101 
# Server's IP
export SERVER_IP=10.1.1.102  
# Repeat for Node3 with AGENT_IP=10.1.1.103
export USER=admin  k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER --ssh-key /home/admin/.ssh/id_rsa

Test Your Cluster:

Verify that all nodes are connected and recognized by the cluster.

export KUBECONFIG=/home/admin/kubeconfig kubectl config use-context default kubectl get node -o wide

Install Metallb:

Metallb will provide a network load balancer for your cluster.You loadbalancer service will obtain IP address same as your home network one.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml

Configure Parameters:

Create configuration files for Metallb to define the range of IP addresses it can use.

# Create IPAddressPool.yaml
cat << EOF > IPAddressPool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool
  namespace: metallb-system
spec:
  addresses:
  - 10.1.1.70-10.1.1.99
EOF
 
# Create metallb.yaml
cat << EOF > metallb.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: config
  namespace: metallb-system
EOF
 
kubectl apply -f metallb.yaml IPAddressPool.yaml

Install Nginx:

Nginx will act as the ingress controller, managing access to your cluster’s services.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.7.1/deploy/static/provider/baremetal/deploy.yaml

Configure Helm & Install Rancher:

Set up Helm, a package manager for Kubernetes, and use it to install Rancher for cluster management.

# Install Helm
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh

# Set KUBECONFIG
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Add Helm repo for Rancher
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

# Create namespace for Rancher
kubectl create namespace cattle-system

# Install Rancher with Helm
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.yourdomain.cc --set bootstrapPassword=YourPasswordHere --set ingress.tls.source=secret 

# Expose Rancher and Portainer via LoadBalancer
kubectl expose deployment rancher -n cattle-system --type=LoadBalancer --name=rancher-lb --port=443

Set Environment Variable for Future Use with kubectl and Helm:

Add KUBECONFIG to your .bashrc for easy future access to your cluster.

echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

クリエイティブ・コモンズ・ライセンス
この 作品 は クリエイティブ・コモンズ 表示 - 非営利 - 改変禁止 4.0 国際 ライセンスの下に提供されています。

@Bintis 著作权，不许抄。

Backlinks