Deploying K8s on Xen-Orchestra/XCP-NG with Terraform and Ansible

Preface

A little while ago I wrote a guide on how to deploy a k8s stack using Terraform and Ansible on a XCP-NG host runnign Xen Orchestra. While I think the guide was OK, I did cut quite a few corners on it and I want to put that right. Also, I was working ok K8s stuff again and wanted something that was a little easier to re-deploy, just incase I messed something up real bad. Like setting the pods network to my home network’s subnet causing the servers to stop responding… Doh.

Anyways, I will walk you through how to deploy K8s on XCP-NG with Xen Orchestra. This is designed for Xen Orchestra only but you might be able to adapt the code with a little bit of googling for other platforms like VMWare or Proxmox.

Terraform

Terraform has a module for XCP-NG. A prerequisite of this guide is to already have a VM template running Rocky-Linux. Also, in the template VM you will need the guest-tools installed as that is required to get edditinal information out of the VM when it is created. Also, you will obviously need a Xen Orchestra instance running. It can ether be one running in docker or the one that you can deploy directly from the hypervisor.

If you have any suggestions or questions, please feel free to reach out to me on MASTODON.

First, we need to set up the provider. Create provider.tf

terraform {
  required_providers {
    xenorchestra = {
      source = "terra-farm/xenorchestra"
    }
  }
}

It’s now time to download the module. Run: terraform init. The output should look a little like this:

We now need to setup the data vars in vm.tf

data "xenorchestra_pool" "pool" {
  name_label = "HOSTNAME"
}

data "xenorchestra_template" "rocky_template" {
  name_label = "TEMPLATE NAME"
}

data "xenorchestra_sr" "sr" {
  name_label = "STORAGE NAME" 
  pool_id = data.xenorchestra_pool.pool.id
}

data "xenorchestra_network" "network" {
  name_label = "Pool-wide network associated with XXX"
  pool_id = data.xenorchestra_pool.pool.id
}

Replace the temp values with your values. The name_label can be a value with spaces in. I got confused when it came to storage, as “Local Storage” didn’t seem like it would be a valid input and expected XCP-NG to have some other value. Nevertheless, it was correct.

We now want to test this, but we need to provide credentials. I use a .env file in my Terraform directory. The contents of it is:

export XOA_URL=ws://XEN_ORCHESTRA_IP
export XOA_USER=USERNAME
export XOA_PASSWORD=PASSWORD
export TF_VAR_ssh_username=USERNAME
export TF_VAR_ssh_password=PASSWORD

To add these values to your shell, just run eval $(cat .env).

You can now run terraform plan and you should see something like this:

The last thing we need to create for our Terraform code to be able to work is creating the varibles file so it can use the ssh_password we included in the .env file. Create a file called varibles.tf

variable "ssh_username" {
  description = "SSH usernme"
  type = string
  sensitive = true
}

variable "ssh_password" {
  description = "SSH password"
  type = string
  sensitive = true
}

If all is working, we can now get to the juicy part… creating the VMs.

resource "xenorchestra_vm" "k8s_master" {
  memory_max = 8589869056
  cpus = 2
  name_label = "K8S Master Node"
  wait_for_ip = true
  template = data.xenorchestra_template.rocky_template.id

  network {
    network_id = data.xenorchestra_network.network.id
  }

  disk {
    sr_id = data.xenorchestra_sr.sr.id
    name_label = "k8s master node disk"
    size = 50214207488
  }

  connection {
    type     = "ssh"
    user     = var.ssh_username
    password = var.ssh_password
    host     = self.ipv4_addresses[0]
  }

  provisioner "remote-exec" {
    inline = ["sudo hostnamectl set-hostname k8s-master.local"]

  }
}

This is a supper simple VM setup.

resource is how you define a VM you want to create.
memory_max is how you define the max amount of ram you want the host to use.
cpus is how many cores you want.
name_label is just a label for the VM.
wait_for_ip is to ensure that the VM gets a IP before we try and connect to it. This provents random errors.
template is the template you want the VM to be created from.
network will create the interface with
- network_id being the nic we want to use, which is what we defined earlier in the data section.
disk is the last part we define.
- sr_id is the storage pool we defined in the data section.
- name_label is the name of the disk.
size is how large we want the disk.

You will also see that in the connection section we refere to var.ssh_username and var.ssh_password. This is to pull in the data from the varibles file we setup earlier.

You can use what ever values you want here, but it is strongly recommended that you have at least 2 CPU cores and 8GB of ram.

We can now run terraform plan to see what it will create. If all went according to plan you can now run terraform apply and it should go ahead and create the VM for you.

It’s now time to create the worker nodes. In the previous guide, I reccomended creating two diffrent resources for the worker nodes but it is much easier to create multipule hosts from one resource.

resource "xenorchestra_vm" "k8s_worker_node" {
  count = 2
  memory_max = 8589869056
  cpus = 2
  name_label = "K8S Worker Node ${count.index}"
  wait_for_ip = true
  template = data.xenorchestra_template.rocky_template.id

  network {
    network_id = data.xenorchestra_network.network.id
  }

  disk {
    sr_id = data.xenorchestra_sr.sr.id
    name_label = "k8s worker node ${count.index} disk"
    size = 50214207488
  }

  connection {
    type     = "ssh"
    user     = var.ssh_username
    password = var.ssh_password
    host     = self.ipv4_addresses[0]
  }

  provisioner "remote-exec" {
    inline = ["sudo hostnamectl set-hostname k8s-worker-${count.index}.local"]

  }
}

One diffrence in this section is we have count = 2 which means it will create two instances of the resource. We also refrence ${count.index} in some places. That is so we can have the description and name to match what instance it is.

Now you have two worker nodes defined in the code, you can run terraform plan again. It is a good idea to always run plan before so you know what is going to happen, though apply dose give you a heads-up. If all is good, and it says it’s going to create two more VMs, you can run terraform apply. If it is getting confused you can always run terraform destroy to get back to the beginning, this isn’t an issue as we have nothing running yet.

After I have run it you should see it saying it has created 3 resources.

When I go to my dashboard now, I see the new VMs we created:

Lastly, we want to make our lives a little easier when deploying to ansible. What we can do is generate a ansible_inventory file from this terraform.

We want to add this to the end of the vm.tf file.

resource "local_file" "ansible_inventory" {
  content = <<-EOT
    [k8s_all]
    k8s-master.local ansible_ssh_host=${xenorchestra_vm.k8s_master.ipv4_addresses[0]} ansible_user=harley
    ${join("\n", formatlist("k8s-worker-%v.local ansible_ssh_host=%v ansible_user=harley", range(length(xenorchestra_vm.k8s_worker_node)), [for node in xenorchestra_vm.k8s_worker_node : node.ipv4_addresses[0]]))}

    [k8s_master]
    k8s-master.local ansible_ssh_host=${xenorchestra_vm.k8s_master.ipv4_addresses[0]} ansible_user=harley

    [k8s_worker]
    ${join("\n", formatlist("k8s-worker-%v.local ansible_ssh_host=%v ansible_user=harley", range(length(xenorchestra_vm.k8s_worker_node)), [for node in xenorchestra_vm.k8s_worker_node : node.ipv4_addresses[0]]))}

  EOT
  filename = "${path.module}/ansible_inventory.ini"
}

output "ansible_inventory_path" {
  value = local_file.ansible_inventory.filename
  description = "Path to the generated Ansible inventory file."
}

Most of this code is just generating a file so I won’t go into detail on that but there is a interesting section. It’s the join() part! Let me go through that. range(length(xenorchestra_vm.k8s_worker_node)): Creates a sequence of numbers from 0 to the number of worker nodes minus one. [for node in xenorchestra_vm.k8s_worker_node : node.ipv4_addresses[0]]: Extracts the first IPv4 address for each worker node. formatlist(...): Formats each worker node’s entry according to the pattern provided. join("\n", ...): Joins all formatted entries with newlines to form a single string.

If you need to edit that section, I recomend having a play. It can be a little fiddly but can be really powerful for generating a bunch of diffrent things like reports or diffrent varible files for other configuration tools.

Ansible

Now we move over to setting up the host with Ansible. There is lots to define, so let’s get cracking.

All Hosts

First, lets create a seperate direcotry for ansible outside the terraform folder. We also will want to copy over the ansible_inventory file into that directory.
Now we are going to want to create a valut for storing our secrets. With ansible, there is a build in tool for securing passwords called ansible-vault.

Start by creating a dir called vars then run ansible-vault create vars/default.yml. It will prompt you twice for a password. Now we can create our first playbook. To edit the file, you run ansible-vault edit vars/default.yml. You will want to add default_user_password: PASSWORD

Create a file called initial-setup.yml. We will want to start by defining the top of the file:

---
- name: Basic k8s node configuration
  hosts: k8s_all
  user: USER
  vars_files:
    - vars/default.yml
  vars:
    ansible_ssh_pass: "{{ default_user_password  }}"
  become: yes
  become_user: root
  become_method: sudo
  tasks:

name is a title for the job.
hosts allows you to limit what hosts this section will be run on. Here we are running it on all, but later we will use it to configure master nodes and worker nodes separately.
user is what user you want to connect as.
vars_files is specifying a variables file. In this case, it is our encrypted file.
vars is specifying what variables to use. In this case, the password used for ssh.
become is specifying if you want to become another user.
become_user is what user you want to become.
become_method is how you want to become that user. In our case, we are using sudo.
tasks is where you put what you want to run.

Remember to change the user to the user you setup

Ok, now we can move onto what we want to run. To start we want to add our public key, this isn’t 100% essential but is a nice to have as we don’t want to have to enter our password each time. Add this:

    - name: Set authorized key taken from file
      ansible.posix.authorized_key:
        user: harley
        state: present
        key: "{{ lookup('file', '/home/USER/.ssh/id_rsa.pub') }}"

Now we want to disable swap:

    - name: Turn off swap
      command: sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Set SELinux to permissive.

    - name: Set SELinux to permissive
      ansible.posix.selinux:
        policy: targeted
        state: permissive

We now want to create a dir called src/container.io. In that folder, we want to create containerd.conf with this as the content:

overlay
br_netfilter

We now want to add a section to transfer that to the hosts.

    - name: Copy container configuration
      ansible.builtin.copy:
        src: ./src/container.io/containerd.conf
        dest: /etc/modules-load.d/containerd.conf
        owner: root
        group: root
        mode: '0644'

We also want to ensure the modules are present. To do that add this:

    - name: Ensure moduales are present
      community.general.modprobe:
        name: "{{ item  }}"
        state: present
      loop:
        - overlay
        - br_netfilter

Now we want to configure some network options required for k8s. Add this to your Ansible file:

    - name: Enable network settings for k8s
      ansible.posix.sysctl:
        name: "{{ item  }}"
        value: '1'
        state: present
      loop:
        - net.bridge.bridge-nf-call-iptables
        - net.ipv4.ip_forward
        - net.bridge.bridge-nf-call-ip6tables

We now want to add Docker and K8S yum repos. Ansible has a built-in module ansible.builtin.yum_repository. Add this:

    - name: Add docker repo
      ansible.builtin.yum_repository:
        name: docker-ce-stable
        baseurl: https://download.docker.com/linux/centos/$releasever/$basearch/stable
        description: Docker Repo
        enabled: yes
        gpgcheck: yes
        gpgkey: https://download.docker.com/linux/centos/gpg

    - name: Add k8s repo
      ansible.builtin.yum_repository:
        name: kubernetes
        baseurl: https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
        description: k8s package
        enabled: yes
        gpgcheck: yes
        gpgkey: https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key

Next thing is to install the packages required.

    - name: Installing Packages
      yum:
        name:
          - containerd.io
          - vim
          - tree
          - git
          - kubelet
          - kubeadm
          - kubectl
        state: latest

We now want to add the hosts to /etc/hosts. This is a requirement for k8s. Well, it is a requirment if you don’t have DNS resolution via DNS.

    - name: Get all hosts and their IP addresses
      set_fact:
        host_entries: |
          {% for host in groups['all'] %}
          {{ hostvars[host]['ansible_ssh_host'] }} {{ host }} {{ host.split('.')[0] }}
          {% endfor %}

    - name: Append entries to /etc/hosts
      blockinfile:
        path: /etc/hosts
        block: |
          {{ host_entries }}
        marker: "# {mark} ANSIBLE MANAGED BLOCK"

We now want to configure containerd to use CGroups. Then we want to reboot the hosts to make sure all is applied.

    - name: Configure containerd
      shell: |
        containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
        sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

    - name: Reboot machines
      reboot:

Master Node

Now we have the basic configuration, we can move onto the master node. Here we will configure the firewall, enable k8s services and initialise kubelet.

First we want to define the name of the section, become and other boilerplate stuff. I named this file k8s-configuration.yml

---
- name: Configure k8s Master node
  hosts: k8s_master
  user: USER
  become: yes
  become_user: root
  become_method: sudo
  tasks:

Here we use the hosts section to limit it to just the master nodes (what is specified in hosts file).

We want to configure the firewall to open k8s ports along with some other useful ones.

    - name: Open ports in firewall
      ansible.posix.firewalld:
        port: "{{ item  }}"
        permanent: yes
        state: enabled
      loop:
        - 6443/tcp
        - 2379/tcp
        - 2380/tcp
        - 10250/tcp
        - 10251/tcp
        - 10252/tcp
        - 10257/tcp
        - 10259/tcp
        - 179/tcp
        - 22/tcp
        - 80/tcp
        - 443/tcp
        - 4789/udp

The firewalld module doesn’t have a built-in way to specify multiple ports, so we use a loop. Where we specify port we add "{{ item }}" which refers to the loop, and it will substitute the value with the current element in the loop. This can be really useful for keeping your playbooks neat.

We also want to reload the firewall. We can use the systemd module.

    - name: reload service firewalld
      systemd:
        name: firewalld
        state: reloaded

Now we have the ports, we can finally get to setting up k8s. You will first want to start containerd which is once again a built-in Ansible module.

    - name: Enable containerd
      systemd_service:
        name: containerd.service
        state: started
        enabled: true

We can now run kubeadm init and start the kubelet service.

    - name: k8s init
      shell: sudo kubeadm init --control-plane-endpoint="{{ inventory_hostname }}" --ignore-preflight-errors=all --pod-network-cidr=192.168.50.0/16

You will want to ensure that the --pod-network-cidr doesn’t conflict with your local network, but that is unlikely with that CIDR.

We can now enable the kubelet service, once again with systemd module.

    - name: Enable Kubelet
      ansible.builtin.systemd_service:
        name: kubelet.service
        state: started
        enabled: true

This should be it! The master node should be up and running. I will show you later how to check on it.

Worker nodes

We will take a different approach on this host, calling it from the master node. This is so we can pass the join command over.

You currently still want to be in the master node section. We want to run a command to print the join command. We can then pass that over to the worker section we will create after.

    - name: get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

Here we run a command and use a register to store the value. We can then add a loop to call the worker node section (that we haven’t created yet).

    - name: Configure worker nodes 
      add_host:
        name: "{{ item  }}"
        k8s_join_command: "{{ join_command_raw.stdout  }}"
      loop: "{{ groups['k8s_worker'] }}"

We now want to create the code for the worker section. You will want to add the typical boilerplate stuff.

- name: Configure k8s Worker nodes
  hosts: k8s_worker
  user: USER
  become: yes
  become_user: root
  become_method: sudo
  tasks:

The main thing to note is the hosts section is set to k8s_worker.

We want to configure the firewall for the worker node. The worker node has fewer ports, but the Ansible code is the same (just with fewer ports specified).

    - name: Open ports in firewall
      ansible.posix.firewalld:
        port: "{{ item  }}"
        permanent: yes
        state: enabled
      loop:
        - 30000-32767/tcp
        - 10250/tcp
        - 179/tcp
        - 22/tcp
        - 80/tcp
        - 443/tcp
        - 4789/udp

We also want to reload the firewall here as well.

    - name: reload service firewalld
      systemd:
        name: firewalld
        state: reloaded

We now want to enable containerd service and kubelet.

    - name: Enable containerd
      ansible.builtin.systemd_service:
        name: containerd.service
        state: started
        enabled: true

    - name: Enable Kubelet
      ansible.builtin.systemd_service:
        name: kubelet.service
        state: started
        enabled: true

The last thing we need to do is run the join command.

    - name: Add worker node
      shell: "{{ k8s_join_command }}  --ignore-preflight-errors=all"

We refer to the variable and add --ignore-preflight-errors=all because… well, sometimes it complains. This probably isn’t the best idea, but we can always kill the node if it isn’t behaving properly.

You can now run the playbooks:

\> ansible-playbook --inventory-file hosts initial-setup.yml --ask-vault-password

\> ansible-playbook --inventory-file hosts k8s-configuration.yml --ask-vault-password

CONGRATULATIONS! You now have a working k8s stack that can be deployed automatically! This can be hugely useful for deploying a dev stack.

Configure kubectl

You will need to copy the default template, so ssh onto the master host and run:

\> cp /etc/kubernetes/admin.conf ~/.kube/config
\> sudo chown $USER:$USER ~/.kube/config

. You can now run kubectl get nodes and you should see:

If you have kubectl locally, you can copy the config we made to your local and manage the host from your local shell. You will also need to add the hosts to /etc/hosts if you don’t have DNS name resolution for hosts on your network.

If you want the complete project, you can find it here: gitlab.com/godfrey.online

Sources:

linuxtechi.com deploy kubernetes cluster on rhel
- Referance for what needed to be configured.
tecadmin.net setup kubernetes cluster using ansible
- Used for insperation on how to copy the output of the command to the worker nodes.

Preface#

Terraform#

Ansible#

All Hosts#

Master Node#

Worker nodes#

Configure kubectl#