In this guide, I will show you how to automate the deployment of k8s on a XCP-NG server. We will use Terraform and Ansible to deploy and configure the hosts.

This guide still largely applies to other distributions of Linux, the main section you would have to change is the adding the repos and installing the packages. You will need to find the repos and the package names might be slightly different. If you search for Ansible repo/package management for your distribution, you should find it with little issue.

Terraform

Terraform has a built-in module for XCP-NG. A prerequisite to this is you need to install Rocky-Linux to a VM manually and then convert it to a template. When creating the template you will need to install the guest-tools as that will be required to get information from the host later.

You will also need a working Xen Orchestra instance for this to work.

  1. First, we need to set up the provider. Create provider.tf 
terraform {
  required_providers {
    xenorchestra = {
      source = "terra-farm/xenorchestra"
    }
  }
}

It’s now time to download the module. Run: terraform init. The output should look a little like this: image

  1. We now need to setup the data vars in vm.tf
data "xenorchestra_pool" "pool" {
  name_label = "HOSTNAME"
}

data "xenorchestra_template" "rocky_template" {
  name_label = "TEMPLATE NAME"
}

data "xenorchestra_sr" "sr" {
  name_label = "STORAGE NAME" 
  pool_id = data.xenorchestra_pool.pool.id
}

data "xenorchestra_network" "network" {
  name_label = "Pool-wide network associated with XXX"
  pool_id = data.xenorchestra_pool.pool.id
}

Replace the temp values with your values. The name_label can be a value with spaces in. I got confused when it came to storage, as “Local Storage” didn’t seem like it would be a valid input and expected XCP-NG to have some other value. Nevertheless, it was correct.

We now want to test this, but we need to provide credentials. I use a .env file in my Terraform directory. The contents of it is:

export XOA_URL=ws://XEN_ORCHESTRA_IP
export XOA_USER=USERNAME
export XOA_PASSWORD=PASSWORD

To add these values to your shell, just run eval $(cat .env). There is probably a better way to do this, but for now this will do.

You can now run terraform plan and you should see something like this: image

  1. If all is working, we can now get to the juicy part… creating the VMs. We want to start simple and add complexity as we go. Let’s start with:
resource "xenorchestra_vm" "k8s_master" {
  memory_max = 8589869056
  cpus = 2
  name_label = "K8S Master Node"
  template = data.xenorchestra_template.rocky_template.id

  network {
    network_id = data.xenorchestra_network.network.id
  }

  disk {
    sr_id = data.xenorchestra_sr.sr.id
    name_label = "k8s master node disk"
    size = 50212254720
  }
}

This is a supper simple VM setup.

  • resource is how you define a VM you want to create.
  • memory_max is how you define the max amount of ram you want the host to use.
  • cpus is how many cores you want.
  • name_label is just a label for the VM.
  • template is the template you want the VM to be created from.
  • network will create the interface with
    • network_id being the nic we want to use, which is what we defined earlier in the data section.
  • disk is the last part we define.
    • sr_id is the storage pool we defined in the data section.
    • name_label is the name of the disk.
  • size is how large we want the disk.

You can use what ever values you want here, but it is strongly recommended that you have at least 2 CPU cores and 8GB of ram.

We can now run terraform plan to see what it will create. If all went according to plan you can now run terraform apply and it should go ahead and create the VMs for you.

  1. It’s now time to create the worker nodes. We will want to copy the master node twice and change some values. This is for worker node one.
resource "xenorchestra_vm" "k8s_worker_node_1" {
  memory_max = 8589869056
  cpus = 2
  name_label = "K8S Worker Node 1"
  template = data.xenorchestra_template.rocky_template.id

  network {
    network_id = data.xenorchestra_network.network.id
  }

  disk {
    sr_id = data.xenorchestra_sr.sr.id
    name_label = "k8s worker node 1 disk"
    size = 50212254720
  }
}

Now you have two worker nodes, you can run terraform plan again. It is a good idea to always run plan before so you know what is going to happen, though apply dose give you a heads-up. If all is good, and it says it’s going to create two more VMs, you can run terraform apply. If it is getting confused you can always run terraform destroy to get back to the beginning, this isn’t an issue as we have nothing running yet.

If you are creating multiple of the same VMs you can use count but we will not go into that today.

  1. Awesome! Three VMs! All done with terraform, right? No, one thing that will make figuring out which host is which is setting the hostname. Terraform has something for that, it’s call provisioner using remote-exec. You will want to add something like this to each resource:
  connection {
    type     = "ssh"
    user     = "USER"
    password = "PASS"
    host     = self.ipv4_addresses[0]
  }

  provisioner "remote-exec" {
    inline = ["sudo hostnamectl set-hostname HOSTNAME.local"]
  }

You will also want to put wait_for_ip = true somewhere near the top of the resource block, otherwise terraform will try and fail before the VM has an IP.

To be clear, putting credentials into your terraform file isn’t the best idea. I want to limit the scope of this, so I have decided not to go through best practices.

image

Now we really are all good to go. If you have already run terraform apply would run terraform destroy and then terraform apply, just so we can ensure that it works from start to end.

After I have run it you should see it saying it has created 3 resources. image

When I go to my dashboard now, I see the new VMs we created: image

Ansible

Now we move over to setting up the host with Ansible. There is lots to define, so let’s get cracking.

All Hosts

  1. First, let’s create a hosts file. If you go into your XCP-NG, pull out the IPs and place them into here:
[k8s_all]
k8smaster.local ansible_ssh_host=X.X.X.X
k8sworker1.local ansible_ssh_host=X.X.X.X
k8sworker2.local ansible_ssh_host=X.X.X.X
[k8s_master]
k8smaster.local ansible_ssh_host=X.X.X.X
[k8s_worker]
k8sworker1.local ansible_ssh_host=X.X.X.X
k8sworker2.local ansible_ssh_host=X.X.X.X

This isn’t the cleanest file, and you can probably put groups under k8s_all, but I couldn’t figure out how. Nevertheless, this should be good.

  1. OK, unlike terraform, we are going to follow best practices with Ansible passwords (I promise this isn’t just because I don’t know best practices for terraform.) We want to create a vault.

Start by creating a dir called vars then run ansible-vault create vars/default.yml It will prompt you twice for a password. Now we can create our first playbook. To edit the file, you run ansible-vault edit vars/default.yml. You will want to add default_user_password: PASSWORD

Create a file called initial-setup.yml. We will want to start by defining the top of the file:

---
- name: Basic k8s node configuration
  hosts: k8s_all
  user: USER
  vars_files:
    - vars/default.yml
  vars:
    ansible_ssh_pass: "{{ default_user_password  }}"
  become: yes
  become_user: root
  become_method: sudo
  tasks:
  • name is a title for the job.
  • hosts allows you to limit what hosts this section will be run on. Here we are running it on all, but later we will use it to configure master nodes and worker nodes separately.
  • user is what user you want to connect as.
  • vars_files is specifying a variables file. In this case, it is our encrypted file.
  • vars is specifying what variables to use. In this case, the password used for ssh.
  • become is specifying if you want to become another user.
  • become_user is what user you want to become.
  • become_method is how you want to become that user. In our case, we are using sudo.
  • tasks is where you put what you want to run.

Remember to change the user to the user you setup

  1. Ok, now we can move onto what we want to run. To start we want to add our public key, this isn’t 100% essential but is a nice to have as we don’t want to have to enter our password each time. Add this:
    - name: Set authorized key taken from file
      ansible.posix.authorized_key:
        user: harley
        state: present
        key: "{{ lookup('file', '/home/USER/.ssh/id_rsa.pub') }}"
  1. Now we want to disable swap:
    - name: Turn off swap
      command: sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  1. Set SELinux to permissive.
    - name: Set SELinux to permissive
      ansible.posix.selinux:
        policy: targeted
        state: permissive
  1. We now want to create a dir called src/container.io. In that folder, we want to create containerd.conf with this as the content:
overlay
br_netfilter

We now want to add a section to transfer that to the hosts.

    - name: Copy container configuration
      ansible.builtin.copy:
        src: ./src/container.io/containerd.conf
        dest: /etc/modules-load.d/containerd.conf
        owner: root
        group: root
        mode: '0644'

We also want to ensure the modules are present. To do that add this:

    - name: Ensure moduales are present
      community.general.modprobe:
        name: "{{ item  }}"
        state: present
      loop:
        - overlay
        - br_netfilter
  1. Now we want to configure some network options required for k8s. Add this to your Ansible file:
    - name: Enable network settings for k8s
      ansible.posix.sysctl:
        name: "{{ item  }}"
        value: '1'
        state: present
      loop:
        - net.bridge.bridge-nf-call-iptables
        - net.ipv4.ip_forward
        - net.bridge.bridge-nf-call-ip6tables
  1. We now want to add Docker and K8S yum repos. Ansible has a built-in module ansible.builtin.yum_repository. Add this:
    - name: Add docker repo
      ansible.builtin.yum_repository:
        name: docker-ce-stable
        baseurl: https://download.docker.com/linux/centos/$releasever/$basearch/stable
        description: Docker Repo
        enabled: yes
        gpgcheck: yes
        gpgkey: https://download.docker.com/linux/centos/gpg

    - name: Add k8s repo
      ansible.builtin.yum_repository:
        name: kubernetes
        baseurl: https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
        description: k8s package
        enabled: yes
        gpgcheck: yes
        gpgkey: https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
  1. Next thing is to install the packages required.
    - name: Installing Packages
      yum:
        name:
          - containerd.io
          - vim
          - tree
          - git
          - kubelet
          - kubeadm
          - kubectl
        state: latest
  1. We now want to add the hosts to /etc/hosts. This is a requirement for k8s. Well, it is a requirment if you don’t have DNS resolution via DNS.
    - name: Get all hosts and their IP addresses
      set_fact:
        host_entries: |
          {% for host in groups['all'] %}
          {{ hostvars[host]['ansible_ssh_host'] }} {{ host }} {{ host.split('.')[0] }}
          {% endfor %}          

    - name: Append entries to /etc/hosts
      blockinfile:
        path: /etc/hosts
        block: |
          {{ host_entries }}          
        marker: "# {mark} ANSIBLE MANAGED BLOCK"
  1. We now want to configure containerd to use CGroups. Then we want to reboot the hosts to make sure all is applied.
    - name: Configure containerd
      shell: |
        containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
        sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml        

    - name: Reboot machines
      reboot:

Master Node

Now we have the basic configuration, we can move onto the master node. Here we will configure the firewall, enable k8s services and initialise kubelet.

  1. First we want to define the name of the section, become and other boilerplate stuff. I named this file k8s-configuration.yml
---
- name: Configure k8s Master node
  hosts: k8s_master
  user: USER
  become: yes
  become_user: root
  become_method: sudo
  tasks:

Here we use the hosts section to limit it to just the master nodes (what is specified in hosts file).

  1. We want to configure the firewall to open k8s ports along with some other useful ones.
    - name: Open ports in firewall
      ansible.posix.firewalld:
        port: "{{ item  }}"
        permanent: yes
        state: enabled
      loop:
        - 6443/tcp
        - 2379/tcp
        - 2380/tcp
        - 10250/tcp
        - 10251/tcp
        - 10252/tcp
        - 10257/tcp
        - 10259/tcp
        - 179/tcp
        - 22/tcp
        - 80/tcp
        - 443/tcp
        - 4789/udp

The firewalld module doesn’t have a built-in way to specify multiple ports, so we use a loop. Where we specify port we add "{{ item }}" which refers to the loop, and it will substitute the value with the current element in the loop. This can be really useful for keeping your playbooks neat.

We also want to reload the firewall. We can use the systemd module.

    - name: reload service firewalld
      systemd:
        name: firewalld
        state: reloaded
  1. Now we have the ports, we can finally get to setting up k8s. You will first want to start containerd which is once again a built-in Ansible module.
    - name: Enable containerd
      systemd_service:
        name: containerd.service
        state: started
        enabled: true

We can now run kubeadm init and start the kubelet service.

    - name: k8s init
      shell: sudo kubeadm init --control-plane-endpoint="{{ inventory_hostname }}" --ignore-preflight-errors=all --pod-network-cidr=192.168.50.0/16

You will want to ensure that the --pod-network-cidr doesn’t conflict with your local network, but that is unlikely with that CIDR.

We can now enable the kubelet service, once again with systemd module.

    - name: Enable Kubelet
      ansible.builtin.systemd_service:
        name: kubelet.service
        state: started
        enabled: true

This should be it! The master node should be up and running. I will show you later how to check on it.

Worker nodes

We will take a different approach on this host, calling it from the master node. This is so we can pass the join command over.

  1. You currently still want to be in the master node section. We want to run a command to print the join command. We can then pass that over to the worker section we will create after.
    - name: get join command
      shell: kubeadm token create --print-join-command
      register: join_command_raw

Here we run a command and use a register to store the value. We can then add a loop to call the worker node section (that we haven’t created yet).

    - name: Configure worker nodes 
      add_host:
        name: "{{ item  }}"
        k8s_join_command: "{{ join_command_raw.stdout  }}"
      loop: "{{ groups['k8s_worker'] }}"
  1. We now want to create the code for the worker section. You will want to add the typical boilerplate stuff.
- name: Configure k8s Worker nodes
  hosts: k8s_worker
  user: USER
  become: yes
  become_user: root
  become_method: sudo
  tasks:

The main thing to note is the hosts section is set to k8s_worker.

  1. We want to configure the firewall for the worker node. The worker node has fewer ports, but the Ansible code is the same (just with fewer ports specified).
    - name: Open ports in firewall
      ansible.posix.firewalld:
        port: "{{ item  }}"
        permanent: yes
        state: enabled
      loop:
        - 30000-32767/tcp
        - 10250/tcp
        - 179/tcp
        - 22/tcp
        - 80/tcp
        - 443/tcp
        - 4789/udp

We also want to reload the firewall here as well.

    - name: reload service firewalld
      systemd:
        name: firewalld
        state: reloaded
  1. We now want to enable containerd service and kubelet.
    - name: Enable containerd
      ansible.builtin.systemd_service:
        name: containerd.service
        state: started
        enabled: true

    - name: Enable Kubelet
      ansible.builtin.systemd_service:
        name: kubelet.service
        state: started
        enabled: true
  1. The last thing we need to do is run the join command.
    - name: Add worker node
      shell: "{{ k8s_join_command }}  --ignore-preflight-errors=all"

We refer to the variable and add --ignore-preflight-errors=all because… well, sometimes it complains. This probably isn’t the best idea, but we can always kill the node if it isn’t behaving properly.

You can now run the playbooks:

\> ansible-playbook --inventory-file hosts initial-setup.yml --ask-vault-password

image

\> ansible-playbook --inventory-file hosts k8s-configuration.yml --ask-vault-password

image

CONGRATULATIONS! You now have a working k8s stack that can be deployed automatically! This can be hugely useful for deploying a dev stack. With a bit (lot) of work, this code could also be used to deploy a production stack.

Configure kubectl

You will need to copy the default template:

\> cp /etc/kubernetes/admin.conf ~/.kube/config
\> sudo chown $USER:$USER ~/.kube/config

. You can now run kubectl get nodes and you should see:

image

If you have kubectl locally, you can copy the config we made to your local and manage the host from your local shell. You might need to add the hosts to /etc/hosts if you don’t have DNS name resolution for hosts on your network.

If you want the complete project, you can find it here: gitlab.com/godfrey.online

Sources: