My setup with Proxmox and Kubernetes

Coming from my homelab post, Proxmox Virtual Environment is a powerful hypervisor for my setup, PVE can be installed as the OS and replace Windows, Hyper-V in my new machines. And Kubernetes is a powerful container orchestration platform that helps to deploy and manage backend containers. In this post, I want to explain how I maximize hardware resources with this combo.

Setup

Proxmox is a Debian-based Linux distribution, I can deploy and manage virtualized environments through a web console or a command line. I used Etcher to flash a USB drive with Proxmox iso then Install Proxmox VE (Graphical). The detailed steps are easy to follow: How to Install Proxmox VE.

Kubernetes (k8s) is an open source system for automating deployment, scaling, and management of containerized applications. This guide helps with steps on How to Install Kubernetes on Ubuntu 22.04. Helm chart is the package manager for Kubernetes, it is really useful to install open source packages.

Machines structure

My hardware rigs are with Intel 2680v4 CPU and 128GB ECC RAM, the CPU has 14 cores and 28 threads. I break down a rig to 3 or 4 virtual machines with Proxmox, a VM has 4 vCPUs 16GB RAM or 8 vCPUs 32GB RAM, which are equivalent to AWS EC2 t3.xlarge and t3.2xlarge.

I have Ubuntu Server 22.04 on all VMs, use them to host backend services (PSQL, Mongo, Elastic, vLLM) and make each VM a k8s worker node. By using xlarge and 2xlarge config, a VM has enough resources for 1 or 2 BE services, and can mix with smaller BE containers using k8s. This is the key practice that helps to maximize my hardware resources.

Mornitoring Tools

There are many tools to monitor cluster health and resources.

Prometheus and Grafana: Prometheus is a popular tool to collect and query real-time metrics, and Grafana is useful to visualize metrics using graphs. I can Deploy Prometheus and Grafana on Kubernetes using Helm

Prometheus and Grafana will require Persistent Volume claims for their storage, using TrueNAS is the best way to achieve it, and Democratic CSI provides a convenient way to make connection from k8s pods to TrueNAS. Moving to TrueNAS and Democratic CSI for Kubernetes Persistent Storage is tough to set up but it works beautifully after the struggle.

Graylog is the portal for centralized logs, ELK stack is more popular but a bit overkill for my use, Graylog is simple to set up: Ubuntu Installation

Syslog library helps to collect system logs and send to Graylog: Setup syslog server on Ubuntu for Centralized Logs management

Fluent Bit helm chart helps to collect k8s logs and send to Graylog: Fluent Bit Kubernetes

Bonus

HiveOS is a popular Operating System for crypto miners, it is easy to manage GPUs and crypto algorithms though the cloud WebUI. HiveOS can be installed as a Proxmox Virtual Machine, by import the os image disk and config it as a USB drive. The setup steps are similar to Running Tails as a VM with persistence on Proxmox

Proxmox Private Network: I have a query to build 1 dedicated machine for a standalone Proxmox cluster, Proxmox private network is quite simple to set up an internal network for this niche purpose: How to Create a Private Network in Proxmox