Skip to main content

How This Project Came to Be

· 3 min read
vnwnv
vnwnv
Starbase Cluster Maintainer

How This Project Came to Be​

It all started when I purchased a dedicated server from Hetzner. I wanted to deploy Kubernetes on it. Since I only had one dedicated server, I had to consider virtualization. In this scenario, Proxmox VE was the best fit—open source, widely used, and much lighter and easier to maintain compared to OpenStack.

Some Issues​

Hetzner requires MAC address binding. I used a Linux Bridge and OpenWRT as a software router to manage the internal network of the virtual machines. I divided the network using VLANs and implemented a single-arm routing setup. As a result, you might notice the default DNS search path is set to .lan. I also used V2Ray/Xray for network proxy and authentication.

Previously, I had deployed a Debian-based k3s cluster. Over time, I found that the system required regular maintenance. Although automated tasks could be configured, manual intervention was still occasionally necessary. To reduce maintenance on the host system, I started looking for another operating system. I quickly discovered:

At first, I tried Talos, which was indeed very convenient and allowed for fast cluster deployment. However, I ran into issues when trying to mount Hetzner's SMB storage into the cluster. Due to Talos’s design, it lacks flexibility and is not ideal for "non-standard" clusters. Adding packages into its image was a cumbersome process. That said, Talos has many ideas worth learning from. I needed a distribution that fits my current usage scenario, with the following qualities:

  • Immutable infrastructure
  • Automatic maintenance and updates
  • Easy package management with good extensibility and a rich package ecosystem
  • A large user base, ensuring it remains maintained for years
  • If commercial, the parent company should have a good community reputation

So I chose openSUSE MicroOS. It uses BTRFS for atomic updates and has the largest number of packages among these options. It provides rolling updates in sync with openSUSE Tumbleweed, and can automatically roll back if something goes wrong. Moreover, it separates the /var partition, so there's no performance concern due to Copy-on-Write (CoW).

Cloud-init​

I chose cloud-init mainly because it has been around for a long time and is widely used. It integrates well with Proxmox VE and keeps the code simple. It also makes future migration to other virtualization platforms easier. I didn't use Ignition or Combustion primarily due to ecosystem support and integration complexity.

Multi-Server Allocation​

After everything was running smoothly on my own server, I started deploying Kubernetes for my team. Soon, I encountered the challenges of multi-server deployment. Deploying on a Proxmox VE cluster is quite different from a single-node setup. Without CEPH, I had to define VM allocation rules in Terraform. So I modified the original code to evenly distribute VM nodes across the cluster.

The First Two Versions​

After successfully deploying Kubernetes for my team, I realized this project might benefit others as well. I decided to open source it and move it to GitHub. Since some of the previous commit messages included links to my personal server, I had to create a new repository and archive the original one.

I also added usage documentation. That’s why the first release after moving to GitHub is version v0.3.0.

That’s the story behind this project. You're welcome to join the discussions on GitHub if you have any questions or thoughts!