Insights
Making sense of Proxmox bootloaders
Proxmox installer can be quite mysterious, it will try to support all kinds of systems, be it UEFI or BIOS and let you choose several very different filesystems on which the host system will reside. But on one popular setup - UEFI system without SecureBoot on ZFS - it will set you up, out of blue, with a different bootloader than all the others - and it is NOT blue - as GRUB would have been. This is, nowadays, completely unnecessary and confusing. UEFI or BIOS There are two widely known types of …
January 1, 2025
Why Proxmox offer full feature set for free
Proxmox VE has been available free of charge to download and run for a long time, which is one of the reasons it got so popular amongst non-commercial users, most of which are more than happy to welcome this offering. After all, the company advertises itself as a provider of “powerful, enterprise-grade solutions with full access to all functionality for everyone - highly reliable and secure”. Software license They are also well known to stand for “open source” software as …
December 26, 2024
A glimpse at Proxmox Quality Assurance
This post follows up on the previous finding that there is no difference in the eventual content no-subscription and test software repositories as publicly made available by Proxmox. Routine Every software house has some sort of testing routine (QA) to ensure the obviously bad versions of their packages never reach their user. It starts with rudimentary unit tests that a developer is supposed to make and have accompany their newly written code, these would also help find out any regressions - …
December 26, 2024
The 'no-subscription' repository
Proxmox repositories are not all the same. The enterprise one “contains the most stable packages and is suitable for production use”, whilst the no-subscription one “can be used for testing and non-production use”, however this is often borderline contradicted on their own Community Forum where home users are expected to “just use the no-subscription repo” and indeed, there’s also the test repository which “contains the latest packages and is …
December 26, 2024
How Proxmox VE shreds your SSDs
Time has come to revisit the initial piece on inexplicable writes that even empty Proxmox VE cluster makes, especially we have already covered what we are looking at: a completely virtual filesystem with a structure that is completely generated on-the-fly, some of which never really exists in any persistent state - that is what lies behind the mountpoint of /etc/pve and what the process of pmxcfs created the illusion of. We know how to set up our own cluster probe that the rest of the cluster …
December 20, 2024
The pmxcfs mountpoint of /etc/pve
This post will provide superficial overview of the Proxmox cluster filesystem, also dubbed pmxcfs that goes beyond the official terse: a database-driven file system for storing configuration files, replicated in real time to all cluster nodes Most users would have encountered it as the location where their guest configurations are stored and simply known by its path of /etc/pve. Mountpoint Foremost, it is important to understand that the directory itself as it resides on the actual system disk …
December 8, 2024
Some lesser known quorum options
Proxmox do not really cater much for cluster deployments at a small scale of 2-4 nodes and always assume High Availability could be put to use in their approach to the out-of-the-box configuration. It is very likely for this reason that some great features of Corosync configuration are left out of the official documentation entirely. Tip You might want to read more on how Proxmox utilise Corosync in a separate post. Quorum provider service Proxmox need a quorum provider service votequorum to …
December 1, 2024
The Proxmox time bomb watchdog
The title of this post is inspired by the very statement of “[watchdogs] are like a loaded gun” from Proxmox wiki. Proxmox include one such active-by-default tool on every single node anyway. There’s further misinformation, including on official forums, when watchdogs are “disarmed” and it is thus impossible to e.g. isolate genuine non-software related reboots. Design flaws might get your node auto-reboot with no indication in the GUI. The CLI part is undocumented …
November 21, 2024
The Proxmox Corosync fallacy
Unlike some other systems, Proxmox VE does not rely on a fixed master to keep consistency in a group (cluster). The quorum concept of distributed computing is used to keep the hosts (nodes) “on the same page” when it comes to cluster operations. The very word denotes a select group - this has some advantages in terms of resiliency of such systems. The quorum sideshow Is a virtual machine (guest) starting up somewhere? Only one node is allowed to spin it up at any given time and while …
November 16, 2024
Improved SSH with hidden regressions
If you pop into the release notes of PVE 8.2, there’s a humble note on changes to SSH behaviour under Improved management for Proxmox VE clusters: Modernize handling of host keys for SSH connections between cluster nodes ([bugreport] 4886). Previously, /etc/ssh/ssh_known_hosts was a symlink to a shared file containing all node hostkeys. This could cause problems if conflicting hostkeys appeared in /root/.ssh/known_hosts, for example after re-joining a node to the cluster under its old …
November 10, 2024
Passwordless SSH can lock you out
If you follow standard security practices, you would not allow root logins, let alone connections over SSH (as with Debian standard install). But this would deem your PVE unable to function properly, so you can only resort to fix your /etc/ssh/sshd_config with the option: PermitRootLogin prohibit-password That way, you only allow connections with valid keys (not password). Prior to this, you would have copied over your public keys with ssh-copy-id or otherwise add them to …
November 8, 2024
Why Proxmox VE shreds your SSDs
Tip This post has a more detailed follow-up available here. You must have read, at least once, that Proxmox recommend “enterprise” SSDs for their virtualisation stack. But why does it shred regular SSDs? It would not have to, in fact the modern ones, even without PLP, can endure as much as 2,000 TBW per life. And where do the writes come from? ZFS? Let’s have a look. The below is particularly of interest for any homelab user, but in fact everyone who cares about wasted system …
November 3, 2024