The 'no-subscription' repository

Proxmox repositories are not all the same. ¹ The enterprise one “contains the most stable packages and is suitable for production use”, whilst the no-subscription one “can be used for testing and non-production use”, however this is often borderline contradicted on their own Community Forum ² where home users are expected to “just use the no-subscription repo” ³ and indeed, there’s also the test repository which “contains the latest packages and is primarily used by developers to test new features.”

There was a concern once that the (then new) “no-subscription” repository was exactly this, ⁴ but such times are long gone.

Packages for test

It is quite straightforward to find out what is in which package in terms of newly submitted code - the Git repositories of Proxmox ⁵ always contain specific “bump” commit message upon each package built - at that very point.

Let’s have a look at the kernel repository. We will use pure apt ⁶ and git, ⁷ command-line utilities, but you can check also online: ⁸

apt install -y git
git clone git://git.proxmox.com/git/pve-kernel.git
cd pve-kernel
git log --pretty=format:'%H %cr %s'

3f4d88534cd080507a65c1de4bf452fbea1215a7 6 days ago d/scripts: add --full-index parameter to export-patchqueue
24bfa444380036339f0043be8daa8540efea6285 6 days ago backport fix for kvm performance regression with Intel Emerald Rapids
f3ec9c1f627db2e8ae3c516fd42a2e466adbfbee 6 days ago patches: kernel: switch to using full index for patch files
6fe4c2cb8d34567b975d1166baa90df5c93b7016 3 weeks ago update ABI file for 6.11.0-2-pve
e9bbc10267e730a1dcac03ca1be2af9afb86b190 3 weeks ago update firmware list
f4e8296418f29283f24eddcb4c001e2f899e23e2 3 weeks ago bump version to 6.11.0-2
a872f278b9c4b0574d900feef5a615f7b81e84aa 3 weeks ago rebase patches on top of Ubuntu-6.11.0-13.14
b50ca75ce28bb57fa17c097555a940cb129ebc79 3 weeks ago update sources to Ubuntu-6.11.0-13.14
d4dcf2b536cc84e76d2d3f8d42b866cd47034e26 4 weeks ago revert Ubuntu patch disabling IOMMU functionality for Skylake iGPU
21160ccd404b829d1d3aacf4a93745946c8afdba 4 weeks ago fix #5926: cherry-pick ACS-quirk fix from linux-stable/6.11.7
2519c2669d3f6e5c521415b87022056a885eaf18 7 weeks ago d/rules: enable CONFIG_MEMCG_V1 needed for legacy containers
a681faec23eeb4a4f116f59988b0a84b6785cdf5 7 weeks ago cherry-pick fix mitigating host reboot issue affecting certain AMD Zen4 CPU models
035534f88e9d435494e060b179246ac121024aaf 9 weeks ago update ABI file for 6.11.0-1-pve
bd86b34a040881e5a1df6d80773530d977318136 9 weeks ago bump version to 6.11.0-1
e4a79ff8123852e9456bdde9ad12905f032a23c6 9 weeks ago update fwlist for 6.11.0
ecf3bf697db4d7c50a98e23fe8117e508d385902 9 weeks ago update ZFS submodule to include 6.11 compat patches
560cce22a751ae3b3ee83da56852eac1af7ae694 9 weeks ago update submodule and rebase patches for Oracular 6.11 based kernel
---8<---

Now let’s check our freshly installed system on no-subscription repository:

apt update
apt-cache search proxmox-kernel

The latest is 6.11.0-2:

---8<---
proxmox-kernel-6.11.0-2-pve-signed-template - Template for signed kernel package
proxmox-kernel-6.11.0-2-pve-signed - Proxmox Kernel Image (signed)
proxmox-kernel-6.11.0-2-pve - Proxmox Kernel Image
---8<---

Based on the “bump” commits from the development this is 3 weeks old and available to us.

To be fair here, it’s not going to be default kernel we get though:

apt depends proxmox-default-kernel

proxmox-default-kernel
  Depends: proxmox-kernel-6.8

But when we swap our APT sources for test repository, we get exactly the same result.

We can check when it was packaged - with ar: ⁹

apt download proxmox-kernel-6.11.0-2-pve
ar tv proxmox-kernel-6.11.0-2-pve_6.11.0-2_amd64.deb

---8<---
rw-r--r-- 0/0      4 Dec  4 10:29 2024 debian-binary
rw-r--r-- 0/0 161368 Dec  4 10:29 2024 control.tar.xz
rw-r--r-- 0/0 105916668 Dec  4 10:29 2024 data.tar.xz

And it’s exactly the same for both repositories.

It also matches the Git “bump” commit message from above: ¹⁰

git show f4e8296418f29283f24eddcb4c001e2f899e23e2

commit f4e8296418f29283f24eddcb4c001e2f899e23e2
Author: ---8<---
Date:   Wed Dec 4 11:29:45 2024 +0100

    bump version to 6.11.0-2
 
---8<---

 # increment KREL for every published package release!

---8<---

+  * updated sources to Ubuntu-6.11.0-13.14

---8<---

That’s exactly the same time (our system is on UTC/GMT) and the message also divulges there’s a new version number for every package build.

Versioning

There’s different ways to approach versioning of releases of software, but a major rule is to never label any two different builds the same, otherwise there would be confusion which build of “the same version” runs somewhere. One can create specially designated “test builds” and then simply make those that are “good enough” for public release available as rebuilt (or at least re-packaged) under different versioning scheme. Or - as in this case - builds are versioned uniformly and incrementally, but then only the “good ones” get to be released.

Let’s check what versions were available previously, on no-subscription - we will use grep ¹¹ and sort ¹² for convenience:

apt-cache pkgnames proxmox-kernel | grep pve$ | sort -V

proxmox-kernel-6.2.16-6-pve
proxmox-kernel-6.2.16-8-pve
proxmox-kernel-6.2.16-9-pve
proxmox-kernel-6.2.16-10-pve
proxmox-kernel-6.2.16-11-pve
proxmox-kernel-6.2.16-12-pve
proxmox-kernel-6.2.16-13-pve
proxmox-kernel-6.2.16-14-pve
proxmox-kernel-6.2.16-15-pve
proxmox-kernel-6.2.16-16-pve
proxmox-kernel-6.2.16-17-pve
proxmox-kernel-6.2.16-18-pve
proxmox-kernel-6.2.16-19-pve
proxmox-kernel-6.2.16-20-pve
proxmox-kernel-6.5.3-1-pve
proxmox-kernel-6.5.11-1-pve
proxmox-kernel-6.5.11-2-pve
proxmox-kernel-6.5.11-3-pve
proxmox-kernel-6.5.11-4-pve
proxmox-kernel-6.5.11-5-pve
proxmox-kernel-6.5.11-6-pve
proxmox-kernel-6.5.11-7-pve
proxmox-kernel-6.5.11-8-pve
proxmox-kernel-6.5.13-1-pve
proxmox-kernel-6.5.13-2-pve
proxmox-kernel-6.5.13-3-pve
proxmox-kernel-6.5.13-4-pve
proxmox-kernel-6.5.13-5-pve
proxmox-kernel-6.5.13-6-pve
proxmox-kernel-6.8.1-1-pve
proxmox-kernel-6.8.4-1-pve
proxmox-kernel-6.8.4-2-pve
proxmox-kernel-6.8.4-3-pve
proxmox-kernel-6.8.4-4-pve
proxmox-kernel-6.8.8-1-pve
proxmox-kernel-6.8.8-2-pve
proxmox-kernel-6.8.8-3-pve
proxmox-kernel-6.8.8-4-pve
proxmox-kernel-6.8.12-1-pve
proxmox-kernel-6.8.12-2-pve
proxmox-kernel-6.8.12-3-pve
proxmox-kernel-6.8.12-4-pve
proxmox-kernel-6.8.12-5-pve
proxmox-kernel-6.8.12-6-pve
proxmox-kernel-6.11.0-1-pve
proxmox-kernel-6.11.0-2-pve

It’s been a long time that some were skipped, e.g. 6.2.16-7 is not there.

Now we do the same with test repository and … get the same result, exactly same was skipped there as well.

We do NOT know since when the packages have been made available in which repository, but since they hold the same packages, this would mean the test builds were all good enough to proceed to be potentially installed by all users, presumably shortly after passing some “good enough” check. They would need to be incredible quality (all 45 of them), i.e. test repository would be almost a mirror of the no-subscription one and presumably just used to make previously already well-tested software available a little earlier to some than to most.

And indeed, there was an announcement on the Community Forum ¹³ when kernel 6.8 was “opt-in” via this route - April 5, 2024 - and about to become “new default kernel in the next Proxmox VE 8.2 point release (Q2'2024).” And PVE v8.2 was then released April 24, 2024. ¹⁴

It is also notable, that kernel 6.11 that was announced similarly ¹⁵ - November 2, 2024 - and which will NOT become a default for v8, even since 2 months, has fraction of the “bugreports” in the announcement thread.

But perhaps this is not fair, kernels can be swapped back easily when things go wrong. And it is also well supported by Proxmox tooling. ¹⁶

Internal testing

Let’s have a look at the difference between test and no-subscription of some regular package, such as pve-manager:

apt list -a pve-manager

pve-manager/stable 8.3.2 all [upgradable from: 8.3.1]
pve-manager/stable,now 8.3.1 all [installed,upgradable to: 8.3.2]
pve-manager/stable 8.3.0 all
pve-manager/stable 8.2.10 all
pve-manager/stable 8.2.9 all
pve-manager/stable 8.2.8 all
pve-manager/stable 8.2.7 all
pve-manager/stable 8.2.6 all
pve-manager/stable 8.2.5 all
pve-manager/stable 8.2.4 all
pve-manager/stable 8.2.3 all
pve-manager/stable 8.2.2 all
pve-manager/stable 8.1.11 all
pve-manager/stable 8.1.10 all
pve-manager/stable 8.1.8 all
pve-manager/stable 8.1.5 all
pve-manager/stable 8.1.4 all
pve-manager/stable 8.1.3 all
pve-manager/stable 8.0.9 all
pve-manager/stable 8.0.8 all
pve-manager/stable 8.0.7 all
pve-manager/stable 8.0.6 all
pve-manager/stable 8.0.5 all
pve-manager/stable 8.0.4 all
pve-manager/stable 8.0.3 all
pve-manager/stable 8.0.0~9 all
pve-manager/stable 8.0.0~8 all

This is the same result from both repositories. All the same package versions were available.

Let’s match them against Git repositories: ¹⁷

git clone git://git.proxmox.com/git/pve-manager.git
git -C pve-manager log --pretty=format:'%s %cr' | grep "^bump version"

bump version to 8.3.2 10 days ago
bump version to 8.3.1 4 weeks ago
bump version to 8.3.0 5 weeks ago
bump version to 8.2.10 5 weeks ago
bump version to 8.2.9 5 weeks ago
bump version to 8.2.8 9 weeks ago
bump version to 8.2.7 3 months ago
bump version to 8.2.6 3 months ago
bump version to 8.2.5 4 months ago
bump version to 8.2.4 7 months ago
bump version to 8.2.3 7 months ago
bump version to 8.2.2 8 months ago
bump version to 8.2.1 8 months ago
bump version to 8.2.0 8 months ago
bump version to 8.1.11 8 months ago
bump version to 8.1.10 9 months ago
bump version to 8.1.9 9 months ago
bump version to 8.1.8 9 months ago
bump version to 8.1.7 9 months ago
bump version to 8.1.6 10 months ago
bump version to 8.1.5 10 months ago
bump version to 8.1.4 12 months ago
bump version to 8.1.3 1 year, 1 month ago
bump version to 8.1.2 1 year, 1 month ago
bump version to 8.1.1 1 year, 1 month ago
bump version to 8.1.0 1 year, 1 month ago
bump version to 8.0.9 1 year, 1 month ago
bump version to 8.0.8 1 year, 1 month ago
bump version to 8.0.7 1 year, 2 months ago
bump version to 8.0.6 1 year, 4 months ago
bump version to 8.0.5 1 year, 4 months ago
bump version to 8.0.4 1 year, 5 months ago
bump version to 8.0.3 1 year, 6 months ago
bump version to 8.0.2 1 year, 6 months ago
bump version to 8.0.1 1 year, 6 months ago
bump version to 8.0.0 1 year, 6 months ago
bump version to 8.0.0~9 1 year, 6 months ago
bump version to 8.0.0~8 1 year, 7 months ago
bump version to 8.0.0~7 1 year, 7 months ago
bump version to 8.0.0~6 1 year, 7 months ago
bump version to 8.0.0~5 1 year, 7 months ago
bump version to 8.0.0~4 1 year, 7 months ago
bump version to 8.0.0~3 1 year, 7 months ago
bump version to 8.0.0~2 1 year, 7 months ago
bump version to 8.0.0~1 1 year, 7 months ago

---8<---

So this looks “better” - some of the versions did NOT make it to the public repositories, at all.

There’s some testing before it even reaches the test repository.

But only Proxmox know how much of such testing and for what duration, before it makes it to the test repository, and eventually becoming publicly available to non-subscribers. And they won’t say much on this beyond that there “is no fixed schedule, packages go into the public test repositories once they have cleared internal testing.” ¹⁸

Enterprise repository

You are at will to explore your enterprise repository in the same manner as we did above for the other two. You would find which package versions where skipped and how much delay there was between they “caught up” with the most recent development. The reason it cannot be exactly publicly covered in detail in a post lies in the licensing terms of Proxmox subscriptions: ¹⁹

(Re-)Distributing Software packages received under this Subscription Agreement to a third party or using any of the subscription services for the benefit of a third party is a material breach of the agreement. Even though the open-source license applicable to individual software packages may give you the right to distribute those packages (this limitation is not intended to interfere with your rights under those individual licenses).

As vaguely as it walks the line between keeping the packages under a free license and keeping a lid on which those “good enough” packages are, it would arguably qualify the disclosure of what exact versions are currently in the repository as using the services (access to the list being such) “for the benefit of a third party.”

Conclusion

There really is no difference between packages that are eventually available in the test and no-subscription repository and you can also learn more about how Proxmox implement their quality assurance the packages go through before becoming public separately.

There’s certain steps that can be taken to avoid bad surprises, for instance kernel pinning or putting packages on hold, however this would need one to rigorously follow what the reasons for each new package release were - a mere enhancement, an ordinary bugfix, or a critical vulnerability patch. Some of it can be mitigated by keeping your PVE instance not exposed to the outside world. This obviously only helps you if you are not hosting 3rd party guests, at which point the attack vectors are harder to control.