The installation of a supported Proxmox VE server should be done via bare-metal ISO installer. In some cases it makes sense to install Proxmox VE on top of a running Debian Bullseye 64-bit, especially if you want a custom partition layout. For this How-To any official Bullseye installation medium should work.
For this Demo, I am installing 3 KVM VM’s. As you can do VM nesting. Also, you can do this on Vmware or Hyper V. Here is the Architecture.
I reciently install proxmox on Debian and the install failed. Here is the error:
Hit:1 http://download.proxmox.com/debian/pve buster InRelease
Hit:2 http://security.debian.org/debian-security buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster InRelease
Hit:4 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
root@px1:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up pve-manager (6.3-3) ...
Job for pveproxy.service failed because the control process exited with error code.
See "systemctl status pveproxy.service" and "journalctl -xe" for details.
dpkg: error processing package pve-manager (--configure):
installed pve-manager package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of proxmox-ve:
proxmox-ve depends on pve-manager; however:
Package pve-manager is not configured yet.
dpkg: error processing package proxmox-ve (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)
The resolution was to make sure the network was set to a static IP:
# The primary network interface
iface enp4s0 inet static
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 184.108.40.206 220.127.116.11
The second issue was to make sure there was a host entry. Comment out the 127.0.0.1 and make sure to use the full host.domain host as below. Leaving out the end “px1” caused the issue.
After updating/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers. Ceph reports a non-zero exit status from zabbix_sender.
Check to install zabbix-server.
apt install zabbix-sender
# ceph mon versions
"ceph version 16.2.7 (f9aa029788115b5df5eeee328f584156565ee5b7) pacific (stable)": 4
Check Debian version
# cat /etc/debian_version
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Failed to send data to Zabbix
mon px1 is low on available space
mon: 3 daemons, quorum px1,px2,px3 (age 8d)
mgr: px1(active, since 8d), standbys: px2, px3
osd: 32 osds: 32 up (since 8d), 32 in (since 11d)
pools: 7 pools, 640 pgs
objects: 3.52M objects, 13 TiB
usage: 53 TiB used, 124 TiB / 176 TiB avail
pgs: 639 active+clean
client: 0 B/s rd, 23 MiB/s wr, 0 op/s rd, 34 op/s wr
ceph health detail
HEALTH_WARN Failed to send data to Zabbix; mon px1 is low on available space
MGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix
/usr/bin/zabbix_sender exited non-zero: zabbix_sender : DEBUG: answer 
zabbix_sender : Warning: incorrect answer from server 
Setting “debug mgr = 20” yields no additional information. Zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4:
The Zabbix plugin actively sends information to a Zabbix server like:
The plugin requires that the zabbix_sender executable is present on all machines running ceph-mgr. It can be installed on most distributions using the package manager.
Installing zabbix_sender can be done under Ubuntu or CentOS using either apt or dnf.
On Ubuntu Xenial:
# apt install zabbix-agent
# dnf install zabbix-sender
You can enable the zabbix module with:
# ceph mgr module enable zabbix
Two configuration keys are vital for the module to work:
The parameter zabbix_host controls the hostname of the Zabbix server to which zabbix_sender will send the items. This can be a IP-Address if required by your installation. The identifier parameter controls the identifier/hostname to use as source when sending items to Zabbix. This should match the name of the Host in your Zabbix server. When the identifier parameter is not configured the ceph- of the cluster will be used when sending data to Zabbix.
This would for example be ceph-c4d32a99-9e80-490f-bd3a-1d22d8a7d354
Additional configuration keys which can be configured and their default values:
Configuration keys can be set on any machine with the proper cephx credentials, these are usually Monitors where the client.admin key is present.
ceph zabbix config-set
Since our host acts as a router we have to make sure it’s kernel has all IP packet forwarding features activated. Take a look at ‘/etc/sysctl.conf’ and make sure that the following two lines aren’t commented out:
Lastly make sure your host won’t send ICPM “redirect” messages to guests, telling them to find the gateway by themselves. This won’t work with our particular network setup. Add the following to ‘/etc/sysctl.conf’:
Networking for nodes Incomplete – this is not currently working):
Set up 3 NIC’s
ens192 – primary for Public IP. This is the public IP for the installation.
ens224 – proxmox VE Cluster Network
ens256 – Ceph Install
Once installed, edit /etc/network/interfaces on all 3 nodes. Change the IP addresses accordingly. Remeber to backup the files before editing.
Example /etc/network/interfaces for px1 node 1.
iface lo inet loopback
iface ens192 inet static
post-up echo 1 > /proc/sys/net/ipv4/conf/ens192/proxy_arp
iface vmbr0 inet static
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.1.0/24' -o ens192 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.1.0/24' -o ens192 -j MASQUERADE
#Private Network for VM Creation
iface ens224 inet static
#Proxmox VE Cluster Network
iface ens256 inet static
Setting up the cluster
Hosts file edits. Backup hosts file and change to this on all nodes. This is not the hostname but an identifier.
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/urandom.
Writing corosync key to /etc/corosync/authkey.
Writing corosync config to /etc/pve/corosync.conf
Restart corosync and cluster filesystem
From man corosync.conf
ringnumber – This specifies the ring number for the interface. When using the redundant ring protocol, each interface should specify separate ring numbers to uniquely identify to the membership protocol which interface to use for which redundant ring. The ringnumber must start at 0.
Adding nodes. Login to a node you want to add, node2, node3, node4, etc. In this example, login to node 2 (10.0.0.152)and the IP for node1 is 10.0.0.151.
Run the following where”