DevMemo – proxmox

Blog: Automate Proxmox VM Creation With Cloud-Init

Mon, 06 Feb 2023 15:10:26 -0800

This post gives you step-by-step instructions of how to automate proxmox VM creation. I’ll use ubuntu as an example.

Overview

The overall strategy is to use the ubuntu cloud image to create a Proxmox VM template, and attach a cloud-init disk to it. After that, we can clone the template to create new VM, and new VM will automatically bootstrap itself, e.g., changing the hostname.

Prepare tools

Install virt-customize

apt-get install libguestfs-tools

Prepare The Ubuntu Cloud Image

Use the following commands to prepare the ubuntu cloud image:

# Create a new directory for our image building.
mkdir ubuntu-cloud-image
cd ubuntu-cloud-image

# Download the cloud image.
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

# Resize the image to 16GB. Feel free to change it to your desired size.
qemu-img resize noble-server-cloudimg-amd64.img 16G

# Install avahi-deamon for mdns.
virt-customize -a noble-server-cloudimg-amd64.img --run-command 'apt update && apt install avahi-daemon -y && cloud-init clean --logs --machine-id --configs all --reboot && sudo truncate -s 0 /etc/machine-id /var/lib/dbus/machine-id'

Prepare The Proxmox VM Template

Then, we prepare the Proxmox VM template with the following steps.

Create A Temporary VM

# Create a VM with the following configuration:
# * vmid: 1000 (feel free to change this)
# * memory: 4GB
# * network: attached to bridge vmbr0
qm create 1000 --name ubuntu-noble-template --memory 4096 --net0 virtio,bridge=vmbr0

Attach The Ubuntu Cloud Image

# Attach the ubuntu cloud image onto the VM.
# Note that here ssd is my storage name. You'll need to replace it with yours.
qm importdisk 1000 noble-server-cloudimg-amd64.img ssd

# Make the attached disk scsi0.
qm set 1000 --scsihw virtio-scsi-pci --scsi0 ssd:vm-1000-disk-0

# Set scsi0 as the first boot device.
qm set 1000 --boot c --bootdisk scsi0

Create A Cloud-init Drive

# Create a cloud-init drive.
qm set 1000 --ide2 ssd:cloudinit

# Configure your cloud-init drive. Replace the username, password and sshkeys for your case.
qm set 1000 --ciuser swe --cipassword swe-passwd --sshkeys ~/.ssh/id_rsa.pub --ipconfig0 ip=dhcp

Convert The VM Into A Template

# Convert the VM into a template.
qm template 1000

Create VM

# Create a VM using the template, and set the name to ubuntu-vm1.
qm clone 1000 1101 --full --name ubuntu-vm1

# Create another VM using the template, and set the name to ubuntu-vm2.
qm clone 1000 1102 --full --name ubuntu-vm2

To access these VMs, you can do

# Use the following command to access ubuntu-vm1 if you have mdns included in your template, and you are within the same LAN as ubuntu-vm1.
ssh swe@ubuntu-vm1.local

# Use the following command to access ubuntu-vm2 if you have mdns included in your template, and you are within the same LAN as ubuntu-vm2.
ssh swe@ubuntu-vm2.local

Blog: Using LVM For Proxmox

Thu, 19 Jan 2023 15:02:22 -0800

If you are trying to use Proxmox to set up a homelab for the first time, it must be very confusing to see all the storage related terms, e.g., LVM, LVM-Thin. This post talks about everything you need to know to understand how Proxmox uses LVM as its storage infrastructure.

LVM Basics

LVM (Logical Volume Management) is actually a Linux technology. It is not Proxmox proprietary. Proxmox can leverage LVM as its storage infrastructure, but LVM itself is not part of Proxmox.

Let’s try to understand the basics of LVM, and then we try to understand how Proxmox uses it.

What Is LVM?

The overall goal of LVM is to create a virtual storage layer on top of the physical storage layer, so that the storage usage can be more flexible.

For example, if you have two 1TB disks, but you want to use them together as your root directory. LVM provides a solution to this.

How Does LVM Work?

The following image presents the overall architecture of LVM. (The image comes from wikipedia)

There are a couple of layers involved (from bottom to top):

On the bottom layer, we have the physical disk itself. These are your hard drives.
Above the hard drives, we have partitions. Each hard drive is splitted into a few partitions.
PV (physical volume) is a LVM concept. It is a representation of a physical volume. It is mapped to a single disk partition.
VG (volume group) is also a LVM concept. A VG contains a couple of PVs.
LV (logical volume) is a logical volume, which is basically a virtual volume.
On top of each LV, a file system can be created, and the LV device can be mounted to an accessible directory.

In this design, logical volume does not have to tie to a single physical disk. Logical volume can be created over multiple physical disks.

Installation LVM

On debian and ubuntu, you can do

sudo apt install lvm2

How To Use LVM?

Here are some commands to manage PV, VG and LV.

# Create a PV
pvcreate <disk-device-name>

# Remove a PV
pvremove <disk-device-name>

# List all PVs
pvs

# Create a VG
vgcreate <vg-name> <disk-device-name>

# Remove a VG
vgremove <vg-name>

# List all VGs
vgs

# Create a LV
lvcreate -L <lv-size> -n <lv-name> <vg-name>

# Remove a LV
lvremove <vg-name>/<lv-name>

# List all LVs
lvs

An Example

For example, I have a empty /dev/sdb device,

swe@ubuntu-server:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 63.3M 1 loop /snap/core20/1778
loop1 7:1 0 141.4M 1 loop /snap/docker/2285
loop2 7:2 0 55.6M 1 loop /snap/core18/2667
loop3 7:3 0 55.4M 1 loop /snap/core18/2066
loop5 7:5 0 67.6M 1 loop /snap/lxd/20326
loop6 7:6 0 49.6M 1 loop /snap/snapd/17883
loop7 7:7 0 49.8M 1 loop /snap/snapd/17950
loop8 7:8 0 91.9M 1 loop /snap/lxd/24061
loop9 7:9 0 139.4M 1 loop /snap/docker/2343
sda 8:0 0 32G 0 disk
├─sda1 8:1 0 1M 0 part
└─sda2 8:2 0 32G 0 part /
sdb 8:16 0 32G 0 disk
sr0 11:0 1 4M 0 rom
sr1 11:1 1 1024M 0 rom
swe@ubuntu-server:~$

let’s create a PV on it

swe@ubuntu-server:~$ sudo pvcreate /dev/sdb
 Physical volume "/dev/sdb" successfully created.
swe@ubuntu-server:~$ sudo pvs
 PV VG Fmt Attr PSize PFree
 /dev/sdb lvm2 --- 32.00g 32.00g
swe@ubuntu-server:~$

and then create a VG on the PV

swe@ubuntu-server:~$ sudo vgcreate test-vg /dev/sdb
 Volume group "test-vg" successfully created
swe@ubuntu-server:~$ sudo vgs
 VG #PV #LV #SN Attr VSize VFree
 test-vg 1 0 0 wz--n- <32.00g <32.00g
swe@ubuntu-server:~$

and then create a LV on the VG

swe@ubuntu-server:~$ sudo lvcreate -L 10G -n test-lv test-vg
 Logical volume "test-lv" created.
swe@ubuntu-server:~$ sudo lvs
 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
 test-lv test-vg -wi-a----- 10.00g
swe@ubuntu-server:~$

Now we can create a file system on the LG, and mount it for real usage:

swe@ubuntu-server:~$ sudo mkfs.ext4 /dev/test-vg/test-lv
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 5c329fcf-76ec-450a-8d96-dfb816538e3e
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

swe@ubuntu-server:~$ cd /
swe@ubuntu-server:/$ sudo mkdir mount
swe@ubuntu-server:/$ sudo mount /dev/test-vg/test-lv /mount
swe@ubuntu-server:/$ cd mount/
swe@ubuntu-server:/mount$ ls
lost+found
swe@ubuntu-server:/mount$

Now, we have an ext4 filesystem on the LV:

swe@ubuntu-server:/mount$ lsblk -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
loop0 squashfs 0 100% /snap/core20/1778
loop1 squashfs 0 100% /snap/docker/2285
loop2 squashfs 0 100% /snap/core18/2667
loop3 squashfs 0 100% /snap/core18/2066
loop5 squashfs 0 100% /snap/lxd/20326
loop6 squashfs 0 100% /snap/snapd/17883
loop7 squashfs 0 100% /snap/snapd/17950
loop8 squashfs 0 100% /snap/lxd/24061
loop9 squashfs 0 100% /snap/docker/2343
sda
├─sda1
└─sda2 ext4 761b4178-9020-4642-ac79-d172a2a0a06d 21G 28% /
sdb LVM2_member biMIK2-WI2D-ypLl-tVGw-p8hZ-CynW-vo8swy
└─test--vg-test--lv ext4 5c329fcf-76ec-450a-8d96-dfb816538e3e 9.2G 0% /mount
sr0 iso9660 cidata 2023-01-12-13-13-29-00
sr1

How Does Proxmox Use LVM?

Now you have a basic understanding of LVM. So how does Proxmox use LVM as its storage infrastructure?

In a few words, Proxmox uses LVs for two purposes:

Use LVs as the client OS disks.
Use a LV as the host OS root directory.

For example,

root@host3:~$ pvs
 PV VG Fmt Attr PSize PFree
 /dev/nvme0n1p3 ssd lvm2 a-- <1.82t 738.50g
root@host3:~$
root@host3:~$ vgs
 VG #PV #LV #SN Attr VSize VFree
 ssd 1 13 0 wz--n- <1.82t 738.50g
root@host3:~$
root@host3:~$ lvs
 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
 root ssd -wi-ao---- 96.00g
 vm-1001-disk-0 ssd -wi-ao---- 16.00g
 vm-2000-cloudinit ssd -wi-ao---- 4.00m
 vm-2000-disk-0 ssd -wi-ao---- 16.00g
 vm-2002-cloudinit ssd -wi-ao---- 4.00m
 vm-2002-disk-0 ssd -wi-ao---- 16.00g
 vm-5000-disk-0 ssd -wi-ao---- 200.00g
 vm-7000-disk-0 ssd -wi-a----- 16.00g
 vm-7000-disk-1 ssd -wi-a----- 500.00g
 vm-8000-cloudinit ssd -wi-a----- 4.00m
 vm-8000-disk-0 ssd -wi-a----- 32.00g
 vm-8000-disk-1 ssd -wi-a----- 200.00g
 vm-8001-disk-0 ssd -wi-a----- 32.00g
root@host3:~$

The above commands shows that on one of my Proxmox servers, I have a PV, /dev/nvme0n1p3, a VG, ssd, and a few LVs. Each of the client OS has its own LVs.

Using LV as the host OS storage device makes a lot of sense, which is the whole point of LVM.

LVM-Thin

By default, Proxmox does not use LVM. Instead, it uses LVM-thin. The difference between LVM-thin and LVM is that:

LVM allocates your storage blocks when you create LVs. With LVM, your LV’s storage is guaranteed for your use.
LVM-thin allocates your storage blocks when you write data to LVs. LVM-thin provides a higher disk utilization, but it can’t guarantee you can use all your LV storage.