PVE mit ZFS und native encryption
Proxmox Virtual Environment hat meinen alten XEN Server abgelöst. Die Festplattenvolumes für virtuelle Server und Daten werden wieder verschlüsselt.
Installation von PVE mit ZFS
Die Installation erfolgte mittels ISO Datei auf einem USB-Stick. Als Filesystem wählte ich ZFS mit RAID 1 auf zwei SSD mit Standard Parametern.
Lokale Anpassungen
Repository PVE-No-Subscription und Warnung
Das Update Repository No-Subscription hinzufügen:
. /etc/os-release && echo "deb http://download.proxmox.com/debian/pve $VERSION_CODENAME pve-no-subscription" >> /etc/apt/sources.list
Deaktivierung der nervigen Warnung über eine fehlende Subscription: Über /etc/rc.local starte ich ein kleines script remove-proxmox-subscription-notice.sh - siehe Files Sektion.
Encrypted ZFS Volume
Ich habe ein Verzeichnis '/etc/cryptkey' angelegt, dass alle zur Verschlüsselung notwendigen Dateien aufnimmt:
- keymanager_zfs.sh - Key Management Tool
- keyscript.sh - Startscript (Bootprozess)
- key_zfs.conf - Konfigurationsdetails
- root - ZFS KEY (SSL verschlüsselt)
root# ls -l /etc/cryptkey -rwxr-x--- 1 root root 9542 Dec 19 22:40 keymanager_zfs.sh -rwxr-x--- 1 root root 680 Dec 19 20:49 keyscript_zfs.sh -r-------- 1 root root 859 Dec 19 21:46 key_zfs.conf -r-------- 1 root root 130 Dec 19 20:49 root
Das Key Management Tool keymanagement.sh habe ich in einem Forum gefunden. Damit wird ein zufälliger 64-Byte KEY mit SSL verschlüsselt und mit einem (SSL-) Passwort versehen abgespeichert.
Dieser ZFS KEY (/etc/cryptkey/root) kann nur mit dem SSL Passwort ausgelesen werden. Das Passwort wird aber nicht auf der Festplatte gespeichert, die das ZFS Volume beherbergt. Trotzdem soll der Bootvorgang das ZFS Volume mit diesem KEY ohne (manuelle!) Eingabe des Passworts entschlüsseln können - aber nicht müssen.
Die Lösung: Der KEY wird nicht nur im File root mittels SSL verschlüsselt gespeichert, sondern auch auf einem USB-Stick. Dabei kann ein anderes Passwort verwendet werden - muss aber nicht. Ich verwende für jeden USB-Stick ein individuelles Passwort, welches ich nirgends extra speichern möchte. Als Passwort verwende ich daher die ID des USB-Sticks: /dev/disk/by-id/<usb-stick-id> Dieses steht auf dem Server automatisch zur Verfügung, ohne das es irgendwo gespeichert ist. Der Server kann also das verschlüsselte ZFS Volume mit dem auf dem Stick gespeicherten, SSL verschlüsseltem KEY entschlüsseln, indem er als Passwort die USB-Stick ID verwendet.
Steckt der USB-Stick nicht im Server, wird der KEY aus dem File root ausgelesen, was allerdings eine manuelle Eingabe des Passwortes erfordert. Das wird im Bootvorgang also fehlschlagen, der Server bootet, ohne das verschlüsselte ZFS Volume zu mounten.
Ein Aufruf des Startscriptes keyscript_zfs.sh liest in diesem Falle den ZFS KEY aus dem File /etc/cryptkey/root aus, fragt dabei das Passwort manuell ab, und bindet das Volume in einem solchen Fall nachträglich ein.
ZFS KEY anlegen
ZFS KEY erzeugen, mit SSL verschlüsseln und speichern. Default ist 64BYTE mit Filename = <userid> (also root).
root# cd /etc/cryptkey ./keymanager_zfs.sh makekey
encrypted ZFS Volume anlegen
Verschlüsseltes ZFS Volume rpool/encrypted_data mit KEY (Default ist Filename = <userid>, also root) anlegen.
./keymanager_zfs.sh zfscreate -v rpool/encrypted_data
Unlock encrypted ZFS Pool
Verschlüsseltes ZFS Volume rpool/encrypted_data mit KEY (Default ist Filename = <userid>, also root) einbinden.
./keyscript_zfs.sh zfs get keystatus NAME PROPERTY VALUE SOURCE rpool keystatus - - rpool/ROOT keystatus - - rpool/ROOT/pve-1 keystatus - - rpool/data keystatus - - rpool/encrypted_data keystatus available -
Install ZFS-KEY on USB-Stick
Konfiguration lesen, ZFS Key mit anderem Passwort erneut verschlüsselt abspeichern und auf USB-Stick schreiben.
. ./key_zfs.conf
./keymanager_zfs.sh useradd -q ${USB-STICK} -m /run/.cryptkey/${USB-STICK}
dd if=/run/.cryptkey/${USB-STICK} of=/dev/disk/by-id/${USB-STICK} bs=${USB_BSIZE} count=${USB_COUNT} seek=${USB_SKIP}
ZFS Volume in PVE Storage einbinden
pvesm add zfspool encrypted-zfs -pool rpool/encrypted_data
Files
/root
remove-proxmox-subscription-notice.sh
Das Script remove-proxmox-subscription-notice.sh sdeaktiviert die Subscription Warnung im Webinterface. Ich starte es über /etc/rc.local.
#!/bin/sh
sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service
/etc/rc.local
Deaktiviere Subscription Warnung, aktiviere das encrypted ZFS Volume.
#!/bin/sh -e
# local init script
# load ZFS encryption key on boot from USB Stick
if [ -r /etc/cryptkey/key_zfs.conf ]; then
/etc/cryptkey/keyscript_zfs.sh
fi
/root/remove-proxmox-subscription-notice.sh
# evtl. Sleepmodus für HDD Festplatten aktivieren
#[ -e /dev/sda ] && hdparm -Y -S 242 /dev/sda || true
#[ -e /dev/sdb ] && hdparm -Y -S 242 /dev/sdb || true
mount -a
exit 0
/etc/cryptkey
key_zfs.conf
Definition wo auf dem USB Stick der ZFS KEY gespeichert wird, und wie das ZFS Volume heisst:
- ENC_ZFSVOL - verschlüsselte(s) ZFS Volume(s)
- USB_STICK - USB Stick Name (ID)
# config file for ZFS encryption with multiple keyfiles # # concept: the keyfile for ZFS encryption is stored # on an USB Stick, encrypted with a fixed password (used for booting) # on filesystem, encrypted with a different password (not in any script) # # this gives the possibility to load the ZFS encryption key # - at boot without password prompt ==> if the USB_STICK is connected # - manually with a different(!) password ==> if the USB_STICK is defect # name of encrypted ZFS volume ENC_ZFSVOL="rpool/encrypted_data hddpool/encrypted_data" # name of USB Stick on /dev/disk/by-id USB_STICK=usb-Memorybird_P_100074AE3400109-0:0 # where the encrypted password is stored on USB Stick (raw data before 1st partition) USB_SKIP=2560 USB_COUNT=130 USB_BSIZE=1 USB_PASSWORD=$USB_STICK
keyscript.sh
Startscript für den Bootvorgang, wird aus /etc/rc.local gestartet.
#!/bin/bash
CONFIG=/etc/cryptkey/key_zfs.conf
SCRIPT=/etc/cryptkey/keymanager_zfs.sh
ENC_KEY=/run/.cryptkey/usb_key_zfs.enc
[ -r $CONFIG ] || exit 0
. $CONFIG
for ZFSVOL in $ENC_ZFSVOL; do
if [ -e /dev/disk/by-id/$USB_STICK ]; then
# extract encrypted key from usb stick
mkdir -p $(dirname $ENC_KEY)
chmod 400 $(dirname $ENC_KEY)
dd if=/dev/disk/by-id/$USB_STICK of=$ENC_KEY bs=$USB_BSIZE count=$USB_COUNT skip=$USB_SKIP >/dev/null 2>&1
chmod 400 $ENC_KEY
$SCRIPT zfsload -s $(dirname $ENC_KEY) -n $(basename $ENC_KEY) -p "$USB_STICK" -v $ZFSVOL || \
$SCRIPT zfsload -s $(dirname $CONFIG) -v $ZFSVOL
else
$SCRIPT zfsload -s $(dirname $CONFIG) -v $ZFSVOL
fi
done
keymanager_zfs.sh
Das Key Management Tool habe ich in einem Forum gefunden: keymanager.sh
#!/bin/bash
# zfs multiple passwords
# found at: https://forums.gentoo.org/viewtopic-t-1122263-start-0.html
#
# keymanager.sh
case $1 in
"makekey")
#default values
keysize=64
keyname=$USER
[ ! -z $SUDOUSER ] && keyname=$SUDOUSER
store=.
cnt=1
ls=1
for a in "$@"
do
let cnt+=1
case $a in
-k | --keysize)
keysize=${!cnt}
ls=1
;;
-s | --keystore)
store=${!cnt}
ls=1
;;
-n | --keyname)
keyname=${!cnt}
ls=1
;;
-p | --password)
pass=${!cnt}
ls=1
;;
-*)
echo $1: unknown argument $a
exit 255
;;
*)
# if we get here twice in a row, something is wrong
[ $ls = 0 ] && echo $1: unknown argument $a && exit 254
ls=0
;;
esac
done
mkdir -p $store
key=`openssl rand -base64 $keysize |tr -d '\n='`
[ $? != 0 ] && echo "Keygeneration failed" && exit 3
key="KEY${key:0:$keysize}"
[ -e ${store}/${keyname} ] && echo Key exists. I will not overwrite ${store}/${keyname} && exit 252
if [ -z $pass ] ; then
openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 <<< ${key} > ${store}/${keyname}
else
openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -pass pass:${pass} <<< ${key} > ${store}/${keyname}
fi
[ $? != 0 ] && echo "Encryption failed" && rm ${store}/${keyname} && exit 4
chmod 600 ${store}/${keyname}
;;
"zfscreate")
#default values
keyname=$USER
[ ! -z $SUDOUSER ] && keyname=$SUDOUSER
store=.
opt=""
cnt=1
ls=1
for a in "$@"
do
let cnt+=1
case $a in
-s | --keystore)
store=${!cnt}
ls=1
;;
-n | --keyname)
keyname=${!cnt}
ls=1
;;
-p | --password)
pass=${!cnt}
ls=1
;;
-v | --volname)
vol=${!cnt}
ls=1
;;
-o)
opt="$opt -o ${!cnt}"
ls=1
;;
-*)
echo $1: unknown argument $a
exit 255
;;
*)
# if we get here twice in a row, something is wrong
[ $ls = 0 ] && echo $1: unknown argument $a && exit 254
ls=0
;;
esac
done
[ -z $vol ] && echo $1 needs volumeName. Use -v volumename
[ ! -r ${store}/${keyname} ] && echo Key not accessible ${store}/${keyname} && exit 123
if [ -z $pass ] ; then
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d < ${store}/${keyname}`
else
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d -pass pass:${pass} < ${store}/${keyname}`
fi
[ "KEY" != ${key:0:3} ] && echo Key decrypt failed && exit 10
defopt=""
[[ "$opt" != *"encryption="* ]] && defopt="$defopt -o encryption=on"
[[ "$opt" != *"keyformat="* ]] && defopt="$defopt -o keyformat=passphrase"
zfs create $defopt $opt $vol <<< $key
;;
"zfsload")
#default values
keyname=$USER
[ ! -z $SUDOUSER ] && keyname=$SUDOUSER
store=.
opt=""
cnt=1
ls=1
for a in "$@"
do
let cnt+=1
case $a in
-s | --keystore)
store=${!cnt}
ls=1
;;
-n | --keyname)
keyname=${!cnt}
ls=1
;;
-p | --password)
pass=${!cnt}
ls=1
;;
-v | --volname)
vol=${!cnt}
ls=1
;;
-*)
echo $1: unknown argument $a
exit 255
;;
*)
# if we get here twice in a row, something is wrong
[ $ls = 0 ] && echo $1: unknown argument $a && exit 254
ls=0
;;
esac
done
[ -z $vol ] && echo $1 needs volumeName. Use -v volumename
zfs get keystatus |grep $vol |grep " available" && echo Key already loaded, nothing to do && exit 0
[ ! -r ${store}/${keyname} ] && echo Key not accessible ${store}/${keyname} && exit 123
if [ -z $pass ] ; then
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d < ${store}/${keyname}`
else
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d -pass pass:${pass} < ${store}/${keyname}`
fi
[ "KEY" != "${key:0:3}" ] && echo Key decrypt failed && exit 10
zfs load-key $vol <<< $key
;;
"useradd")
#default values
keyname=$USER
[ ! -z $SUDOUSER ] && keyname=$SUDOUSER
newkeyname=$USER
[ ! -z $SUDOUSER ] && newkeyname=$SUDOUSER
store=.
opt=""
cnt=1
ls=1
for a in "$@"
do
let cnt+=1
case $a in
-s | --keystore)
store=${!cnt}
ls=1
;;
-n | --keyname)
keyname=${!cnt}
ls=1
;;
-p | --password)
pass=${!cnt}
ls=1
;;
-m | --newkeyname)
newkeyname=${!cnt}
ls=1
;;
-q | --newpass)
newpass=${!cnt}
ls=1
;;
-*)
echo $1: unknown argument $a
exit 255
;;
*)
# if we get here twice in a row, something is wrong
[ $ls = 0 ] && echo $1: unknown argument $a && exit 254
ls=0
;;
esac
done
[ ! -r ${store}/${keyname} ] && echo Key not accessible ${store}/${keyname} && exit 123
if [ -z $pass ] ; then
echo Decrypting key
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d < ${store}/${keyname}`
else
key=`openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -d -pass pass:${pass} < ${store}/${keyname}`
fi
[ "KEY" != ${key:0:3} ] && echo Key decrypt failed && exit 10
[ -e ${store}/${newkeyname} ] && echo New key exists. I will not overwrite ${store}/${newkeyname} && exit 252
if [ -z $newpass ] ; then
echo Encrypting key
openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 <<< ${key} > ${store}/${newkeyname}
else
openssl enc -aes-256-cbc -a -pbkdf2 -iter 1000000 -pass pass:${newpass} <<< ${key} > ${store}/${newkeyname}
fi
chmod 600 ${store}/${newkeyname}
;;
*)
iam=`basename $0`
echo Usage:
echo "$iam makekey [-k keySize] [-s keystoreFolder] [-n keyname] [-p password]"
echo " Generates at random string (base_64 characters) of length keySize "
echo " encrypts it by password, and stores the encrypted string in"
echo " folder keystoreFolder."
echo " If password is not given, it will be requested from stdin."
echo " If no keyname is given, the current username will be used as keyname."
echo " If no keystoreFolder is given, the current folder will be used."
echo " If no keysize is given, default 64 will be used"
echo
echo "$iam zfscreate [-s keystoreFolder] [-n keyname] [-p password] -v volumename [-o zfsoption [-o ....]]"
echo " Creates a zfs volume named volumename encrypted with key"
echo " previously generated by makekey."
echo " Default zfsoption are -o encryption=on -o keyformat=passphrase"
echo " Additional zfsoptions may be specified."
echo " Password is used to decrypt key. If password is not given, it "
echo " will be requested from stdin."
echo " If no keyname is given, the current username will be used as keyname."
echo " If no keystoreFolder is given, the current folder will be used."
echo
echo "$iam zfsload [-s keystoreFolder] [-n keyname] [-p password] -v volumename"
echo " Decrypts key previously generated by makekey and executes zfs load-key"
echo " Password is used to decrypt key. If password is not given, it "
echo " will be requested from stdin."
echo " If no keyname is given, the current username will be used as keyname."
echo " If no keystoreFolder is given, the current folder will be used."
echo
echo "$iam useradd [-s keystoreFolder] [-n keyname] [-p password] [-m newkeyname] [-q newpass] "
echo " Makes a key previously generated by makekey available to anotheruser."
echo " That is, decrypts the key saved under keyname using password, then"
echo " saves it under newkeyname encryoted by using newpass"
echo " Password is used to decrypt key. If password is not given, it "
echo " will be requested from stdin."
echo " Newpassw is used to encrypt key. If newpass is not given, it "
echo " will be requested from stdin."
echo " If no keyname is given, the current username will be used as keyname."
echo " If no newkeyname is given, the current username will be used as keyname."
echo " If no keystoreFolder is given, the current folder will be used."
;;
esac