Home Lab - "starhaven"

Overview

Everything below - VM provisioning, OS hardening, monitoring deployment, workstation configuration - runs through Ansible playbooks, not manual clicks in the Proxmox UI.

The engineering-notes sections on this page (hardening, scores, access design, backups, services, workstations, open items) were either read directly from the live Ansible repos and playbooks, or gathered from the fleet itself via read-only ansibleuser facts at the time of writing. Where something couldn't be verified live, that's called out rather than assumed.

Hardware

• Server: Dell Precision 3630 Tower
• Hypervisor: Proxmox VE
• CPU: Intel Core i5-8500 @ 3.00GHz (6 cores, 6 threads)
• RAM: 64GB DDR4 2666MHz (4×16GB)
• GPU: NVIDIA GTX 1050 Ti (passthrough to Jellyfin)

Storage

• NVMe: Crucial P1 1TB - Proxmox OS, LVM-thin
• ZFS Pool: WD 8TB @ 7200rpm (vm-8tb-hdd0) - VMs/LXC, 7.27T total, 33% allocated, HEALTH: ONLINE
• Backup HDD: WD Blue 1TB @ 7200rpm - offsite-synced via rclone

NVMe (LVM-thin) holds the Proxmox OS itself - currently 49G of 94G used (55%) on the live host, separate from the ZFS pool so a guest's storage churn can't threaten the hypervisor's own root filesystem.

→ Full ZFS breakdown: pool status, dataset list, tuning (atime/compression/ARC), scrub/TRIM schedule

Memory & CPU tuning

Unlike the rest of this page, this section is applied directly on the host over SSH, not yet codified into ansible-proxmox - flagged honestly rather than implied as automated (see the open-items list at the bottom).

VM / LXC Inventory

Current lineup as of the last update - vmid, name, and purpose, not the original provisioning plan. Two guests from earlier writeups (dev.server.wow, gitea-dev.server.wow) have since been retired.

Non-gaming services - what and why

Ansible Automation Stack

Three repos cover the full lifecycle: VM/LXC provisioning on the Proxmox host, day-to-day fleet operations, and workstation configuration on top of a fresh install - each picks up exactly where the previous one leaves off.

A single dedicated ansibleuser automation account - SSH-key-only, NOPASSWD sudo - is used across essentially every managed host in both repos, rather than personal accounts. Every credential it needs (the Proxmox API token, the sudo/become password, per-VM VNC and login passwords) is Ansible Vault-encrypted in group_vars/host_vars, never plaintext - confirmed directly in ansible-proxmox/group_vars/all/main.yml (vault_become_password, pm_api_token_secret) and ansible-stationctl/group_vars/all/main.yml (vm_enduser_password_hash, vnc_password).

Not a curated highlight reel - this is the real toolchain, in the order it touches a host: bare metal → provisioned → hardened → monitored → recorded.

Hardening: the current state, and what was deliberately left out

Every managed host - both workstations and the whole ansible-proxmox fleet - runs from the same shared security-harden.yml: CIS Level 1 controls plus a curated set of Lynis-suggested extras, kept byte-for-byte identical between ansible-proxmox and ansible-stationctl by hand (no symlink; copied verbatim in the same session whenever either changes). What differs per host is which version of that file was actually last run there - see the scores below.

The layered stack: SSH key-only + no root login + approved cipher/MAC/KEX sets, PAM password-complexity + brute-force lockout (pam_faillock), UFW default-deny-incoming with only SSH and declared per-host ports open, fail2ban on the SSH jail, AppArmor in enforce mode, and auditd with a CIS-mapped ruleset.

What was deliberately declined, and why

A real bug, found and fixed

Security scores - what they mean, and a real before/after

For anyone not steeped in this: Lynis is an open-source security auditing tool. It inspects a running Linux system - SSH config, firewall state, kernel settings, password policy, installed packages, and dozens of other areas - and reports a single hardening index from 0-100. It is not a vulnerability scanner and the number isn't "percent secure"; it's roughly "how much of Lynis's own hardening checklist does this specific host actually pass," weighted by how relevant each check is to that machine. A stock, freshly-installed Linux box typically scores in the 55-65 range without any hardening applied at all. Anything in the high 70s to 80s reflects real, deliberate configuration work, not defaults.

The "before" column below shows what ad hoc, manually-applied security looks like on this fleet - UFW rules and SSH basics had been set up by hand over time, but never run through the full, current CIS-benchmark playbook. The "after" column is the same seven hosts immediately following a real ansible-playbook security-harden.yml run, same session, both scores from actual Lynis audits rather than an estimate.

Consistent +7 to +8 point jump across the whole ansible-proxmox fleet from one playbook run, with zero service downtime - ufw status and each service's listening port were checked on every host immediately after, and every UFW rule and every monitored/media/music service was still up. remote-workstation's own history (75 → 76 → 81 → 82 across three separate hardening rounds this project) is the same pattern at smaller scale, and local-workstation converges from the same playbook so it's not a coincidence both workstations land on exactly 82. The remaining gap between the workstations (82) and the rest of the fleet (79-81) wasn't root-caused as part of this pass - auditd is inactive on both sides but for different, unverified-in-depth reasons (the LXCs likely hit the standard unprivileged-container audit-namespace limitation, the workstations have it structurally disabled via audit=0) - worth a closer diff the next time this list gets revisited.

Network exposure and access design

Nothing on the LAN is port-forwarded to the internet. The two workstations connect over Tailscale, with stable Tailnet IPs used in inventory specifically because LAN DHCP leases aren't (a real reboot once moved a host's LAN IP and broke nothing, because the automation never depended on it).

Backups - what's actually scheduled

Proxmox-native vzdump backup jobs are managed declaratively via pve-backup-jobs.yml (idempotent, API-first, no SSH to the node required) - this is the actual mechanism, not just "backups happen":

The "offsite-synced via rclone" backup HDD mentioned above is real (WD Blue 1TB) but its schedule/retention isn't codified in either Ansible repo - it runs outside of what's checked into source control, so this page can't state its actual cadence with the same confidence as the vzdump jobs above.

The two managed workstations, specifically

local-workstation (bare-metal ThinkPad E490, "PeakPulse") and remote-workstation (VM 115, "Nebula") converge to the same environment via one playbook set, but they're desktops first - the concerns here are different from the server-side fleet:

• Kernel performance tuning trades some compliance surface (the audit=0 tradeoff above) for desktop responsiveness - nowatchdog, noirqdebug, threadirqs, preempt=full, tuned vm.swappiness.
• VNC restart safety gate - before restarting the VNC server, the playbook checks for an established client connection and pauses for explicit confirmation rather than silently dropping an active session.
• Dotfile sync via a tray applet, not a login-time script - keeps both machines' shell config converged without a surprising delay at every login.
• A separate personal SSH trust (bean↔kenny, distinct from ansibleuser) for editing each machine as a human rather than through the automation account, with ControlMaster/ControlPersist multiplexing for fast repeat connections.

Monitoring & Observability

• OptiPing: Live service uptime dashboard - OptiPing (custom uptime monitor), running on lxc-optiping
• Grafana: Real-time infrastructure dashboards - exported dashboard JSON
• Prometheus: Metrics collection and alerting
• Proxmox Exporter: Starhaven Proxmox Node Exporter - custom Python exporter for Proxmox VE metrics, running on lxc-proxexport

Prometheus/Grafana are the standard pair; the node exporter and OptiPing are custom Python tools rather than off-the-shelf alternatives. The specific reasoning for writing custom over adopting an existing exporter/uptime-checker wasn't found documented in either tool's repo at the time of writing - worth adding a line to one of those READMEs if the reasoning is worth preserving.

Smart Home - Raspberry Pi 4

• Device: Raspberry Pi 4 (BCM2711, 4GB RAM), 99.9% uptime
• Stack: Home Assistant, Node-RED, Zigbee2MQTT, Docker, Apache, MQTT
• Integrations: HACS (Local WebRTC, Local Tuya - cloud-free), 20+ Zigbee devices, custom Node-RED automations

A lighter, separate stack, and not Ansible-managed the same way as the rest of the lab (it isn't present in either repo's inventory). Local-only integrations specifically (HACS Local WebRTC, Local Tuya) so camera/smart-plug control doesn't depend on a vendor's cloud being up.

Planned Upgrades

• CPU: Intel Core i7-8700 (6C/12T)
• RAM: Expand to 128GB DDR4 - not treated as urgent: ZFS ARC and VM ballooning are both elastic (see the memory-tuning section above), so adding a second, much larger storage pool doesn't by itself force a RAM purchase, only a possible ARC hit-ratio tradeoff worth revisiting once the new pool is live.
• Storage: A new mirrored ZFS pool alongside the existing single-disk vm-8tb-hdd0 (which has zero redundancy today) - see the ZFS roadmap for the drive choice and reasoning.
• Services: Nextcloud VM, Wiki.js, and more

Still open, honestly

• Backup coverage gaps - several service VMs and all four monitoring LXCs have no scheduled vzdump job (see backups section).
• The RPi bastion architecture is designed but not deployed - setup-rpi-bastion.yml is written; [rpi-bastions] in inventory is empty.
• The vmid: 115 collision - confirmed lxc-wazuh's entry in lxcs.yml is the stale one (115 is really remote-workstation) - still needs the actual fix in the repo.
• MAC addresses aren't yet recorded in ansible-stationctl's host_vars, which blocks the ARP cross-check step of its IP-drift detection - currently skipped with a note on every run rather than verified.
• Two pre-existing, unrelated issues surfaced while checking services post-hardening (neither caused by the hardening pass - security-harden.yml never touches Docker or VNC): the navidrome Docker container reports unhealthy in docker ps despite serving traffic fine, and lxc-grafana/lxc-prometheus/lxc-proxexport turned out to be running Ubuntu apt sources rather than actual Debian, which is worth reconciling with their hostnames/templates at some point.
• The auditd gap between the workstations and the rest of the fleet (see the scores section) - both sides have it inactive, for two different and not-yet-confirmed reasons. Worth diffing instead of assuming.
• The memory/CPU tuning section above isn't in Ansible yet - it's all applied by hand over SSH, which is exactly how the stale-swappiness-file bug it describes was able to sit unnoticed for months. Codifying it (sysctl drop-in, zram config, ksmtuned config, balloon floors) into ansible-proxmox would make it drift-checkable the same way the rest of the fleet is.