New backup server just dropped!
This server is awfully slow, and the mdraid setup is awfully slow, and I doubt that this will be a good experience, but we now have a backup server again?
- Tried Disko and nixos-anywhere
- Tried using mdraid
- Found that md is ancient and bad
- Found that disko is 100% extra steps, and a lot more complicated and noisy than just formatting your disks yourself
- Found that systemd-boot doesn't support mdraid
- Found that we probably don't need to mirror the boot partition :)
- Found that old hardware is slow
- Found that old hardware can have poor support for iPXE with UEFI, and might do weird BIOS stuff on you when you least expect it
- Reaffirmed that zfs is love
Current disk layout:
- mdraid for boot/root disk
- 4TB WD Red with 500MiB ESP with systemd-boot, Remaining mdraid - Old?
- 4TB WD Red with 500MiB Unused partition, Remaining mdraid - Old?
- zfs pool "tank" for the actual backup data
- 8TB Toshiba MG08 - New
- 8TB Exos 7E10 - New
TODO:
- Document the death of Toriel on the wiki
- Document Bakke on the wiki
- ... describing the poco loco disk layout
- Start backing stuff up
- Restic? Borg? Rsync?
- Make backup retention policy and zfs snapshot system
- Document backup procedures
Reviewed-on: Drift/pvv-nixos-config#87
Co-authored-by: Felix Albrigtsen <felix@albrigtsen.it>
Co-committed-by: Felix Albrigtsen <felix@albrigtsen.it>
This PR is made while moving Ildkule from PVE on joshua, to Openstack on stack.it.ntnu.no.
- The main monitoring dashboard is moved from https://ildkule.pvv.ntnu.no to https://grafana.pvv.ntnu.no.
- A new service is added: uptime-kuma on https://uptime.pvv.ntnu.no.
- The (hardware) configuration for ildkule is updated to fit the new virtualization environment, boot loader, network interfaces, etc.
- Metrics exporters on other hosts should be updated to allow connections from the new host
As this is the first proper server running on openstack, and therefore outside our main IP range, we might discover challenges in our network structure. For example, the database servers usually only allow connections from this range, so Ildkule can no longer access it. This should be explored, documented and/or fixed as we move more services.
Reviewed-on: Drift/pvv-nixos-config#36
Co-authored-by: Felix Albrigtsen <felix@albrigtsen.it>
Co-committed-by: Felix Albrigtsen <felix@albrigtsen.it>
