nix-dotfiles-v2/developmentlog.md

# Development log

## legolas: NVMe (Corsair MP600 ELITE / Phison E21) suspend/resume crashes

The drive hung on resume from S3 (deep) suspend — boot logs ended at `PM: suspend entry (deep)` with no resume, followed by a hard reboot. The Phison E21 can't recover from the D3 transition that S3 forces.

Investigation also found two existing workarounds were doing nothing:

- The udev rule `ATTR{d3cold_allowed}="0"` was a no-op — that sysfs attribute doesn't exist on this Phison endpoint or its parent root port (`0000:00:1d.0`).
- The `nvme-resume-fix` service was broken: `ExecStart` used `>` redirection without a shell, so `echo` just printed the string and never wrote to the `rescan` sysfs file. It never actually rescanned.

Changes (`hosts/legolas/hardware-configuration.nix`):

- Switched `mem_sleep_default` from `deep` (S3) to `s2idle` (modern standby). s2idle avoids the deep D3 path that hangs the drive.
- Removed the no-op `d3cold_allowed` rule and the broken rescan service.
- Added a correct runtime-PM-off udev rule on both the NVMe endpoint (`0000:6e:00.0`) and its parent root port (`0000:00:1d.0`), keeping the PCIe wakeup disable.

Note: runtime PM was already off on the endpoint (`power/control=on`); that alone never fixed the crash because system suspend uses a separate code path. Verify after rebuild that the cmdline has `mem_sleep_default=s2idle` (the running generation showed a stale duplicate `deep`).

## Headless home-manager configs (galadriel, elros, elrond)

Added minimal home-manager configs for `gunalx` and `root` on headless hosts. These skip all GUI/desktop modules (niri, stylix, foot, noctalia, etc.) and keep only CLI tools: bash (with sops AI keys), git, nixvim (full config reused), opencode, atuin, sshconfig, sops.

Files:
- `home/gunalx-headless.nix` — gunalx's minimal config, same module set as `gunalx.nix` minus all GUI imports.
- `home/root.nix` — identical CLI stack, explicitly sets `home.username`/`home.homeDirectory` to root.
- `home/sops.nix` — parameterized age key paths to use `config.home.homeDirectory` so both users work without duplication.
- `home/sshconfig.nix` — removed `ssh-askpass-fullscreen` (unwired GUI dependency, was dead weight).
- `flake.nix` — added `headlessHmModules` let binding (shared by all three hosts) wiring `home-manager.users.gunalx` + `home-manager.users.root`.

Manual step: root needs the sops age key at `/root/.config/sops/age/key.txt` (or SSH key at `/root/.ssh/nixos`) to decrypt AI API secrets. `age.generateKey = true` will create a new key, but it won't match the encrypted secrets — copy gunalx's age key to root's path, or re-encrypt the secrets with root's age public key.

## galadriel: kernel 7.0 EOL

Linux 7.0 was removed from nixpkgs (EOL). `pkgs.zfs` (stable) is now 2.4.2, same version previously pulled from unstable. Switched `boot.kernelPackages` to the default (6.18) and `boot.zfs.package` to `pkgs.zfs`. No pool compatibility concern since ZFS version is unchanged.

## legolas: lid-close still killing the NVMe — stale `deep` + logind suspend

Root cause of the recurring "SSD dies when lid closed": the *booted* generation carried a duplicate `mem_sleep_default=deep` after `mem_sleep_default=s2idle` (last wins), so `/sys/power/mem_sleep` was `s2idle [deep]` — every suspend went to S3. The source config only had `s2idle`; the stale param lived in the old generation. ACPI advertises S3 (`supports S0 S3 S4 S5`) but S3 is broken for the Phison E21 on this Skylake-era Intel platform, so only s0/s2idle is reliable. Separately, `HandleLidSwitch` was unset → defaulted to `suspend`, which was the actual lid-close trigger.

Changes (`hosts/legolas/hardware-configuration.nix`):
- `services.logind.lidSwitch`/`lidSwitchExternalPower`/`lidSwitchDocked` = `ignore`. Lid close no longer suspends; niri's `switch-events` still locks via noctalia, so the laptop stays on (RAM powered) but can't enter the S3 path.
- `systemd.services.force-s2idle` writes `s2idle` to `/sys/power/mem_sleep` at boot — guarantees the active mode is s2idle even if a future cmdline carries a stray `deep`.
- Suspend (s2idle) remains available via `systemctl suspend`; hibernation stays disabled.

Also fixed `home/niri.nix` lid-close command from the undocumented `lockScreen lock` to the v5-canonical `session lock` (matching the `Mod+M` binding).

Verify after rebuild+reboot: `/proc/cmdline` has no `deep`, and `/sys/power/mem_sleep` shows `[s2idle] deep`.

## Bumped locally packaged packages

Version + hash bumps for the upstream-tracking packages under `packages/`. All hashes recomputed from the actual fetched outputs (src FOD builds for the `postFetch` sources, `nurl` for plain `fetchFromGitHub`, FOD builds for `npmDepsHash`/`vendorHash`).

- `llama-cpp-nightly` + `ui.nix`: `b9665` -> `b10034` (src + npmDepsHash).
- `llama-swap` + `ui.nix`: `v226` -> `v240` (src + vendorHash + ui npmDepsHash). UI version is inherited.
- `s2cpp`: pin `e48ce8e` (2026-04-12) -> `2c33261` (2026-05-27).
- `intel-sycl`: `nightly-2026-01-01` -> `nightly-2026-07-13` (src hash only).

Not bumped: `qwen-asr` (still at upstream latest `1.0.3`); the model packages (`s2-model`, `whisper-models`, `z-image-models`) and `android-cli` use fixed URLs with no version tag to bump.

Note: `intel-sycl` is packaged but not currently wired into the flake (`llama-cpp-nightly-sycl` leaves `syclStdenv = null` and is marked broken). The vc-intrinsics pin in `intel-sycl/pins` was left as-is. Builds must use `--builders ""` locally — the configured remote builder `aragon` is unreachable and its SSH retries stall fetches.

## restic backup architecture

Hub-and-spoke via `modules/restic-server.nix` + `modules/restic-client.nix`. `services.restic.server` on galadriel (rest-server, tailnet-only on `${hostname}:8008`, no nginx — Tailscale already encrypts the transport), one shared restic repo at `/var/lib/restic/main`. Every host incl. galadriel runs `services.restic.backups.main` against `rest:http://restic:<transportpw>@galadriel:8008/main/`. Single transport credential (one htpasswd user) + single repo encryption password. Restic tags snapshots per-host, so one repo holds all hosts and retention is per-host by default (`forget` defaults to `--group-by host,paths`).

- `appendOnly = true` (`lib.mkDefault`): clients write but can't forget/prune over HTTP — the one cheap control stopping a compromised client from wiping the shared history.
- Prune: native prune-only job on galadriel (`services.restic.backups.prune`, `paths = [ ]` — the module's documented escape hatch) running `forget --prune` + `check` against the local path. Must be local; append-only blocks it over HTTP for everyone.
- Offsite: `restic copy` from the local repo to `restic/offsite_repository`. The restic module has NO native `copy` (only backup/prune/check — checked against master), so this is the one hand-rolled systemd service. Picked over per-host dual-destination backups to keep the offsite upload centralized and the offsite secret off the clients.

Secrets (`secrets/secrets.yaml`): `restic/server_htpasswd` (bcrypt hash, server), `restic/repository` (repo URL incl. transport pw, clients), `restic/repo_password` (encryption key, clients + server prune/offsite), `restic/offsite_repository` (offsite URL, galadriel). On galadriel both modules declare `restic/repo_password`; the server's `owner = "restic"` wins the merge, and root (the client job user) reads it regardless.

Manual: add secrets; make every using-host a sops recipient; `restic init` the offsite repo once with `repo_password`; set `services.restic.backups.main.paths` per host. Caveat: `listenAddress = "${hostname}:8008"` relies on Tailscale MagicDNS resolving at socket-activation time — bind the tailnet IP directly if it ever flakes at boot.