Consider switching NFSv3 for NFSv4 #284

New Issue

oysteikt · 2026-01-15T07:02:05+01:00

oysteikt commented

2026-01-15 07:02:05 +01:00

NFSv4 is not really an upgrade in the traditional sense, it's more like a switch from one system to another. And for most people this does not necessarily make sense - it does not bring any performance improvements apart from edge cases. But I suspect we might be one of those edge cases.

One of the features added in NFSv4 is compound requests - bundling several requests into a single action (e.g. lookup, stat, open, read). For interconnected servers in the same rack, this is usually not where the bottleneck is. But due to us sending our traffic from demiurgen and wegonke through an openvpn tunnel with non-trivial roundtrip times, these compound requests might have potential to speed up interaction with files in homedirs. Lots of requests for lots of small files would be likely be improved, and might make our terminals more responsive. We should at least test it out and see if we want it.

NFSv4 is not really an upgrade in the traditional sense, it's more like a switch from one system to another. And for most people this does not necessarily make sense - it does not bring any performance improvements apart from edge cases. But I suspect we might be one of those edge cases. One of the features added in NFSv4 is compound requests - bundling several requests into a single action (e.g. lookup, stat, open, read). For interconnected servers in the same rack, this is usually not where the bottleneck is. But due to us sending our traffic from demiurgen and wegonke through an openvpn tunnel with non-trivial roundtrip times, these compound requests might have potential to speed up interaction with files in homedirs. Lots of requests for lots of small files would be likely be improved, and might make our terminals more responsive. We should at least test it out and see if we want it.

oysteikt added the salt new stuff services exploration networking labels 2026-01-15 07:02:05 +01:00

oysteikt added this to the Kanban project 2026-01-15 07:02:05 +01:00

oysteikt moved this to Low priority in Kanban on 2026-01-15 07:03:15 +01:00

felixalb commented

2026-01-15 21:29:39 +01:00

There are a few other differences, but I am not sure how they affect us yet. A few thoughts:

Permission/ownership checking is based on uids on NFSv3, and usernames on NFSv4. Should not matter for human users, but we should think through if we have any special system users like "mail" that require the correct perms and differ between systems. This must be considered in both directions (breaking systems by blocking access, or leaking files by granting too much access)
Do all our clients support NFSv4(.1?)? If something old (tom? any remaining freebsd terminals?) doesn't, we might have to upgrade them first, or run both types of NFS
Would normal usage become problematic with NFSv4 locks and leases?
- I assume most user-software like e-mail and irc clients won't create exclusive locks, and most people (hopefully) don't run postgres on the login box
Do we use NFS for anything other than users home directories?
Do we do any permission trickery and reach into peoples homedirs on non-microbel hosts?
- With root squashing, I think mod_userdir will keep working on tom
- All our webmail clients use IMAP now, not direct folder access
On an unstable connection (the VPN tunnel has been somewhat flaky lately), NFSv4 may crash harder than NFSv3.
In addition to the compound requests you mention, NFSv4.1 has other (probably even more important) performance improvements such as client-side caching
microbel is older than the Wii U and the original Raspberry Pi, with old drives in a spooky mdraid, etc. At the same time, we need new storage systems for VMs, backup, etc., is this migration a good time to also move homedirs to new hardware?
- If someone wants to design a shed where you keep bikes and yak shears, this might also include upgrading or replacing the e-mail stack (see #111)

There are a few other differences, but I am not sure how they affect us yet. A few thoughts: - Permission/ownership checking is based on uids on NFSv3, and usernames on NFSv4. Should not matter for human users, but we should think through if we have any special system users like "mail" that require the correct perms and differ between systems. This must be considered in both directions (breaking systems by blocking access, or leaking files by granting too much access) - Do all our clients support NFSv4(.1?)? If something old (tom? any remaining freebsd terminals?) doesn't, we might have to upgrade them first, or run both types of NFS - Would normal usage become problematic with NFSv4 locks and leases? - I assume most user-software like e-mail and irc clients won't create exclusive locks, and most people (hopefully) don't run postgres on the login box - Do we use NFS for anything other than users home directories? - Do we do any permission trickery and reach into peoples homedirs on non-microbel hosts? - With root squashing, I think mod_userdir will keep working on tom - All our webmail clients use IMAP now, not direct folder access - On an unstable connection (the VPN tunnel has been somewhat flaky lately), NFSv4 may crash harder than NFSv3. - In addition to the compound requests you mention, NFSv4.1 has other (probably even more important) performance improvements such as [client-side caching](https://www.rfc-editor.org/rfc/rfc5661#section-10) - microbel is older than the Wii U and the original Raspberry Pi, with old drives in a spooky mdraid, etc. At the same time, we need new storage systems for VMs, backup, etc., is this migration a good time to also move homedirs to new hardware? - If someone wants to design a shed where you keep bikes and yak shears, this might also include upgrading or replacing the e-mail stack (see https://git.pvv.ntnu.no/Drift/issues/issues/111)

oysteikt commented

2026-01-16 07:31:28 +01:00

Permission/ownership checking is based on uids on NFSv3, and usernames on NFSv4. Should not matter for human users, but we should think through if we have any special system users like "mail" that require the correct perms and differ between systems

I don't think we have any UID/GID mismatches though, the ownership checks should in theory be enforced exactly the same way as it currently is. Also, we don't really use NFS for anything else than home user dirs, and the mail is not handled through NFS - likely for this reason among others.

Do all our clients support NFSv4(.1?)

Tom is ancient at this point, we should switch it out anyway. But NFSv4 is dinosaur material. NFSv4.1 was released in 2010, long before the release of oldoldoldoldstable debian 9 stretch, which tom is currently running.

Would normal usage become problematic with NFSv4 locks and leases

No clue, but I also think there's no good way to determine that for most of our software except for trying it out. But like, the NFSv4 locks are supposed to work by the semantics of how a lock is supposed to work anyway, no? What are the likely pitfalls?

Do we use NFS for anything other than users home directories?

Tom user web hosting, gopher on isvegg, else no

Do we do any permission trickery and reach into peoples homedirs on non-microbel hosts?

Not AFAIK, the only place we'd need that would be tom or maybe isvegg, but I think both are pretty default setups that just follow the rules as given.

On an unstable connection (the VPN tunnel has been somewhat flaky lately), NFSv4 may crash harder than NFSv3.

Because of the lock leases?

In addition to the compound requests you mention, NFSv4.1 has other (probably even more important) performance improvements such as client-side caching

Maybe we should try deploying https://github.com/sbu-fsl/fsl-tc-client?

microbel is older than the Wii U and the original Raspberry Pi, with old drives in a spooky mdraid, etc. At the same time, we need new storage systems for VMs, backup, etc., is this migration a good time to also move homedirs to new hardware?

I don't think these issues need to be codependent. AFAIK, there's nothing in the way of hosting NFSv4 in parallel with NFSv3 and just trying it out. If nothing else, we could at least set up a dummy server on the other side of the VPN tunnel and test it out on demiurgen and wegonke.

> Permission/ownership checking is based on uids on NFSv3, and usernames on NFSv4. Should not matter for human users, but we should think through if we have any special system users like "mail" that require the correct perms and differ between systems I don't think we have any UID/GID mismatches though, the ownership checks should in theory be enforced exactly the same way as it currently is. Also, we don't really use NFS for anything else than home user dirs, and the mail is not handled through NFS - likely for this reason among others. > Do all our clients support NFSv4(.1?) Tom is ancient at this point, we should switch it out anyway. But NFSv4 is dinosaur material. NFSv4.1 was released in 2010, long before the release of oldoldoldoldstable debian 9 stretch, which tom is currently running. > Would normal usage become problematic with NFSv4 locks and leases No clue, but I also think there's no good way to determine that for most of our software except for trying it out. But like, the NFSv4 locks are supposed to work by the semantics of how a lock is supposed to work anyway, no? What are the likely pitfalls? > Do we use NFS for anything other than users home directories? Tom user web hosting, gopher on isvegg, else no > Do we do any permission trickery and reach into peoples homedirs on non-microbel hosts? Not AFAIK, the only place we'd need that would be tom or maybe isvegg, but I think both are pretty default setups that just follow the rules as given. > On an unstable connection (the VPN tunnel has been somewhat flaky lately), NFSv4 may crash harder than NFSv3. Because of the lock leases? > In addition to the compound requests you mention, NFSv4.1 has other (probably even more important) performance improvements such as [client-side caching](https://www.rfc-editor.org/rfc/rfc5661#section-10) Maybe we should try deploying https://github.com/sbu-fsl/fsl-tc-client? > microbel is older than the Wii U and the original Raspberry Pi, with old drives in a spooky mdraid, etc. At the same time, we need new storage systems for VMs, backup, etc., is this migration a good time to also move homedirs to new hardware? I don't think these issues need to be codependent. AFAIK, there's nothing in the way of hosting NFSv4 in parallel with NFSv3 and just trying it out. If nothing else, we could at least set up a dummy server on the other side of the VPN tunnel and test it out on demiurgen and wegonke.

felixalb commented

2026-01-16 15:36:03 +01:00

Sure, sounds good, me likes!

Note:

Do we use NFS for anything other than users home directories?

Tom user web hosting, gopher on isvegg, else no

That is still homedirs, I was thinking if there are other exports/shares than microbel:/home/pvv that use NFS, but I can't think of any

Sure, sounds good, me likes! --- Note: > > Do we use NFS for anything other than users home directories? > Tom user web hosting, gopher on isvegg, else no That is still homedirs, I was thinking if there are other exports/shares than microbel:/home/pvv that use NFS, but I can't think of any

oysteikt commented

2026-01-16 16:02:59 +01:00

No, you can check microbels /etc/exports, there's nothing else there

No, you can check microbels `/etc/exports`, there's nothing else there

felixalb commented

2026-01-16 16:22:09 +01:00

Yes, I have, but I haven't checked every other servers mounts and exports yet, as the issue doesn't seem to be limited to Microbel

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Drift/issues#284