Can't Access NFS Shares After Ubuntu Update

[BeebMaster]

I did a Ubuntu (22.04) update earlier and now RISC PC says this:

"Host can't support program (ie. wrong version)"

I can't even take a screenshot, as !Snapper saves to the NFS share!

I think this is because NFSv2 support has fallen off again:

Code: Select all

sudo cat /proc/fs/nfsd/versions
[sudo] password for ian: 
+3 +4 +4.1 +4.2

In the past I have had to apply a bodge to /etc/default/nfs-kernel-server to get it back:

Code: Select all

RPCNFSDCOUNT="--nfs-version 2 8"

It took me a long time to find that last time this happened (probably on the upgrade to 22.04). That line is still in the file, but it doesn't seem to work any more. Another suggestion was to use this:

Code: Select all

RPCNFSDCOUNT="8 -V 2"

But that doesn't work either.

Help!!!

[sweh]

Check https://bugs.launchpad.net/ubuntu/+sour ... comments/4

[BeebMaster]

Thank you, haven't cracked it yet. The only thing that wasn't already in my /etc/nfs.conf file was the vers2,3,4=y line, which I added but it hasn't helped:

Code: Select all

nfsconf --dump
[general]
 pipefs-directory = /run/rpc_pipefs

[mountd]
 manage-gids = y

[nfsd]
 tcp = y
 udp = y
 vers2 = y
 vers2,3,4 = y
 vers3 = y
 vers4 = y
 vers4.0 = y
 vers4.1 = y
 vers4.2 = y

Code: Select all

sudo cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

Last time it went wrong (looks to have been in January when I upgraded to 22.04) it was a real pain to fix, as the upgrade itself "failed" with all the NFS components going wrong and I had to fix them all (plus other things) manually according to my notes at the time:

Code: Select all

62  9/1/23
    Upgrade to Ubuntu 22.04.  Upgrade failed and NFS not working.  Fix with remove and reinstall nfs-common nfs-kernel-server rpcbind
    rm /lib/systemd/system/nfs-common.service
    sudo service nfs-common start
    sudo service nfs-kernel-server start    
    (nb. may need exports file restoring from this dir if files purged)
    Stop dmesg needing sudo:
    sudo sysctl -w kernel.dmesg_restrict=0
    Permanent fix:
    echo kernel.dmesg_restrict=0 | sudo tee -a /etc/sysctl.d/99-dmesg.conf
    (also added to sudoers file)
    Edit /etc/nfs.conf file to allow upd, tcp and vers2 (see copy in this dir)
    Add sudo to keybinds shortcuts for power off and drives off
    Install latest get-iplayer by adding current repository:
    sudo add-apt-repository ppa:m-grant-prg/utils
    (do not use snap version)
    Import BMTheme.theme in this Dir as Kate theme & change other config options
    Uninstall snap version of Firefox (external source editor not allowed):
    sudo snap remove firefox
    Install apt version by adding repository:
     sudo add-apt-repository ppa:mozillateam/ppa
    Prefer deb version:
    echo '
Package: *
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001
' | sudo tee /etc/apt/preferences.d/mozilla-firefox
    sudo apt install firefox
    
    

I don't have any notes on when I had to add to the options in nfs-kernel-server but I seem to remember that was crucial to getting NFS v2 back.

[BeebMaster]

Well, I've got +2 on the Pi4, so if push comes to shove I suppose I can use the Pi to re-export the NFS mounts so the RISC PC can see them!

[BeebMaster]

If I comment out the version 4 lines in nfs.conf:

Code: Select all

nfsconf --dump
[general]
 pipefs-directory = /run/rpc_pipefs

[mountd]
 manage-gids = y

[nfsd]
 tcp = y
 udp = y
 vers2 = y
 vers2,3,4 = y
 vers3 = y

then that still shows those versions as enabled, after a restart:

Code: Select all

sudo cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

So something other than nfs.conf is determining which NFS versions are enabled. I've got nothing in /etc/nfs.confd, it's an empty directory.

I'm pretty sure there was a kernel update during the update I did earlier, so it must have changed where it reads these options from.

[sweh]

Well, I've got +2 on the Pi4, so if push comes to shove I suppose I can use the Pi to re-export the NFS mounts so the RISC PC can see them!

I don't know if that will work. I know it didn't with SunOS 4 in 1992, but I've not tried with recent Linux systems.

One thing to be careful of though, especially with systemd, is the order of operations; the NFS export has to happen _after_ the disk has mounted, otherwise it might just export the empty mountpoint directory.

[BeebMaster]

Well, there's a workaround, when I finally persuaded the GRUB menu to load, which was to revert to the previous kernel. So with:

Code: Select all

uname -srm
Linux 5.19.0-50-generic x86_64

I now get:

Code: Select all

sudo cat /proc/fs/nfsd/versions
[sudo] password for ian: 
+2 +3 +4 +4.1 +4.2

And RISC PC is alive again!

It's still nothing to do with the contents of nfs.conf which is the same as earlier, ie. the whole of v4 supposed to be disabled:

Code: Select all

nfsconf --dump
[general]
 pipefs-directory = /run/rpc_pipefs

[mountd]
 manage-gids = y

[nfsd]
 tcp = y
 udp = y
 vers2 = y
 vers2,3,4 = y
 vers3 = y

So the latest kernel (6.2 something) has broken NFS v2, and both kernels are ignoring nfs.conf anyway. It's been a long time since I messed about with GRUB options, and when I have I've almost invariably ended up wiping the whole partition table. Is there a way I can default to loading with the previous kernel?

[sweh]

So the latest kernel (6.2 something) has broken NFS v2, and both kernels are ignoring nfs.conf anyway. It's been a long time since I messed about with GRUB options, and when I have I've almost invariably ended up wiping the whole partition table. Is there a way I can default to loading with the previous kernel?

NFS v2 was made option in kernel 6.2 (e.g. https://www.phoronix.com/news/Linux-6.2-NFSD ). It's possible Ubuntu had it turned on in older versions and turned it off in current.

I don't have an ubuntu system at the moment, but there might be files like /boot/config.<kernel_vesion>

On my Debian 11 machine I see

Code: Select all

% grep NFS.*_V2 /boot/config-5.10.0-23-amd64
CONFIG_NFS_V2=m
CONFIG_NFSD_V2_ACL=y

So I can see the V2 is a loadable module so it's available (and it's not NFSD, so it's using older code)

You can change the default kernel in grub by modifying /etc/default/grub but be careful; older kernels might be auto-removed as you install newer ones (to save disk space) so you might end up with a kernel that won't work with NFSv2, anyway.

Although it's harder work you may want to compile your own kernel from source (using the config file from /boot as the template but changing the NFSD_V2 option). It's easy enough if you know what you're doing, but a bit of a learning curve if you've never done it before. But this way you can ensure system upgrades will never remove it. This does carry a compatibility risk, though, so I haven't done this on my machines for over a decade.

[BeebMaster]

Looks like I have managed it without wiping everything. I added to /etc/default/grub

Code: Select all

GRUB_SAVEDEFAULT=true
GRUB_DEFAULT=saved

so it then remembers any previous Grub menu selection so I don't have to do it every time.

Ruined my Friday night this has, but I should be back in business for tomorrow.

[BeebMaster]

On my Debian 11 machine I see

Code: Select all

% grep NFS.*_V2 /boot/config-5.10.0-23-amd64
CONFIG_NFS_V2=m
CONFIG_NFSD_V2_ACL=y

I have /boot/config-6.2.0-26-generic which includes:

Code: Select all

# CONFIG_NFSD_V2 is not set
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y

whereas /boot/config-5.19.0-50-generic has:

Code: Select all

CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y

So that's been the change which breaks NFS v2 for me. I could try compiling my own kernel, but I never have done before!

[BeebMaster]

Well I wonder after all if I have compiled or built my own kernel. I've been having trouble wiping an old data drive over the last few days, getting write speed of 700K/s filling it up from /dev/zero when I would expect 100MB/s so I decided to try the old USB quirk trick I had used before, which involves putting a quirk setting in a file in modprobe.d to disable "UAS" and instead use "USB-Storage" and then adding it to the kernel using update-initramfs -u.

I noticed that it only worked on the 6.2 kernel I am not using, but update-initramfs -u -k all does them all.

(The quirk worked, but didn't help as the drive then reported in the Disc utility that it has 71,464 bad sectors!! So I think it's time to chuck it out, but I would like to wipe it first.)

Now, if this process is akin to "compiling the kernel" I am happy to have a go if I know what to put where to allow NFS v2 to be enabled in kernel.

[BeebMaster]

Mmmm searching for "compile Linux kernel with custom options" looks to be telling me that Initramfs is only part of the setup, so it doesn't seem likely I can rebuild that with an NFS v2 enable option.

Not really sure I want to build my own kernel from scratch, even if it only involves changing one option from the .config file, as presumably it would get overwritten, or need re-doing, every time a new kernel comes out.

[sweh]

initramfs isn't the kernel; it's loaded _after_ the kernel and is a initial "minimal root" filesystem which runs from RAM and has the necessary minimum configurations needed (e.g. modules, config files) to find the real filesystem and bring it online (eg RAID config, missing drivers).

It's kinda in the name; initramfs - initial RAM filesystem

[AndyMc1280]

Having read this thread and PM'd Beebmaster I have a slightly different issue regarding NFS/Ubuntu and Risc OS. He suggested I post here so the experts would see it (apologies for necro)

I have an NFS server set up and running on my RPi4. I can access from windows and linux. It is the RISC OS bit I'm stuck with. Or rather the bit on Linux that lets RO access.
I need to have PCNFSdaemon running on the server but can only find the sources https://sources.debian.org/src/pcnfsd/2.0-4/
I've tried to build that myself with I think not much success as I cant get it working.

Has anyone got an ubuntu .deb build package of Pcnfs I can beg ? It is probably a 30 second job but for some reason I can't do it properly.

ty

Andy

[BeebMaster]

I wonder if it is the same problem I had, which was because NFSv2 wasn't enabled. Does:

Code: Select all

sudo cat /proc/fs/nfsd/versions

include "+2" in the result?

What are you using on the RISC PC to access the Linux shares? In the past I used Sunfish but then moved to ImageNFS. I never got anywhere with Acorn NFS.

An alternative, which is what I do now, is to use the Pi Econet Bridge. My NFS shares are on my Ubuntu PC, which can be accessed on the Pi, and then used as "discs" in the Pi Econet Bridge file server so any Econet station can see them. With the latest Pi Bridge software, long filenames are supported (though might be a bit troublesome on the RISC OS desktop).

[BeebMaster]

Scrabbling around in the dark here but I wonder if rpcbind is the package you need? To get NFS server working on Linux I had to install that and nfs-common and nfs-kernel-server packages.

[AndyMc1280]

I wonder if it is the same problem I had, which was because NFSv2 wasn't enabled. Does:

Code: Select all

sudo cat /proc/fs/nfsd/versions

include "+2" in the result?

Ooo no it doesnt.

What are you using on the RISC PC to access the Linux shares? In the past I used Sunfish but then moved to ImageNFS. I never got anywhere with Acorn NFS.

An alternative, which is what I do now, is to use the Pi Econet Bridge. My NFS shares are on my Ubuntu PC, which can be accessed on the Pi, and then used as "discs" in the Pi Econet Bridge file server so any Econet station can see them. With the latest Pi Bridge software, long filenames are supported (though might be a bit troublesome on the RISC OS desktop).

!Omni (starting with pi) but I do have Image NFS and !Sunfish. Sadly I have zero econet hardware.

[BeebMaster]

I wonder if it is the same problem I had, which was because NFSv2 wasn't enabled. Does:

Code: Select all

sudo cat /proc/fs/nfsd/versions

include "+2" in the result?

Ooo no it doesnt.

That will be it then I think. My Pi4 shows:

Code: Select all

pi@raspberrypi:~ $ uname -srn
Linux raspberrypi 5.10.103-v7l+
pi@raspberrypi:~ $ sudo cat /proc/fs/nfsd/versions
+2 +3 +4 +4.1 +4.2
pi@raspberrypi:~ $ 

This is the current content of my /etc/default/nfs-kernel-server, with NFS v2 enabled in the last line:

Code: Select all

# Number of servers to start up

RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids"

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD="no"

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""

RPCNFSDOPTS="--nfs-version 2,3,4 --debug --syslog"

[AndyMc1280]

Code: Select all

ubuntu@pibuntu:~$ uname -srn
Linux pibuntu 6.5.0-1013-raspi

I suspect my kernel is "too new" as well

Code: Select all

 
 sudo cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

[sweh]

I need to have PCNFSdaemon running on the server but can only find the sources https://sources.debian.org/src/pcnfsd/2.0-4/
I've tried to build that myself with I think not much success as I cant get it working.

pcnfsd; now that's a name I haven't heard in a long time (read that in Alex Guinness's voice).

This may be a red herring.

PCNFS isn't an NFS server, it's an authentication service for NFS clients.

The idea is that you'd send it your username and password and it would return this uid/gid/groups/home/umask associated with that account which the client could then use to access the NFS server itself. If I remember correctly it also handled print services.

This won't solve your problem; as Beebmaster indicates, modern servers don't support NFSv2. Depending on your kernel and platform you might be able to force it into running v2, but it'll get harder and harder.

If your machine has ethernet then you might want to run the PiEconetBridge software in _AUN_ mode. You don't need econet for this (you don't even need a Pi for this; I've run it on a RedHat VM for test purposes). This uses more Acorn "native" file server constructs and avoids having to fight NFS.

[SKS1]

Looks like the stock 6.2 kernel has NFSv2 support removed. From the interwebs: "we have to boot with 5.x kernel or rebuild the kernel ourselves (with CONFIG_NFSD_V2=y)"

[BeebMaster]

As I said earlier in this thread, the workaround I found for Ubuntu was to lock to a pre-version 6 kernel which I still had installed. But the danger in that is

a. I might be letting in some security flaw by not keeping up to date
b. The old kernel might get deleted when I am not looking and jigger up grub and make my PC unbootable

So it's not really sustainable long-term. I am going to have to find some time (and I think a spare, disposable machine) where I can have a go at building my own kernel with NFSv2.

In the case of the Pi4, I think I just happen to be on an old kernel as I had never done a distribution upgrade since 2019 until last week, and whilst that updated my Raspbian version, it didn't update the kernel. Again I may have to look at whether the Pi kernel can be custom built with NFSv2.

I'm not sure what happens if you do a fresh Pi install from an old image (if they are still around (it's likely I still have the download I used to set up my Pi4 originally)), it might be possible then to start with an older kernel and update the packages only to more recent version.

[BeebMaster]

I have "2019-07-10-raspbian-buster.zip" which I can post somewhere (not on here, it's 1.1GB!) if that isn't around any more.

[sweh]

So it's not really sustainable long-term. I am going to have to find some time (and I think a spare, disposable machine) where I can have a go at building my own kernel with NFSv2.

Until the v2 code (and thus the option to use it) gets removed from the kernel entirely.

Far back in the mists of history, before kernel NFS, there was a _user space_ NFS server. This isn't as performant as the kernel space one, and has some kludginess. Because it's so old it probably doesn't support anything except UDP v2 If you can find that and get it to compile on a modern machine this might be a work-around... but it's probably gonna be lot of work!

[AndyMc1280]

Weirdly
I now have an NFS3 share going and can access from Risc OS on PI

Accessing through Omni
Didn't have to mess with either NFS 2 or PCNFS.


https://www.riscosopen.org/content/docu ... ux-cli-nfs

Applied bits of above to the setup guide from Tech Republic

RoPI can read/write to Share on another Pi running ubuntu.

Which will do me fine as any other sharing to older machines can be done via Sharefs if I cant get them to play directly with NFS

Obvs testing to be done and windows access to be sorted but for now I am happy after and for too long.

EDIT PS running latest kernel too as I said before 6.5.0-1013-raspi. so glad I didn't have to mess with that either.

[BeebMaster]

Glad you got it working, though I am not sure this solution would help my circumstances. I am using RISC PC, and as you would expect, real Acorn ROMs, ie. RISC OS 3.70. As far as I know, everything I have done to access NFS shares using the RISC PC needs the server to enable NFSv2.

It might be different with Omni (is that the same as, or part of, Acorn NFS?) and I can see I had a note that I had to disable NFS v4 to get it working, but there were so many other problems with it that I abandoned it in favour of Sunfish and then Image NFS.

I think the Acorn version appends 8 bytes of load/exec addresses to the actual file from what I recollect.

[rps102]

It might be different with Omni (is that the same as, or part of, Acorn NFS?)

OmniClient, as its name suggests, is a general purpose Filer frontend for many network filing systems. One of them is the OmniNFS module which uses Acorn NFS to do the work (so you end up with 2 modules loaded, as OmniNFS doesn't do much itself).

While AndyMc1280 may be using it on RISC OS Pi (RISC OS 5) on paper there's no reason why it wouldn't work on RISC OS 3.70. You just need Internet 5 or later, and a vaguely recent SharedCLibrary - certainly NFS works here on RISC OS 4.02 with those to criteria met.

I think the Acorn version appends 8 bytes of load/exec addresses to the actual file from what I recollect.

Not any more, that ridiculous-ness was removed in NFS 3.25 in March 2018.

[scruss]

I have "2019-07-10-raspbian-buster.zip" which I can post somewhere (not on here, it's 1.1GB!) if that isn't around any more.

Many of the Raspbian images from 2012-2020 are still available here: https://downloads.raspberrypi.org/raspbian/images/. You'll need to make sure that the Raspberry Pi is at least as old as the image, since old OS releases don't contain new firmware.

[AndyMc1280]

Testing on RO 4.39 and 6

Moved back to raspbian (latest) on an 8GB Rpi 4

Code: Select all

andrew@raspi:~ $ sudo uname -rn
raspi 6.6.20+rpt-rpi-v8

My Fangling about further. NFS 3 share running on Kinetic RPC using the RO5 Build of Omni Client. No issues. The Standard version of Omniclient is rather old (1997) so it is unsurprising that it cant cope.

Haven 't tried 3.1x yet

[philpem]

At the risk of being dunked - has anyone tried Sunfish?

I vaguely remember it being a bit of a pig to configure (compared to OmniClient) but a lot more reliable with modern servers.