Backported cifs.ko fails / Better SMB support

qowy · 24 February 2022 13:47

Hi,
since the Vero 4k is still using 4.9 it exists in this weird in between state that it supports SMB 3.0 but not with encryption. This usually leads to it being rejected form Windows Servers with SMB ecryption (for other devices) and having to fallback to SMB 2.1
Additionally I really would like to use enryption

Since the nice folks at Samba provide backports for the 4.9 Kernel I tried compiling my own cifs.ko

I installed vero364-headers-4.9.113-60-osmc and aarch64-toolchain-osmc and then used chroot to build the module

This worked. However trying to insmod the resulting cifs.ko results in the error

 cifs: Unknown symbol user_key_payload (err 0)

Now I presume that these backports work with the mainline 4.9 kernel (otherwise I would have to take this upstream) so any idea why this failes with the vero 4k kernel?

I will additionally try to maybe find a point in time in the backports before this symbol was required and try again. But I am not deep enough in the code to figure exactly what that will break

qowy · 24 February 2022 15:06

Ok looking at the compilation again I see this compiler warning

fs/cifs/connect.c:2635:13: warning: implicit declaration of function 'user_key_payload'; did you mean 'user_key_payload_rcu'? [-Wimplicit-function-declaration]

Changing this successfully compiles and loads the module.
Now the question remains is this an error in the backports or is the Vero 4k kernel missing a patch here.

Anyhow, smb3 still does not work with this compiled module

[4069478.670436] CIFS VFS: failed to connect to IPC (rc=-11)
[4069478.672118] CIFS VFS: session ffffffc052130e00 has no tcon available for a dfs referral request
[4069478.672401] CIFS VFS: cifs_mount failed w/return code = -11

I am guessing the dfs stuff is just an irrelevant error but the IPC connection failure might again be caused by the kernel.
So the question remains is this something I can take to the samba maintainers or is there some issue with the Vero kernel

sam_nazarko · 25 February 2022 21:06

Hi,

Thanks for bringing this to my attention.

The branch seems to be based on the initial 4.9.0 release.
I’m sure that some of the commits are now in the upstream kernel.

Do you know if there is another patchset somewhere?

Sam

qowy · 26 February 2022 08:46

I don’t know.

I followed this guide LinuxSMB3 build backport - SambaWiki

I am sadly not well enough versed in the linux kernel development to know how all of the patch backporting for older kernels is organized.

sam_nazarko · 27 February 2022 00:55

SMB3 has been in since 3.12.

So you’re after SMB encryption specifically?

Can you explain how to enable this on a Windows Server instance to repro?

Backport doesn’t look trivial, but I can definitely look in to it.

Cheers

Sam

qowy · 27 February 2022 09:06

The problem is that newer Windows Server are either configured or at least encouraged to enforce SMB3 encryption.
While SMB3 seems to have been implemented, up until 4.19 the whole encryption implementation is incomplete and that seems to interfere somehow.
Before the update to kernel 4.9 it seems to have worked with something along the lines of:

mount -t cifs -o vers=3.0

however since 4.9 this fails with an error probably because encryption support is there but not really.

these both fail (or all other variants of version 3.x):

mount -t cifs -o vers=3.0 OR mount -t cifs -o vers=3.0,seal

the only thing working is

mount -t cifs -o vers=2.1

With the 4.19 kernel of the rpi3 image

mount -t cifs -o vers=3.1.1,seal

is working fine with encryption

My windows server is configured to encrypt smb 3 but to not reject clients that do not support it.
My guess is 4.9 has enough support to report I support encryption, but then fails during the actual negotiation.

Get-SmbServerConfiguration
...
EncryptData                            : True
RejectUnencryptedAccess                : False

These can be configured via

Set-SmbServerConfiguration -EncryptData $true

etc.

sam_nazarko · 27 February 2022 12:14

I’ll check it out. I think I read that 4.12 has enough implementation to support things.

Thanks

sam_nazarko · 27 February 2022 12:19

Do you have the error message at hand?

qowy · 28 February 2022 07:39

in both cases (with or without seal) it is mount error(13): Permission denied

dmesg says:

[4389659.370294] CIFS VFS: SMB3 encryption not supported yet
[4389659.373099] CIFS VFS: SMB3 encryption not supported yet
[4389659.374778] CIFS VFS: cifs_put_smb_ses: Session Logoff failure rc=-13
[4389659.375049] CIFS VFS: cifs_mount failed w/return code = -13

And yes the not supported message always appears twice

You are right btw, i was mistaken, according to this even 4.11 should be enough to complete the work on SMB3 encryption

https://wiki.samba.org/index.php/LinuxCIFSKernel

qowy · 22 April 2022 11:48

Any news on this?

sam_nazarko · 22 April 2022 13:11

I looked at this, but couldn’t see a way to backport this in a stable way to our kernel at this time.

sam_nazarko · 22 April 2022 13:15

After looking again, it seems that even 4.8 kernel had sufficient support, so I’ll check config options later to see if anything needs changing.

sam_nazarko · 26 April 2022 04:27

Hi,

I’ve been looking in to this, as I can see that this feature is getting more important. Azure seem to implement this by default now.

Unfortunately I don’t have a way to test this.

But I have backported the functionality you have requested.
This is a very aggressive backport and took a lot of work.

As such, it needs a lot of testing. I have sent you a PM to a test image.

I can’t get this in to the stable release without lots of feedback and testing.

Thanks for your understanding

Sam

qowy · 2 May 2022 08:11

is the HEVC playback issues test image the correct one?

sam_nazarko · 4 May 2022 11:25

Yes, that’s correct.

qowy · 1 January 2023 22:55

Hi,
I never got around to test the implementation in the HEVC Test builds before Bullseye was released.

I have now updated to the current release, and the issue somehow got worse.

mountig with explicit vers=2.1 still works as before.
However using vers=3.0 (with or without seal aka encryption) now failes silently.
No error is returned from the mount command, but a kernel exception in logged in dmesg.

A short excerpt before the register contents start:

[  198.116809] Unable to handle kernel NULL pointer dereference at virtual address 00000014
[  198.119739] pgd = 00000000a7cac5e8
[  198.123221] [0000000000000014] *pgd=0000000000000000, *pud=0000000000000000
[  198.130347] Internal error: Oops: 96000005 [#3] PREEMPT SMP
[  198.136102] Modules linked in: arc4 md4 cifs ccm 8021q aes_ce_blk ablk_helper aes_ce_cipher ghash_ce sha2_ce sha1_ce dhd mali ir_lirc_codec lirc_dev meson_ir amvdec_ports(O) amvdec_mmpeg4(O) amvdec_mh264(O) amvdec_h264mvc(O) amvdec_mmpeg12(O) amvdec_vp9(O) amvdec_vc1(O) amvdec_h265(O) stream_input(O) decoder_common(O) firmware(O) media_clock(O) video_framerate_adapter(O)
[  198.168964] CPU: 0 PID: 3194 Comm: mount.cifs Tainted: G      D    O    4.9.269-16-osmc #1
[  198.177412] Hardware name: Vero4KPlus (DT)
[  198.181728] task: 000000000675a5d3 task.stack: 000000001ba51fa6
[  198.187945] PC is at SMB2_tcon+0x194/0x338 [cifs]
[  198.192848] LR is at SMB2_tcon+0x17c/0x338 [cifs]
[  198.197683] R9  : ffffff8020273940, PFN:3ede2
[  198.202253] R27 : ffffff8009bf5000, PFN: 1bf5
[  198.206823] R29 : ffffff8020273bb0, PFN:3ede2
[  198.211395] R30 : ffffff8002125edc, PFN:3efe5
[  198.215966] pc : [<ffffff8002125ef4>] lr : [<ffffff8002125edc>] pstate: 60000145
[  198.223555] sp : ffffff8020273bb0

sam_nazarko · 2 January 2023 10:29

Without a full log, it’s anyone’s guess.

Sam

qowy · 2 January 2023 16:42

tell me what you need, and I will provide

The full dump in dmesg?

sam_nazarko · 2 January 2023 20:31

That would certainly help. There was some discussion about SMB3 recently,but the user got it working without the need for additional changes.

qowy · 3 January 2023 12:23

Unable to handle kernel NULL pointer dereference at virtual address 00000014

pgd = 00000000a7cac5e8

[0000000000000014] *pgd=0000000000000000, *pud=0000000000000000

Internal error: Oops: 96000005 [#3] PREEMPT SMP

Modules linked in: arc4 md4 cifs ccm 8021q aes_ce_blk ablk_helper aes_ce_cipher ghash_ce sha2_ce sha1_ce dhd mali ir_lirc_codec lirc_dev meson_ir amvdec_ports(O) amvdec_mmpeg4(O) amvdec_mh264(O) amvdec_h264mvc(O) amvdec_mmpeg12(O) amvdec_vp9(O) amvdec_vc1(O) amvdec_h265(O) stream_input(O) decoder_common(O) firmware(O) media_clock(O) video_framerate_adapter(O)

CPU: 0 PID: 3194 Comm: mount.cifs Tainted: G D O 4.9.269-16-osmc #1

Hardware name: Vero4KPlus (DT)

task: 000000000675a5d3 task.stack: 000000001ba51fa6

PC is at SMB2_tcon+0x194/0x338 [cifs]

LR is at SMB2_tcon+0x17c/0x338 [cifs]

R9 : ffffff8020273940, PFN:3ede2

R27 : ffffff8009bf5000, PFN: 1bf5

R29 : ffffff8020273bb0, PFN:3ede2

R30 : ffffff8002125edc, PFN:3efe5

pc : [<ffffff8002125ef4>] lr : [<ffffff8002125edc>] pstate: 60000145

sp : ffffff8020273bb0

x29: ffffff8020273bb0 x28: ffffffc03ec66600

x27: ffffff8009bf5000 x26: ffffffc03ee57c00

x25: 0000000000000008 x24: ffffffc056583400

x23: ffffffc041147f40 x22: ffffffc05430c000

x21: 0000000000000008 x20: 00000000fffffff5

x19: ffffffc04e333800 x18: 0000000011317781

x17: 0000000000007210 x16: 00000000000002a0

x15: 0000000000000000 x14: 0000000000000001

x13: 0000000100000000 x12: 0000000000000176

x11: 0000000000000024 x10: 00000000000008d0

x9 : ffffff8020273940 x8 : ffffffc03ec66f30

x7 : 0000000000000000 x6 : 000000000000967c

x5 : 00000001226d7bc2 x4 : 0000000000000000

x3 : 0000000000000000 x2 : 0000000000000001

x1 : 0000000000000001 x0 : 0000000000000001

X8: 0xffffffc03ec66eb0:

6eb0 00000000 00000000 00000000 00000001 00000000 00000000 00000001 00000000

6ed0 57aa4c80 ffffffc0 3ec66600 ffffffc0 57aa4c80 ffffffc0 3ec9aa00 ffffffc0

6ef0 0a430008 ffffff80 09bddfc8 ffffff80 00000001 00000000 3ec66b38 ffffffc0

6f10 09bf9000 ffffff80 3ec66600 ffffffc0 20273560 ffffff80 20273560 ffffff80

6f30 09086540 ffffff80 f7318fc0 00000000 00000000 00000000 00000000 00000000

6f50 38383931 74656230 2c302e33 3d646975 30303031 6469672c 3030313d 61702c30

6f70 313d7373 30383839 61746562 73752c32 6d3d7265 6f637261 6578652c 75732c63

6f90 f7320406 f7320004 aae04f54 aae04f58 00000000 00000000 00000000 00000000

X19: 0xffffffc04e333780:

3780 01050000 00010000 04400000 00000000 03500000 00000000 01050000 00010000

37a0 04e00000 00000000 03ac0000 00000000 01050000 00010000 0ab40000 00000000

37c0 2e000000 746d7973 2e006261 74727473 2e006261 74736873 62617472 72672e00

37e0 0070756f 6c65722e 65742e61 2e007478 61746164 73622e00 742e0073 2e747865

3800 4e333800 ffffffc0 4e333800 ffffffc0 00000001 00000000 00000000 00000000

3820 00000000 00000000 4e333828 ffffffc0 4e333828 ffffffc0 00000000 00000000

3840 56583400 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000

3860 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

X22: 0xffffffc05430bf80:

bf80 00000000 00000000 5430bf88 ffffffc0 5430bf88 ffffffc0 00000000 00000000

bfa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

bfc0 00000000 00000000 00000000 00000000 09a95654 ffffff80 00000000 00000000

bfe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

c000 8dfe9186 5d8a45b5 8aed92cf 9045da6b 592f57ea 7096eb4e d80d24df b37dbdf3

c020 3d66cb7e ff3221c8 7a8f9c72 fdf2f2dc 92a78793 7665136b 73656369 616c702f

c040 4ec46674 c65128ac 0d6b1fac 00081eea 80000045 0040f53d 266c0680 c867a8c0

c060 4367a8c0 64c6bd01 37957f79 4787671a 01201880 0000363f 0a080101 a45d7d54

X23: 0xffffffc041147ec0:

7ec0 38383931 74656230 00003261 00000000 00000000 00000000 00000000 00000000

7ee0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

7f00 411478c0 ffffffc0 50b13d48 ffffffc0 4da59c08 ffffffc0 41147e88 ffffffc0

7f20 00000000 00000000 000224dc 75650109 706a5f63 0079702e 00000000 00000000

7f40 69775c5c 65732d6e 72657672 2076545c 20646e61 69766f4d 00007365 00000000

7f60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

7f80 41147c00 ffffffc0 00000000 00000000 00000000 00000000 37744836 bbd10785

7fa0 00fcbe23 00000000 0000007e 00010000 80000015 00002055 00000000 00000000

X24: 0xffffffc056583380:

3380 00000000 00000000 56583388 ffffffc0 56583388 ffffffc0 00010001 00000000

33a0 56584a00 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000

33c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

33e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

3400 50a7b810 ffffffc0 50a7b810 ffffffc0 56583410 ffffffc0 56583410 ffffffc0

3420 00000001 00000000 56583428 ffffffc0 56583428 ffffffc0 00000000 00000000

3440 00000000 00000000 50a7b800 ffffffc0 00000003 00000001 00000000 00000000

3460 00000000 00000000 00000000 00000000 00000000 00000000 80000015 00002055

X26: 0xffffffc03ee57b80:

7b80 3ee57b80 ffffffc0 3ee57b80 ffffffc0 00000000 00000000 00000000 00000000

7ba0 00000000 00000000 00000000 00000000 00000000 00000000 0921d350 ffffff80

7bc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

7be0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

7c00 3ee57dc0 ffffffc0 b354b176 637a4fd9 ef715fc5 764fff67 b15fe3f7 25652e59

7c20 4b4d5570 135420e2 cd641c7f 124e818c 34e6d50e 6dfec3d0 47a62aed 9c499784

7c40 47400fc4 6d032cb1 f4e02f28 00000000 00000000 00000000 00000000 00000000

7c60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

X28: 0xffffffc03ec66580:

6580 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

65a0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

65c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

65e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

6600 0040001a 00000000 ffffffff ffffffff 00000001 00000000 00000000 00000000

6620 20270000 ffffff80 00000002 00404100 00000000 00000000 00000000 00000000

6640 00000001 00000000 0000000d 00000000 ffffd833 00000000 4db41980 ffffffc0

6660 00000000 00000001 00000078 00000078 00000078 00000000 09bf95e8 ffffff80

Process mount.cifs (pid: 3194, stack limit = 0x000000001ba51fa6)

Stack: (0xffffff8020273bb0 to 0xffffff8020274000)

3ba0: ffffff8020273c40 ffffff8002106b5c

3bc0: ffffffc04e333800 ffffffc056583400 ffffffc0554efa00 0000000000000006

3be0: ffffffc056583410 0000000000000008 ffffffc04e342858 ffffffc0574d3000

3c00: 0000000020273c20 ffffffc03ee57c00 0000000000000008 ffffff8002106b88

3c20: ffffffc03ee57c00 000000000000004c ffffffc05430c000 0000000000000036

3c40: ffffff8020273c80 ffffff80021087c0 ffffffc0554efa00 ffffffc04e342800

3c60: ffffffc056583400 0000000000000006 0000000000000000 ffffffc050a7b800

3c80: ffffff8020273cf0 ffffff80020facd8 0000000000000000 0000000000000000

3ca0: ffffffc0554efa00 ffffffc04e342800 ffffff8002158010 0000000000000000

3cc0: ffffffc03ee57a60 ffffffc0574d3000 ffffff8009bf5000 ffffffc03ec66600

3ce0: ffffffc0574d3000 0000000000000000 ffffff8020273d60 ffffff80091ffaa8

3d00: ffffffc0574d3000 ffffffc03ac73000 0000000000000000 ffffffc041147980

3d20: ffffff8002158010 0000000000000000 ffffffc03ee57a60 ffffff80091ffa3c

3d40: ffffffc0574d3000 ffffffc03ac73000 0000000000000000 ffffffc041147980

3d60: ffffff8020273db0 ffffff800921c46c ffffffc03ee57a40 ffffff8002158010

3d80: 0000000000000000 ffffffc041147980 ffffffc03ee57a60 ffffffc0574d3000

3da0: ffffffc041147d40 ffffffc0574d3000 ffffff8020273df0 ffffff800921f744

3dc0: 0000000000000020 0000000000000000 0000000000000000 ffffffc041147980

3de0: 0000000000000001 ffffff8002158010 ffffff8020273e80 ffffff8009255cf8

3e00: ffffffc0574d3000 ffffffc041147d40 00000000aae04ebc ffffffc041147980

3e20: 0000000000000000 0000000000000011 00000000000001b3 0000000000000015

3e40: ffffff8009bf5000 ffffffc03ec66600 ffffffc055fec020 ffffffc047f76b40

3e60: 00000000aae04ebc ffffffc041147980 0000000000000000 0000000000000011

3e80: 0000000000000000 ffffff8009083b40 fffffffffffffe62 000000406a715000

3ea0: ffffffffffffffff 00000000f72884ea 0000000060000030 0000000000000000

3ec0: 00000000ffba38a1 00000000aae04ebc 00000000aae04e1c 0000000000000000

3ee0: 00000000abc5d360 00000000abc5c358 00000000abc5d360 0000000000000015

3f00: 00000000f7320000 00000000abc5c308 0000000000000000 00000000ffba38a1

3f20: 00000000aae15fc8 00000000ffba2a40 00000000aae00cab 0000000000000000

3f40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

3f60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

3f80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

3fa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

3fc0: 00000000f72884ea 0000000060000030 00000000ffba38a1 0000000000000015

3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

Call trace:

Exception stack(0xffffff8020273a60 to 0xffffff8020273b90)

3a60: ffffffc04e333800 0000007fffffffff ffffffc074b52590 ffffff8002125edc

3a80: 0000000000000001 0000000000000001 0000000000000001 0000000000000000

3aa0: 0000000000000000 00000001226d7bc2 000000000000967c 0000000000000000

3ac0: ffffffc03ec66f30 ffffff8020273940 00000000000008d0 0000000000000024

3ae0: 0000000000000176 0000000100000000 0000000000000001 0000000000000000

3b00: 00000000000002a0 0000000000007210 0000000011317781 ffffffc04e333800

3b20: 00000000fffffff5 0000000000000008 ffffffc05430c000 ffffffc041147f40

3b40: ffffffc056583400 0000000000000008 ffffffc03ee57c00 ffffff8009bf5000

3b60: ffffffc03ec66600 ffffff8020273bb0 ffffff8002125edc ffffff8020273bb0

3b80: ffffff8002125ef4 0000000060000145

[ffffff8020273bb0+ 144][<ffffff8002125ef4>] SMB2_tcon+0x194/0x338 [cifs]

[ffffff8020273c40+ 64][<ffffff8002106b5c>] cifs_get_tcon+0x114/0x2d8 [cifs]

[ffffff8020273c80+ 112][<ffffff80021087c0>] cifs_mount+0x4e8/0x928 [cifs]

[ffffff8020273cf0+ 112][<ffffff80020facd8>] cifs_do_mount+0xa8/0x3c0 [cifs]

[ffffff8020273d60+ 80][<ffffff80091ffaa8>] mount_fs+0xd0/0x17c

[ffffff8020273db0+ 64][<ffffff800921c46c>] vfs_kern_mount+0x98/0x144

[ffffff8020273df0+ 144][<ffffff800921f744>] do_mount+0x650/0xa64

[ffffff8020273e80+ 0][<ffffff8009255cf8>] compat_SyS_mount+0x118/0x1e8

[0000000000000000+ 0][<ffffff8009083b40>] el0_svc_naked+0x34/0x38

Code: b4000093 39570e60 32000000 39170e60 (b9400ea1)

---[ end trace 8563ba24c929f702 ]---