ISCSI not possible

Help and Support Raspberry Pi
1

I am running my Pi for years now using owncloud / nextcloud. The folder /var/www is stored on an external USB 3,5" drive.
I am running OSMC (Debian based)

for data security I bought a synology NAS, and created a RAID1. Then i created a ISCSI Lun there, and mounted it on my pi.
Whenever I try to copy data from the usb drive to the ISCSI drive, i get the following errors like this:

[May 2 21:23] scsi host1: iSCSI Initiator over TCP/IP
[ +0.028159] scsi 1:0:0:0: Direct-Access QNAP iSCSI Storage 4.0 PQ: 0 ANSI: 5
[ +0.005795] sd 1:0:0:0: Attached scsi generic sg0 type 0
[ +0.014391] sd 1:0:0:0: [sdb] 524288000 512-byte logical blocks: (268 GB/250 GiB)
[ +0.000538] sd 1:0:0:0: [sdb] Write Protect is off
[ +0.000013] sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 08
[ +0.001151] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA
[ +0.032439] sdb: sdb1
[ +0.006503] sd 1:0:0:0: [sdb] Attached SCSI disk
[ +3.255125] BTRFS: device fsid 5df90d67-6a28-4611-b85c-27f2dc8704df devid 1 transid 92 /dev/sdb1
[May 2 21:24] BTRFS info (device sdb1): use zlib compression
[ +0.000028] BTRFS info (device sdb1): enabling auto defrag
[ +0.000016] BTRFS info (device sdb1): enabling inode map caching
[ +0.000010] BTRFS info (device sdb1): disk space caching is enabled
[ +0.000009] BTRFS info (device sdb1): has skinny extents
[ +13.207348] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 19966309, last ping 19966816, now 19967320
[ +0.000065] connection2:0: detected conn error (1022)

it is even not possible to format the disk because I alsways get timeout and conn error messages in dmesg.

Even if I unplug the USB harddisk the error is still there.
If I use a debian based distro everything works fine.

My tests are:

  • Heavy read/writes to the remote LUN => iSCSI: not OK
  • Formatting the local drive, heavy local r/w over USB => drive ok
  • Formatting the remote drive, heavy remote LUN => iSCSI: not ok, even if no USB is plugged in.

My hardware
Raspberry Pi 3+
PRETTY_NAME=“Open Source Media Center”
NAME=“OSMC”
VERSION=“March 2018”
VERSION_ID=“2018.03-2”
ID=osmc
ID_LIKE=debian

2

Has this only recently started happening since you moved to a 3B +?

Can you try dropping the Pi to 100Mbps only using ethtool?

Sam

3

Although I understand you want to solve the problem, iSCSI doesn’t gain you anything with media sharing, and actually makes it harder to share the data with other machines. Just create a partition on the Synology, lay down a filesystem, and share it with NFS.

4

Thanks for your answer, but i use this rasberry also heavily for datasharing using apache. thats why i want to move the www folder from the external usb harddisk to a external RAID1 using ISCSI. I tried nfs but synology and qnap do not allow to switch user ids, so nfs did not work for me.

5

Hi, not it also happened when I had my Pi 3.

I thought it might be a hardware problem and bought the Pi 3+, but no success.
Now I can use the old Pi 3 with Raspian, and it works. But I want to use my OSMC Pi 3+ with iscsi because i also useit for data sharing, multimedia box and so on.

6

Can you try dropping the speed to 100Mbps as a test?

Unfortunately I don’t know a lot about iSCSI or BTRFS to advise.

7

I did using ethtool but no success. If there is only small load to the iscsi device it works, but heavy IO seems to cause a disconnect.

8

I’ve seen this same problem with newer kernels and the Intel 10Gbit adapters when hosting an iSCSI target. I still haven’t figured it out, so use an older kernel (4.10.1 is the last one that worked).

But, I haven’t seen the issue when using Linux as an iSCSI initiator.

I tried nfs but synology and qnap do not allow to switch user ids, so nfs did not work for me.

I’m not sure what you mean by “switch user IDs”. Every NFS server can be set up to allow full access from any user ID. You likely need to disable “root_squash” (or whatever Synology calls it) on the share, and then everything should work fine. You would create the file system as usual on the NAS, and then mount it from the Pi, and then make directories you need and set them to the right permissions. All this should be done as root on the Pi.

1 Like
9

I tried it, but my webserver and especially nextcloud did not work because of wrong access rights.
Thats why i decided to use iscsi.
My old Pi 3 is now a debian box running kernel 4.9.35-v7+ #1014 where ISCSI works as expected so it might be one of the causes.

10

Hi, just bought the official PSU for raspberry, but the error remains. Interestingly i can copy big files for some minutes, then it begins to stuck…

[May 5 10:48] scsi host1: iSCSI Initiator over TCP/IP
[ +0.031593] scsi 1:0:0:0: Direct-Access QNAP iSCSI Storage 4.0 PQ: 0 ANSI: 5
[ +0.003837] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ +0.002597] sd 1:0:0:0: [sdb] 524288000 512-byte logical blocks: (268 GB/250 G iB)
[ +0.000682] sd 1:0:0:0: [sdb] Write Protect is off
[ +0.000025] sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 08
[ +0.001334] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, does n’t support DPO or FUA
[ +0.010337] sdb: sdb1
[ +0.009722] sd 1:0:0:0: [sdb] Attached SCSI disk
[ +0.328048] BTRFS: device fsid 5df90d67-6a28-4611-b85c-27f2dc8704df devid 1 tr ansid 271 /dev/sdb1
[ +4.867307] BTRFS info (device sdb1): disk space caching is enabled
[ +0.000012] BTRFS info (device sdb1): has skinny extents
[ +0.072175] BTRFS info (device sdb1): bdev /dev/sdb1 errs: wr 36, rd 0, flush 0, corrupt 0, gen 0
[ +22.875484] BTRFS info (device sdb1): the free space cache file (91297415168) is invalid, skip it
[May 5 11:00] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 56843, last ping 56472, now 57864
[ +0.000054] connection2:0: detected conn error (1022)
[May 5 11:05] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 87151, last ping 87656, now 88160
[ +0.000035] connection2:0: detected conn error (1022)
[ +22.240074] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 89361, last ping 88864, now 90384
[ +0.000069] connection2:0: detected conn error (1022)
[May 5 11:06] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 91591, last ping 91096, now 92616
[ +0.000059] connection2:0: detected conn error (1022)
[ +22.320025] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 93821, last ping 93335, now 94848
[ +0.000059] connection2:0: detected conn error (1022)
[May 5 11:07] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 96049, last ping 95560, now 97072
[ +0.000053] connection2:0: detected conn error (1022)
[ +22.240023] connection2:0: ping timeout of 5 secs expired, recv timeout 5, la st rx 98273, last ping 97788, now 99296
[ +0.000078] connection2:0: detected conn error (1022)
[ +0.001633] print_req_error: I/O error, dev sdb, sector 218190848
[ +0.000013] sd 1:0:0:0: [sdb] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x0e dri verbyte=0x00
[ +0.000005] sd 1:0:0:0: [sdb] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x0e dri verbyte=0x00
[ +0.000021] sd 1:0:0:0: [sdb] tag#27 CDB: opcode=0x2a 2a 00 0d 01 90 00 00 04 00 00
[ +0.000009] sd 1:0:0:0: [sdb] tag#28 CDB: opcode=0x2a 2a 00 0d 01 3c 00 00 04 00 00
[ +0.000009] print_req_error: I/O error, dev sdb, sector 218206208
[ +0.000006] print_req_error: I/O error, dev sdb, sector 218184704
[ +0.000173] sd 1:0:0:0: [sdb] tag#30 UNKNOWN(0x2003) Result: hostbyte=0x0e dri verbyte=0x00
[ +0.000005] sd 1:0:0:0: [sdb] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x0e dri verbyte=0x00
[ +0.000003] sd 1:0:0:0: [sdb] tag#31 UNKNOWN(0x2003) Result: hostbyte=0x0e dri verbyte=0x00
[ +0.000008] sd 1:0:0:0: [sdb] tag#29 CDB: opcode=0x2a 2a 00 0d 01 7c 00 00 04 00 00
[ +0.000004] sd 1:0:0:0: [sdb] tag#31 CDB: opcode=0x2a 2a 00 0d 01 58 00 00 04 00 00
[ +0.000005] sd 1:0:0:0: [sdb] tag#30 CDB: opcode=0x2a 2a 00 0d 01 8c 00 00 04 00 00
[ +0.000004] print_req_error: I/O error, dev sdb, sector 218191872
[ +0.000006] print_req_error: I/O error, dev sdb, sector 218201088
[ +0.000011] print_req_error: I/O error, dev sdb, sector 218205184
[ +0.000122] sd 1:0:0:0: [sdb] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x0e driv erbyte=0x00
[ +0.000004] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x0e driv erbyte=0x00
[ +0.000003] sd 1:0:0:0: [sdb] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x0e driv erbyte=0x00
[ +0.000008] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x2a 2a 00 0d 00 25 c8 00 04 0 0 00
[ +0.000003] sd 1:0:0:0: [sdb] tag#1 CDB: opcode=0x2a 2a 00 0d 01 9c 00 00 04 0 0 00
[ +0.000004] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x2a 2a 00 0d 01 2c 00 00 04 0 0 00
[ +0.000004] print_req_error: I/O error, dev sdb, sector 218209280
[ +0.000007] print_req_error: I/O error, dev sdb, sector 218113480
[ +0.000005] print_req_error: I/O error, dev sdb, sector 218180608
[ +0.000113] print_req_error: I/O error, dev sdb, sector 218208256
[ +0.000004] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x0e driv erbyte=0x00
[ +0.000004] sd 1:0:0:0: [sdb] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x0e driv erbyte=0x00
[ +0.000007] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x2a 2a 00 0d 01 5c 00 00 04 0 0 00
[ +0.000011] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x2a 2a 00 0d 01 74 00 00 04 0 0 00
[ +0.000016] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 38, rd 0, flush 0, corrupt 0, gen 0
[ +0.000184] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 39, rd 0, flush 0, corrupt 0, gen 0
[ +0.001244] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 40, rd 0, flush 0, corrupt 0, gen 0
[ +0.000096] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 41, rd 0, flush 0, corrupt 0, gen 0
[ +0.000047] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 42, rd 0, flush 0, corrupt 0, gen 0
[ +0.001195] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 43, rd 0, flush 0, corrupt 0, gen 0
[ +0.000140] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 44, rd 0, flush 0, corrupt 0, gen 0
[ +0.000059] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 45, rd 0, flush 0, corrupt 0, gen 0
[ +0.001123] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 46, rd 0, flush 0, corrupt 0, gen 0
[ +0.000163] BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 47, rd 0, flush 0, corrupt 0, gen 0
[ +22.233521] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 100497, last ping 100012, now 101520
[ +0.000054] connection2:0: detected conn error (1022)

13

OK folks, I think i found the root cause:

I have successful tested everything on my Pi3 running Raspian (Debian 9.4). Webserver etc. runs very smooth on my Pi3 with QNAP connected via ISCSI.

Then I switched to Pi3+ which won´t boot (rainbow screen) so I have to run rpi-update getting a newer kernel.

With kernel 4.14.39-v7 on raspian I now have exactly the same behaviour like I had on OSMC before.

So it must be a kernel bug. Wit old kernel 4.9 it did not happen.

Who can help to adress that behaviour?

14

As I understand things. iSCSI works fine on a Pi3 (not plus). The only problems are occurring on a pi3B+.

In your first post, you say that you were using OSMC 2018.03-2. That is the first version that includes support for the extended capabilities of the Pi3B+ and its kernel version was 4.14.26-2. Unless Sam has backported the new ethernet driver/firmware – which I don’t believe he has done – no OSMC kernel version before 4.14.26-2 will have support for the higher-speed ethernet.

I don’t (yet) have a Pi3B+ so I’m unable to run tests but Raspbian and OSMC will use the same ethernet driver/firmware. OSMC is currently on kernel version 4.14.34-1 and Raspbian is on 4.14.39-v7.

But kernel 4.9 didn’t have the new ethernet driver/firmware for the Pi3B+.

My best guess is that this is most likely to be a driver/firmware issue, since, AFAICT, kernel 4.14 still works fine on the Pi3.

The obvious workaround is to use NFS. In post #4 you said:

the meaning of which is unclear to me and @nabsltd. Perhaps you could explain in a bit more detail what the problem is with NFS.

15

Sure.

My Pi3 works fine with raspian running Kernel 4.9. No problems even under heavy usage.
When switching to Pi3+ I need to use rpi-update to get a newer kernel. Using the newer kernel leads to the same ISCSI errors running on OSMC.

Network speed with Pi3 is about 10MBit/s, Pi3+ reaches 25Mbit so the new network interface seem to be faster.

NFS:
I use NFS on the QNAP, and copied all the stuff from the USB Harddisk to the NFS Share.
If using ROOT_SQUASH Cahnge ownership of files has been forbidden, switching the nfs server to NOROOT_SQUASH works, because chang ewoner ship of files worked, so all files are there 1:1.
If I restart the webserver, i see apaches test welcome page. When I open my nextcloud subfolder, i get “access denied for config.php”. But the file exists and has the same rights than on my usb harddisk.

So NFS, which i use heavily at work, did not seem to run on my QNAP/Pi combination. My thought was, even if I use NFSv4, the different userids might cause problems. So I wanted switch back to ISCSI to have full control of my data.

16

Clearly, having a 1:1 UID/GID will work best with NFS, so I’d have thought it’s then a question of getting to the bottom of the “access denied for config.php” error. If it’s simply a permissions problem,it should be relatively simple to sort out.

17

When you copied, did you use “cp -a”? This will make sure that almost everything is copied exactly, including all permissions.

One other thing to look for is if there are hard links in the directory tree, these will not be copied, but instead you will end up with multiple identical copies in place of one file with hard links. This can cause issues, depending on how the application works.

Also, if there are any non-relative soft links (i.e., any that start with “/”), then those still point back to the files on the Pi, and that can also cause problems.

18

I just made a new try today, and after copying the files from usb to synology NFS Share via rsync -avug … everything works. SO i stopped my ISCSI experience which does not work for me now. When copying huge amopunt of files via NFS I got from time to time a hanging terminal window so I had to close it, and restart the rsync process. Maybe the network controller from Pi is not useful for this, I don´t know, but my webserver is now up and running on raspian again using volume on QNAP. My OSMC will only be used now as mediaplayer, and mount stuff via nfs from qnap.
Thanks for your help, I really appreciate this!

1 Like
19

i see that this is really old but if you are still trying or for anyone that may have stumbled upon this as i have. I am just beginning to set up a pi cluster (active / passive) with a third asus tinkerboardS running as a storage node to play around with fail over clustering using ISCSI.

so i am resurrecting this topic to provide a thought. It was mentioned that the ISCSI mount appeared to be working but would fail under high IO load or that is does work on a 3 but not a 3+. This sound like a queuing issue, possibly resolvable by lowering the queue depth to effectively throttle the IO and avoid high response times and time outs. The older versions of the kernal running on the 3 may have a different default value than the 3+. It is probably a setting in the ethernet driver.

I have not personally tested this yet on a PI with iscsi but i have used queue depth tuning in a fibre channel SAN environment so it does work to prevent slow drains in a SAN network.