SFTP stalls when transferring files from NTFS drive

Help and Support Raspberry Pi
1

I need help with a particular problem. I am running RPi 3B+ with OSMC (2018.03-2). The OS is installed on 16GB SD card. I have also SamsungEVO SSD connected via USB<->SATA cable. SSD is formatted with NTFS.

I have configured TVHeadend to save the recordings on Samsung SSD. Then I want to move some from RPi to my Windows machine using SFTP. Files are typically 1-3GB.

I can connect to RPi using Filezilla from Windows or using sftp from linux box alright, but when I start the file transfer, it stalls after few seconds (~5 sec) and then timeouts.

If I move the file locally from the SSD to the SD card and then try to download it by SFTP it goes fine and the transfer speed pretty much maxes out the interface speed (~21MiB/s on RPi 3B+). But for some reason I cannot transfer the same file from the SSD without stalling after few seconds.

I have another RPi 3B (no plus) running raspbian Stretch all updated to the state of the art, to which I tried to connect the SSD as well. From there I have no problems running SFTP at full speed (~11MiB/s on RPi 3B) directly from SSD.

I noticed that on raspbian after I mounted the SSD manually it showed as “ntfs” type, on OSMC the mount shows “fuseblk”, but I guess this is just a semantics and both are running ntfs-3g drivers.

On the other hand, I do not know how to debug this problem and hopefully solve it as of now the SSD on OSMC is basically inaccessible from the other machines.

2

Ouch, that sounds ill. I suggest you start on the vsftpd end on the pi: Look at this post how you can try to get some more details out of vsftpd.

Sorry, missed the fact that you use sftp.

5

Hi,

Could be a hardware issues, have you tried swapping the sd cards over on each pi and tried testing on each device again?

Tried a different PSU on the osmc pi? (Could be under powering the ssd).

Tried a different USB port?

Thanks Tom.

6

@Tom_Doyle I have done the following:

  1. Took Raspbian SD card and put it into 3B+. Let all the USB peripherals (keyboard, dualHD TV tuner and SSD) and power supply in place.
    Raspbian run SFTP ok, though there seemed to be small performance hit when transferring from SSD as it run only at 17 MiB/s

  2. Took RPi 3B, put OSMC SD card into it and connected to the same setup as in 1). SFTP run fine at the limit of 3B ethernet (11 MiB/s)

So I put everything as it was before and run again the SFTP test on OSMC in 3B+. Now to my surprise it run fine even from the SSD (at 19 MiB/s).

So it seems fixed. But what was the problem? The only thing which changed was that I run Raspbian SD card, originally installed on 3B in 3B+ and OSMC card, originally installed on 3B+ in 3B. Is it possible that OSMC card has “reconfigured” when run in 3B to the new hardware and “fixed” the problem as the side effect?

Apart from that I use exactly same USB ports on 3B+ as before.
The SSD disk worked already before in the other situations, when recording from TVHeadend or when copying from SSD to OSMC SD card. I also disconnected it several times when checking it.

7

Hi,

Not sure why it is now resolved, but if it occurs again; I would try a replacment PSU or using a powered USB hub.

Thanks Tom.

8

Unfortunately, the problem did not go away as after a while I suffered it again. I tried stopping and disabling kodi completely, which seemed to help a bit, but nor for long either. It looked like the transfers from the SSD got stuck eventually, regardless what I was doing, or what the system was doing. The system load was not a problem as I was able to record 5 recordings simultaneously and stream another 3 channels to 3 separate clients and the system ran at 30% of CPU utilization.

But once I started either the playback of some recording from the SSD, or started SFTP file transfer, after some time 5-50 seconds the transmission stalled. When looking at the debug output from ntfs-3g driver I did not get any smarter so at the end I gave up and reformatted the drive to ext4 and since then it has been working without any problem.

My guess is that there is problem either with ntfs-3g or fuse, which starves some resource(s) of the OS and eventually blocks the read operation. I did not figure out how to avoid using fuse, so I cannot confirm whether it is one or the other.

If someone is interested, I can run some more tests.

1 Like
9

hey risa2000 I too am running RPi 3B+ Have you tried to just use the regular FTP rather than sftp SSH File Transfer. I was having an issue with transferring files also but sort of figured it out I hope lol FTP issues If you don’t mind me asking why SFTP everything’s going through your internal Network correct.

10

Sorry when I looked at the post I didn’t see all the other threads i just seen your very first post.

11

It is just that I prefer to have minimal number of doors open, so since I already use SSH for management, I use it also for occasional file transfer. Besides, since it works fine over SSH with ext4 file system I believe the problem is not in SFTP/SSH but in ntfs support in OSMC (as I stated in my post above).

1 Like
12

Hi,

Have you tried a different power supply,or a powered usb hub?

Thanks Tom.

1 Like
13

No, I did not, since I do not have any at hand. I have two same power supplies, both rated at 5V/2,5A. I believe it should be sufficient. The SSD itself is Samsung 850 EVO, rated at 2,4 W max (2,2 W average), so it should run fine from USB port. While I cannot prove it, I have already used the drive (including the same USB-SATA cable) on various system (desktops and laptops) without any issue.

I also run SMART test on the drive, which did not report any problem.

1 Like
14

Hi,

2.5 A is the recommended power-supply for the 3B+, so with the ssd drive attached I suspect you are running on the edge. I would recommended investing in a usb hub, the store has one:

4 Port USB 3.0 Powered Hub - OSMC

Thanks Tom.

1 Like
15

@Tom_Doyle, you might be right. I know that my setup is borderline on the power. There are however reasons I want to have it set up this way.

  1. The disk runs fine with ext4 system in the exactly same setup. The transfer speed seems to be even a bit higher than with ntfs. So I am not completely convinced that the power is really the issue.

  2. This is also a proof of concept of the configuration as I would like to run it with the minimal number of components. The already used dualTV WinTV Hauppauge tuner, is for example not recommended to connect over the hub. In my final application I do not want to use powered USB hub. If the RPi could not manage without, I would choose different hardware.

  3. I can simulate the power constraints by straining the tuner at the same time I run SFTP session. If the power was borderline I would expect the system will behave erratically as well when for example the tuner is running both tuners and recording/streaming 5+ channels at the same time.

The result however is that this scenario runs perfectly fine with the SSD being the recording target (for both the recordings and the timeshift) and being the source for the streaming (of another recordings) and at the same time being the source for separate SFTP transfer, when ext4 is used.

On the other hand, if you are interested and can help me debug ntfs-3g/fuse layer to prove they are not the cause, I am willing to look into it.

1 Like
16

So I was having an issue with uploading multiple files via FTP and it was locking up my Raspberry Pi 3+ and someone else suggested a powered USB hub user millham on the forum. so for good measure. I bought one I’ve yet to try to replicate that heavy load issue but it does give good comfort that I have adequate power for both my pi and drive and ability to add more hard drives if needed without worrying about power. I posted earlier in this post a link to my issue you probably already saw it but 15 bucks to have the mind at ease I think it’s worth. I think Tom_Doyle has the right idea. Not to sound like an OSMC Fanboy I wish I would have bought my hub from OSMC to support them but at the time I did not know they sold powered USB hubs.

2 Likes
17

That’s very appreciated, thank you :slight_smile:

1 Like
18

I love the free and open source world granted I have one computer that is Windows that I have to use for music software Ableton 10 but everything I run is Linux and has been for quite some time 20+yrs sometimes I feel guilty as if I’m taking advantage of it all. You guys make it an awesome product and you not only keep it for just your Hardware you have it open for the Raspberry Pi and I think that is awesome I guess I am a fanboy after all haha. I think I’ve been running OSMC for a year and during that time every update has been solid I even wrote a tutorial how to setup OpenVPN with PIA even though I don’t use third-party repositories lol that would require a VPN service but I don’t like the idea that my ISP can see what cool things I’m doing haha. I was running OSMC on a Raspberry Pi 3 and recently purchased the Raspberry Pi 3+ and for people reading this taking out the micro SD card from your original Raspberry Pi 3 and just plugging in into the new Raspberry Pi 3+ works without a hitch. However I did do a fresh install because I’m a nerd. Well I went off on a tangent but to the OSMC people that make this possible thank you very much!

3 Likes
20

Well, if anyone is still interested in some news on my original problem, I did some tests with an old JMicron USB<->SATA/PATA bridge I dug up in the closet.

The bridge can take external power (5V) and connects to the disk by simple SATA cable so in the end I was able to completely eliminate power dependency by powering both the bridge and the SSD from external power supply (5V/12V).

I also disconnected all other USB peripherals (= TV tuner, keyboard), having just ethernet and the bridge connected.

I formatted the drive with two partitions, one ext4 one ntfs, mounted both partitions in OSMC and run SFTP transfers from both partitions (i.e. copying recordings from RPi to my Windows machine), using Filezilla configured for 2 simultaneous transfers.

  1. running the test from ext4 did not show any problem, I was able to transfer approx 10 GB of data in approx 13 files while almost maxing ethernet port speed at 20 MiB/s.

  2. running the test from ntfs partition gave the results as described in my original post. The transfers started and after approx. 5 seconds came to halt.

FWIW I would like to mention that I configured Filezilla with 120 seconds timeout. Which means, once the connection halts and there is no activity for 120 seconds, the SFTP client disconnects from the server, but since there are still some transfers in the queue it immediately reconnects and tries to continue the transfers. Right after the reconnection there is a small time frame of a fraction of a second where the transfer continues and then it halts again until a new timeout.

The logs (journalctl, dmesg) do not show anything.

If I mount the FS with debug option manually (udisks does not allow debug for some reason), like this:

osmc@htpi:~$ sudo mount /dev/sda1 -o debug
Version 2016.2.22AR.1 integrated FUSE 28
Mounted /dev/sda1 (Read-Write, label "SamsungEVO", NTFS 3.1)
Cmdline options: rw,noatime,debug,nofail
Mount options: rw,nofail,allow_other,nonempty,noatime,fsname=/dev/sda1,blkdev,blksize=4096
Ownership and permissions disabled, configuration type 7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
   INIT: 7.18
   flags=0x00000841
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, error: 0 (Success), outsize: 40
<...etc...>

and the corresponding /etc/fstab

osmc@htpi:~$ cat /etc/fstab
/dev/mmcblk0p1  /boot    vfat     defaults,noatime,noauto,x-systemd.automount    0   0
# rootfs is not mounted in fstab as we do it via initramfs. Uncomment for remount (slower boot)
#/dev/mmcblk0p2  /    ext4      defaults,noatime    0   0
UUID="8c06d108-f71e-474f-a876-5ae42aa1f3bc"     /mnt/SamsungEVO ext4    defaults,nofail,noatime,x-systemd.automount   0       0
UUID="E452860F5285E6A0" /mnt/SamsungEVO ntfs    defaults,nofail,noatime,x-systemd.automount     0       0

I do not see any error in the ntfs debug log, all operations are reported with success and then it simply dies/timeouts.

The time it takes to stall/die is quite volatile and can be from few seconds to minute and half, but it halts eventually.

21

If you want you can try to create some SSH service debug information. Perhaps, you can get more helpful information out of this what’s going on your system.

Please, precisely follow these steps to catch debug information for a specific SSH or SFTP problem

  1. open an SSH connection to the OSMC device, user osmc, password osmc
    (the debug information can only be collected for the NEXT SSH-connection. So, prepare your environment/clients, so the problem/issue will occur with the next SSH connection. For SFTP-transfer problems it means you have to connect to the OSMC device with your SFTP-client using appropriate credentials and the next action will be the file transfer which will create another SSH-session for the transport)

  2. sudo systemctl stop sshd
    (this stops the sshd father process but your current session process still keeps living)

  3. sudo /usr/sbin/sshd -D -ddd &>my_sshd_debug.txt
    (this starts a debug sshd which is able to accept one session, only)

  4. reproduce the issue like a failed ssh login or failing sftp transfer
    (the called sshd debug program will automatically terminate when the SSH session is closed; otherwise abort it with ctrl+C after the issue occured)

  5. sudo paste-log my_sshd_debug.txt
    (this transfers the debug information and returns a URL)

  6. sudo systemctl restart sshd
    (restarts the sshd service to normal operation, again)

  7. provide the URL from previous step in this forum topic.

1 Like
22

@JimKnopf I did as you suggested. Just two remarks:

  1. It seems FileZilla opens second SSH channel for file transfer, so I had to start the debug session right before I initiated file transfer in FileZilla client, i.e. after I have connected to the RPi. Which means the initial login is not captured in the log, just the file transfer.

  2. I killed the debug session after the file transfer stalled to avoid cluttering the log with the client timeout and the reconnection.

Here is the log: https://paste.osmc.tv/aniqihacof
For the info, the SFTP file transfer was running for about 34 seconds before it halted. The part in the log related to the transfer itself is quite short, but too cryptic for me to make an opinion.

23

I connected an NTFS formatted hdd to my Pi3B (not +), configured it the same in fstab and collected the same sshd debug information pushing a file to the Pi3 using Filezilla SFTP. While transfer I hit ctlr-C.

Comparing your sshd log with mine does not show any significant difference or hint what could have caused the stalled transfer with your system. Sorry, this was a one-way road.