So I’ve heard references to HPN quite a few times now, but I’m really not sure of their performance claims. First off, their patches are constantly out of date. For example, the latest is now more than a year old (7.5).
Then it seems like the patches are quite invasive, and controversial, according to this discussion. For example, HPN implements the “null cipher”, which disables encryption, a feature the OpenSSH developers will never implement (and will of course enhance performance quite significantly). Those developers also claim there has been improvements to the OpenSSH codebase that make the HPN patches less useful.
Some basic tests seem to confirm this:
anarcat@curie:~(master)$ sftp anarc.at:/srv/test/1000M
Connected to anarc.at.
Fetching /srv/test/1000M to 1000M
/srv/test/1000M 100% 954MB 79.1MB/s 00:12
"sftp anarc.at:/srv/test/1000M" took 25 secs
anarcat@curie:~(master)$ scp anarc.at:/srv/test/1000M .
1000M 100% 954MB 92.0MB/s 00:10
"scp anarc.at:/srv/test/1000M ." took 11 secs
anarcat@curie:~(master)$ ssh anarc.at cat /srv/test/1000M | pv -s 1000000000 > 1000M
953MiO 0:00:10 [87,9MiB/s] [=======================================================================================================================>] 100%
This is pretty close to gigabit saturation (~112Mibyte/s): keep in mind we are bound by the writes on the client side as well here. Here’s an equivalent transfer with raw TCP sockets. On the client:
$ pv 1000M | nc -q 0 server 29481
953MiO 0:00:08 [ 112MiB/s] [================================>] 100%
On the client:
$ nc -q 0 -N -l -p 29481 | pv -s 1000000000 > 1000M
953MiO 0:00:08 [ 106MiB/s] [=======================================================================================================================>] 100%
"nc -q 0 -N -l -p 29481 | pv -s 1000000000 > 1000M" took 12 secs
Notice how the file transfer itself took 8 seconds and saturates the network (~112MiB/s above) but the actual pipeline doesn’t complete before 12 seconds, which is comparable to the timings of the SSH and SCP transfers in the first test. The results are lower on the Vero, because of its slower CPU, as you can see above. But it can still stream encrypted 4k videos with its small CPU…
So I think this will be enough for my needs. Of course, a major provider using SSH extensively might want to look at HPN patches to see if it could improve their service, but at my service level, I don’t think it’s necessary. I’ve used the configuration from this tutorial in my setup, and all my media was indexed quickly enough: it’s not as fast as a local disk, but that’s to be expected anyways.
Feel free, of course, to enhance this discussion with instructions on how to rebuild the latest OpenSSH server with HPN patches, but I suspect this would be more trouble than it’s really worth at this point… 