[Howto] Newbies guide to filesystems

if you get around for it add it to OP :slight_smile:

A more complete set of benchmarks is available, but the runs can be limited to multiple file system benchmarks.

https://www.phoronix-test-suite.com/ it is GPLv3.

My idea with this tread is more or less on how it performs on the device as intended :slight_smile:

I don’t have a Vero to perform these tests and in any case other contributors have done this already, so I thought of running a couple of tests on a quad-core Linux PC to compare read and write speeds between ext4 and NTFS, on the same internal disk. This is not the same comparison as Vero would use these filesystems in real operating scenarios, using USB drive(s), NAS, SMB/NFS, etc. Nevertheless, it is a pure filesystem against filesystem comparison, eliminating differences due to network links between devices, USB protocols, SMB/NFS transfer protocols, etc. The tests were performed on an old 1TB 5400 rpm Seagate SATA:

Model Family:     Seagate Samsung SpinPoint M8 (AF)
Device Model:     ST1000LM024 HN-M101MBB
Serial Number:    S30YJ9HF422853
LU WWN Device Id: 5 0004cf 20d020ff5
Firmware Version: 2BA30001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)


The two partitions are as follows:
    TARGET          SOURCE      FSTYPE    LABEL        OPTIONS
    ├─/home         /dev/sda11  ext4      /home        rw,noatime 
    └─/mnt/Windows  /dev/sda5   fuseblk   Windows8_OS rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096                       

I copied a 3GB file containing random bits (created with /dev/urandom) from an ext4 partition to a ramdisk (on tmpfs) and back, after I flushed the caches before each copying operation. The file was deleted from the destination and the test repeated two more times. Then I repeated the test on an NTFS partition, mounted with ntfs-3g (which uses fuse).

These tests are not constrained by network or USB interface bottlenecks, use the same spinning disk and if you exclude the different position of each partition on the disk, offer a fair comparison of read/write performance between the two filesystems.

I didn’t try the same test with many smaller files, because playing or recording with a Vero device usually involves large video files. Using a 3G file is large enough to load the bus on the MoBo, without taking a long time to complete the test.

  1. Copying 3G from ext4 to tmpfs.
~ # dd if=/home/michael/3G_file of=/mnt/temp/3G_file bs=2048 conv=fsync status=progress
    2973485056 bytes (3.0 GB, 2.8 GiB) copied, 29 s, 103 MB/s
    1500000+0 records in
    1500000+0 records out
    3072000000 bytes (3.1 GB, 2.9 GiB) copied, 29.9496 s, 103 MB/s
  1. Copying 3G from NTFS to tmpfs.
~ # dd if=/mnt/Windows/Users/Lenovo/3G_file of=/mnt/temp/3G_file bs=2048 conv=fsync status=progress
3048720384 bytes (3.0 GB, 2.8 GiB) copied, 28 s, 109 MB/s
1500000+0 records in
1500000+0 records out
3072000000 bytes (3.1 GB, 2.9 GiB) copied, 28.2123 s, 109 MB/s

It seems the read speed on NTFS is marginally higher than ext4! O_o Similar results were obtained with of=/dev/null. The ext4 partition is on /dev/sda11, while the NTFS is on /dev/sda5 and we’re talking about a 2.5"spinning disk. I wonder if the position of each partition on the disk and the corresponding different angular velocity explains the difference in read speeds?

  1. Copying 3G from tmpfs to ext4
~ # dd if=/mnt/temp/3G_file of=/home/michael/3G_file bs=2048 conv=fsync status=progress
3038511104 bytes (3.0 GB, 2.8 GiB) copied, 17 s, 179 MB/s
1500000+0 records in
1500000+0 records out
3072000000 bytes (3.1 GB, 2.9 GiB) copied, 37.5235 s, 81.9 MB/s
  1. Copying 3G from tmpfs to NTFS
~ # dd if=/mnt/temp/3G_file of=/mnt/Windows/Users/Lenovo/3G_file bs=2048 conv=fsync status=progress
3062716416 bytes (3.1 GB, 2.9 GiB) copied, 103 s, 29.7 MB/s
1500000+0 records in
1500000+0 records out
3072000000 bytes (3.1 GB, 2.9 GiB) copied, 103.976 s, 29.5 MB/s

At least on this particular PC and disk although the write speed to NTFS is significantly slower than writing to ext4, read speeds are comparable.

please edit your post and use the code tag

image

the wall of text is horrible

anyways nice contribution :slight_smile:

I still haven’t finished all tests (actually struggle with my SMB tests as I don’t have gigabit on the Vero).
But yes results are quite similar to yours.
If you just use the drive to read files the difference is not that big but if you also write (e.g. torrent client) it’s a different story.
While the CPU used is a fundamental point on the Vero.

Apologies for the wall of text - I tried to use the </> code tag, but for some reason it doesn’t work here (perhaps it doesn’t like uBlock Origin on Firefox). If an administrator can change it the look, please do so to make it easier to read.

On a PC the difference on CPU load is rather small, one core sees up to 10% more load when writing via fuse on NTFS. This is a test which ought to be run on a Vero, to see the real world effects on available resources. The only reason I ran it here on my PC is to isolate other factors and look at a filesystem to filesystem comparison only on a Linux OS. I thought it may be useful to inform user’s decisions on selecting a filesystem, before they start comparing additional protocol and application layers, which can also affect file transfer performance.

Question and a comment, or vice versa :slight_smile:
I bought the Vero to replace a Windows HTPC, thus, my external hard drives were all NTFS. Converting multiple hard drives (terabytes worth of data) to ext4 is not feasible.

Question - when adding new hard drives, can I format the new ones to ext4 while keeping the others NTFS?

There are no issues with using drives with different file systems at the same time if that is what you are asking.

It was, thank you.

Microsoft Publishes exFAT Specification, Encourages Linux Support - Phoronix says that MSFT is Open Sourcing their exFAT specification and adding it to that no-lawsuit patent portfolio for F/LOSS projects.
This should mean that an exFAT kernel driver will be possible someday.

Today we’re pleased to announce that Microsoft is supporting the addition of Microsoft’s exFAT technology to the Linux kernel…It’s important to us that the Linux community can make use of exFAT included in the Linux kernel with confidence.

Are there any advantages to using ext2 rather than ext4 for small (e.g. USB stick flash) drives?

Hi,

Probably will have slightly quicker r/w speeds, due to no journal.

Tom.

ext2 isnt really used that much anymore atleast to my knowledge, would advice using ext4.

Yeah I anyhow added it in the comparison. But seems not much better.

Slowly making progress. In the end I think the recommendation will be the same:
If you just read from the disk it doesn’t make a big difference which FS you use. But if you also write to the disk (e.g. Torrent) then avoid NTFS.
Will still do Samba test and also one point I want to look into is resiliency (e.g. unplug power from the Vero while writing to the FS). And also doing same test with Pi3

Command Vero
Time (secs) SyS Time (Secs) CPU (%)
System User
time sh -c “dd if=/dev/zero of=/mnt/ntfs_test/test.tmp bs=32k count=2000000 && sync” 1844 284 66 40
time sh -c “dd if=/dev/zero of=/mnt/btrfs_test/test.tmp bs=32k count=2000000 && sync” 1609 137 16 0.1
time sh -c “dd if=/dev/zero of=/mnt/ext2_test/test.tmp bs=32k count=2000000 && sync” 1640 303 7.4 0.1
time sh -c “dd if=/dev/zero of=/mnt/ext4_test/test.tmp bs=32k count=2000000 && sync” 1596 299 9.6 0.2
time sh -c “dd if=/dev/zero of=/mnt/exfat_test/test.tmp bs=32k count=2000000 && sync” 1615 156 5.4 0.2
time sh -c “dd if=/dev/zero of=/mnt/ntfs_test/test.tmp bs=4k count=10000000 && sync” 1497 241 50 34
time sh -c “dd if=/dev/zero of=/mnt/btrfs_test/test.tmp bs=4k count=10000000 && sync” 1010 126 14 0.1
time sh -c “dd if=/dev/zero of=/mnt/ext2_test/test.tmp bs=4k count=10000000 && sync” 1050 206 5 0.2
time sh -c “dd if=/dev/zero of=/mnt/ext4_test/test.tmp bs=4k count=10000000 && sync” 1002 192 11 0.1
time sh -c “dd if=/dev/zero of=/mnt/exfat_test/test.tmp bs=4k count=10000000 && sync” 1016 151 8.4 0.6
time sh -c “dd if=/mnt/ntfs_test/test.mkv of=/dev/null bs=4k && sync” (44G File) 1195 146 49 4
time sh -c “dd if=/mnt/btrfs_test/test.mkv of=/dev/null bs=4k && sync” (44G File) 1091 182 42 1
time sh -c “dd if=/mnt/ext2_test/test.mkv of=/dev/null bs=4k && sync” (44G File) 1152 256 30 1.6
time sh -c “dd if=/mnt/ext4_test/test.mkv of=/dev/null bs=4k && sync” (44G File) 1126 175 38 1.4
time sh -c “dd if=/mnt/exfat_test/test.mkv of=/dev/null bs=4k && sync” (44G File) 1176 130 36 2.2
time sh -c "dd if=/mnt/ntfs_test/test.mkv of=/dev/null bs=32k && sync (44G File) 1182 115 38 2.5
time sh -c “dd if=/mnt/btrfs_test/test.mkv of=/dev/null bs=32k && sync” (44G File) 1083 187 30 0.5
time sh -c “dd if=/mnt/ext2_test/test.mkv of=/dev/null bs=32k && sync” (44G File) 1161 245 28 0.6
time sh -c “dd if=/mnt/ext4_test/test.mkv of=/dev/null bs=32k && sync” (44G File) 1150 229 36 0.5
time sh -c “dd if=/mnt/exfat_test/test.mkv of=/dev/null bs=32k && sync” (44G File) 1201 142 47 2.2

Whats the definition of large? Also which Client OS? Just asking as I just copied a 44G file to extfat via SMB. It was slower than expected but I didn’t get an error.

I’ve had both hard errors(windows10 gives an error msg and stops the upload) and “pauses”, where during the upload to Vero4k (both exfat and NTFS), it halted to 0b/s then resumed again to normal speed, to suddenly drop again to resume… … … making the upload taking a LONG time. Those times it has been files around 2-4gb

I tried an 8 GB file from Win 10 to Vero with Samba server from app shop.