https://paste.osmc.tv/epakasusib
So there was around 25GB of free space left on this drive and I wanted to move a 45GB file to it.
So I freed up space so there’s now 67.4GB of free space.
Tried to transfer the file over the network and got an IO error.
So I moved it to a different hard drive on the Vero no problem.
Then using the Vero I tried to move it from one drive to another.
Not sure what’s happening here.
Seeing a bunch of EXT4-fs error (device sdb1): ext4_readdir:183: inode #12: comm waiting: path /media/Seagate 8TB 2/Movies: directory fails checksum at offset 0 on the target drive in the log.
Did you try running the e2fsck command that the error recommends? Is this one of the drives that you had changed the reserved space on?
e2fsck -y -f /dev/sdc1
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Seagate_8TB_2: ***** FILE SYSTEM WAS MODIFIED *****
Seagate_8TB_2: 538/1907744 files (0.4% non-contiguous), 1949369194/1953506385 blocks
exit_code = 1
This drive was formatted with mkfs.ext4 /dev/sda1 -T largefile4 -m 0 -L <LABEL>, then copied everything over.
Run this command and share the output. It will show how many inodes you have free.
sudo dumpe2fs -h /dev/sdb1
BTW, you ran the e2fsck on sdc1, not sdb1 where the error occurred.
[EDIT: I modified the dump command to only show a summary]
This is quite interesting:
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdc1): error count since last fsck: 1
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdc1): initial error at time 1600157494: ext4_validate_block_bitmap:375
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdc1): last error at time 1600157494: ext4_validate_block_bitmap:375
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdb1): error count since last fsck: 118
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdb1): initial error at time 1599970564: ext4_readdir:183: inode 12
Sep 15 01:39:16 osmc kernel: EXT4-fs (sdb1): last error at time 1600158975: ext4_readdir:183: inode 12
So sdc1 has had one error at 1600157494 (September 15, 2020 8:11:34 AM UTC/GMT)
and sdb1 has clocked up 118 errors between 1599970564 (September 13, 2020 4:16:04 AM UTC) and 1600158975 (September 15, 2020 8:36:15 AM UTC), a period of just over 52 hours.
Clearly something is amiss, though I see from the log that you mounted six drives, which is arguably pushing things a bit, so whatever it is doesn’t appear to affect the other four – on this occasion.
So, is the problem mainly confined to the Seagate 8TB 2 (sdb1) device and what hub/s are you using? Are you pushing the drives though one or two USB ports?
osmc@osmc:~$ sudo dumpe2fs -h /dev/sdb1
dumpe2fs 1.43.4 (31-Jan-2017)
Filesystem volume name: Seagate 8TB 2
Last mounted on: /media/Seagate 8TB 2
Filesystem UUID: c07e4ecb-4911-4f64-a1d8-d47b21c2d827
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags: unsigned_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 1907744
Block count: 1953506385
Reserved block count: 0
Free blocks: 4137191
Free inodes: 1907206
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 32
Inode blocks per group: 2
Flex block group size: 16
Filesystem created: Thu Sep 10 18:30:10 2020
Last mount time: Tue Sep 15 12:55:23 2020
Last write time: Tue Sep 15 12:55:23 2020
Mount count: 1
Maximum mount count: -1
Last checked: Tue Sep 15 13:12:55 2020
Check interval: 0 (<none>)
Lifetime writes: 1828 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: cacff971-73c7-4795-a4af-3b3d0c90f0db
Journal backup: inode blocks
Checksum type: crc32c
Checksum: 0x1eac08d7
Journal features: journal_incompat_revoke journal_64bit journal_checksum_v3
Journal size: 1024M
Journal length: 262144
Journal sequence: 0x0000174f
Journal start: 1
Journal checksum type: crc32c
Journal checksum: 0x8192600aI should note that I took the 8TB 2 off the Vero and plugged it into my laptop and I was able to move the files without issue.
I ran that initial e2fsck on 8TB 2 when it was mounted at sdc1.
When you did that, it may have run an fsck correcting the filesystem errors that @dillthedog pointed out.
osmc@osmc:~$ sudo umount /dev/sdb1
osmc@osmc:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 774712 0 774712 0% /dev
tmpfs 899232 16972 882260 2% /run
/dev/vero-nand/root 14499760 9840860 3899300 72% /
tmpfs 899232 0 899232 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 899232 0 899232 0% /sys/fs/cgroup
/dev/sda1 7811937256 7663537132 148383740 99% /media/Seagate 8TB 1
/dev/sde1 7811937256 6107790072 1704130800 79% /media/Seagate 8TB 4
/dev/sdc1 7811937256 7811692736 228136 100% /media/Seagate 8TB 3
/dev/sdd1 4882073396 3321156928 1560900084 69% /media/Seagate 5TB
/dev/sdf1 3905450924 3443631576 461802964 89% /media/Seagate 4TB
tmpfs 179844 0 179844 0% /run/user/1000
osmc@osmc:~$ sudo e2fsck -y -f /dev/sdb1
e2fsck 1.43.4 (31-Jan-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Seagate_8TB_2: 538/1907744 files (0.4% non-contiguous), 1949369194/1953506385 blocksI forgot, you had manually fixed it already when you did the fsck earlier. After you did that fsck did you try to copy the file again via Kodi, or did you move the drive to the laptop and copy?
Yeah, I did the copy on the laptop BEFORE running fsck.
I’m unmounting and checking all my drives one by one right now.
osmc@osmc:~$ sudo umount /dev/sdc1
osmc@osmc:~$ sudo e2fsck -y -f /dev/sdc1
e2fsck 1.43.4 (31-Jan-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: Group 59616 block bitmap does not match checksum.
FIXED.
Seagate_8TB_3: ***** FILE SYSTEM WAS MODIFIED *****
Seagate_8TB_3: 295/1907744 files (1.0% non-contiguous), 1953445255/1953506385 blocksSo when I was making room on 8TB 2 I moved a file from 8TB 2 to 8TB 3.
These are the drives showing the errors.
Something must have happened.
@dillthedog had a good question for you. How are the drives connected? All 6 on 1 USB hub, or 2 hubs on 2 usb ports? 6 drives on 1 hub would seem to me to be really pushing a USB2 connection.
Okay, so each Seagate 8TB is a “Backup Plus Hub” external drive.
They have their own power supply and a built in hub with two USB 3.0 outs.
The Vero has two USB 2.0 ports.
One USB 2.0 port has the RF Remote Dongle in it.
The other USB 2.0 port has 8TB 1 plugged into it.
Then 8TB 2 is plugged into one of the two USB 3.0 ports on 8TB 1.
Do that for all the drives in a daisy chain.
At the end I have two Seagate 2.5" external plugged in.
All the drives spin down when idle. When I watch a movie, the drive with the file spins up and everything plays perfectly!
Even full untouched DV+A UHD Remuxes without issue.
Pretty impressive.
I typically send files to the drives over the network.
But on occasion I have used the Vero’s file manager to move a file from one drive to another (like I did last night to make room).
I’m thinking read duties are fine, and write duties are fine, but maybe when the Vero does read from one and write to another, it messes up.
That many drives connected to a USB2 connection is really pushing things. Especially on a low power device like the Vero. Remember, it’s not designed as a NAS and just doesn’t have the hardware support for that kind of throughput.
I would wonder if when one of the Hub/drives goes to sleep could it cause a hiccup in its internal USB hub?
To narrow it down, you could try copying the file via command line instead of using Kodi. But I doubt that it’s Kodi as there was clearly a problem with the filesystem on at least 2 of your drives.
EDIT, another thought… You might try splitting the drives between both USB ports on the Vero and put the remote dongle into one of the drives USB ports.
Yeah, I don’t use the Vero as a NAS, more like DAS.
I can’t imagine it would be any better if I put half the drives on one USB port and the other half on the other USB port.
I’ve considered shucking them and putting them in a single enclosure (that would allow them to only take up one power outlet on my power strip) but those have their own caveats.
Like if one drive is active then they all spin, instead of the drives not in use staying idle.
Also, my current setup has worked without issue until last night so…
