Yep, it would be because range is filling up the memory. Look for the line I highlighted and change range to xrange.
Ok, works with the fixes from @bmillham
You might also want to exclude external file systems. Otherwise it will scan your servers if you call it with /
With some tweaking:
insignificance_pattern = re.compile('var/log/(faillog|lastlog)$')
could also ignore /media and /mnt. That’s your homework for tomorrow @Oliver_O
Well, it would not cross mount points (otherwise we’d already get in trouble with /dev
, /proc
and others): Did you notice not os.path.ismount(self.location))
in the tree()
function?
I’m aware that the whole thing is neither designed for performance nor is it production quality software. I’ll look at the suggestions. Which python versions did you use?
I’d be even more interested in results from running the script on OSMC. Anyone?
Well it was complaining about my samba mount under /mnt/server so not sure why if it suppose to ignore it
yes as I wrote after I did the fixes from @bmillham it worked on my Pi.
I ran it on two of them and didn’t had any null_bytes.
It’s probably because some mount point is not reported as being a mount point.
I could hardwire the usual suspects (/media
and /mnt
) to exclude these from traversal, but then users could still choose arbitrary directories for such stuff. So maybe add an option to be safe.
My input. I made two changes to the original script:
from __future__ import print_function
and
for index in xrange(1, file_size + 1):
Here’s the result on my test Pi3:
osmc@osmc-test:~$ sudo ./files-null-end.py /
/home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img: ending with 512 null bytes starting at 0 [1K boundary], 512 bytes total
That one file is from the OSMC git clone. The card is a new SanDisk Ultra 16GB.
Yep and as it is an image file I think it is normal to have null bytes at the end
Not so fast.
osmc@osmc-test:~/osmc$ md5sum /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
bf619eac0cdf3f68d496ea9344137e8b /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
Downloaded from Github:
[user@fedora-25 Downloads]$ md5sum bs.img
1e5867dc321146fa8c2d5a6147d5c722 bs.img
The file has not changed for 3 years.
Also:
ls -l /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
-rw-r--r-- 1 osmc osmc 512 Oct 9 08:00 /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
[user@fedora-25 Downloads]$ ls -l bs.img
-rw-rw-r-- 1 user user 43299 Oct 23 13:41 bs.img
So it seems to be all null bytes. I did have a power outage a few weeks back, so it’s possible that it might have corrupted the file.
osmc@osmc-test:~$ md5sum /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
bf619eac0cdf3f68d496ea9344137e8b /home/osmc/osmc/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
Don’t chase ghosts
I assume you didn’t download the raw file
https://github.com/osmc/osmc/raw/master/installer/host/qt_host_installer/w32-lib/usbit32/bs.img
bs.img is a blank 512 or 448 byte file.
It’s intentionally blank and is used to erase the partition table if a user ever wants Windows to see all of the space as unformatted
Here’s an update of the tool for you to try. Includes compatibility fixes, avoids descending into /media
and /mnt
and offers an option to exclude even more directories from traversal.
https://paste.osmc.tv/sivucaceli.py
Example using the new --exclude
option:
show-files-ending-with-null-bytes --exclude /my/mount/path_1 /my/mount/path_2 -- /
Of course, you could also use the new option to exclude directories where you suspect too many legitimate files ending with null bytes.
Confirm your new code is working fine!
Also as written before none of my cards shows any null bytes (other than the bs.img which is supposed to have them)
Excellent!
Which card brands did you test and on which Pi models?
P2: SanDisk Class4
P3: Samsung Evo Plus
I didn’t; I downloaded the file from tree, which I (perhaps rashly) assumed would be a faithful copy of the file from the git clone. So it seems that the raw file is in what seems to be a hidden (“ghost” ) directory, that’s not visible via the browser. And is not the same as the one in tree.
@sam_nazarko Why do the files differ between raw and tree?
The bs file?
I don’t think we include it anymore.
Sam