Hi,
i wonder i see fully used gbit connection from Kodi/Vero when scraping from my gbit connected storage where i have like 5000 Movies (rared). I access it via NFS (fstab). I generally scrapes 12 hours a day and uses 200+ GB of Traffic.
Is this normal ? Seems a bit much just for scraping - not ?
Thanks for any ideas.
This doesn’t seem right. What scraper are you using?
You sure about scrappers? Did you check your outgoing traffic?
Have you taken checked the usage, run a scrape and then check the usage again to make sure that it’s scraping using the data?
The two that come with kodi out of the box. Movie and TV.
i see my network monitor on the firewall. The Server is in another vlan, so I see vero pumping full gbit speed for like 8-12 hours from the Server a day when im not watching anything. I have an auto update script for the scrapers running. I see around 230 Gb of traffic generated by the Vero device (when not even watching)
I will run a test tonight, check usage without scrapers running and then manually start library update.
I have to add the mysql database is on the same machine as the movies - if that matters. Still…200 GB ? hmm
You haven’t port-forwarded SSH or anything have you?
Sam
no I havent. You mean portforward ssh on vero to somewhere else ?
Well if you have the firewall you clearly see which traffic (port) from where to where is going. So maybe post some samples.
ok i startet a movie scaper update. Traffic is around 5 Mbit. So moderate. I noticed in the past, when there was a problem with the nfs server and i navigate to the mount point in veros root fs where i mount the server via fstab. I get into a loop. I can go into the path (nfs Server unreachable) and go on and on, the same dirs appear in the kodi file interface and i can click the same path and it appears again. Maybe that is the source of the problem.
I will keep looking at this over the next few days and report here. I had a huge filesystem repair on the server (xfs) and that fixed some inode problems.
I stopped using these exotic filesystems like xfs or zfs. extXfs was born on linux for the Linux OS and is rock stable. Since I use ext4fs - never had a problem.
You don’t happen to be using shared thumbnails are you?
Although not directed towards your specific issue at hand if you have a large library and feel the need to scrape it so often you might consider splitting your media files into two different folders. The folders that have content that don’t need to be scraped would have “exclude from library updates” turned on in their “set contents”.
This kind of usage is extreme, but if it is really the scraper, it should only happen the first time the library is scraped. After that, only new items should be added to the database.
If your scripts essentially “refresh” everything (which is the equivalent of forgetting each item, then scraping it) every day, that’s not needed.
ok guys, so I use XFS because when i created that fs looong time ago there was no ext4 (ready for production) yet.
I dont use shared thumbnails.
The scraper does what “update library” does from the gui. So i guess its not throwing away anything or is it ?
Have you looked at your Kodi logs to see if they are offering any insight as to what the scraper is doing? Normally you would be asked to post them but I get the impression yours might be a bit large.
Hi,
this is resolved. I had a loop. My NFS Server was malfunctioning, then the mountpoint where the share was mounted to showed the same dir from the server. When you went deeper it showed the same dir again and again. I havent tested it again but since i resoved this issue no more massive traffic…
Thanks for listening 