Scraping advice needed
#1
Hello everyone and thanks for having me here. Smile

After years of accessing all my video content via Kodi's file browser, I'm currently trying to get everything scraped and indexed so I can take full advantage of the actual videos/movies menu: Looks so much nicer, and I can't wait to find out who my favorite actors/directors are. Wink

I'm fully aware that Kodi can't read my mind, so I'm also aware that it's up to me to name and structure my video files in a way that Kodi will be able to figure out. Most of my files are movies, and my file tree is structured in a way that each movie resides in a separate folder AND is also named appropriately - so it would seem that, for scraping, using folder names OR the actual file names would work fine. In preparation, I've also named all folders AND file names to include both the film name and the year it came out.

Which brings me to my first problem (of three): When scraping blockbusters like "Crash" (three-time academy award winner) or "Big Fish" the default scraper chooses other movies that I never heard of before. I could probably get around this by adding .nfo tags to all folders, but that's a ton of work and before I go there, I'd like to discover if/why it's really necessary. Naming a file "Big Fish (2003)" is already as accurate as it gets, and I don't understand why the scraper goes off to choose a film called "Reel Big Fish" instead when that's not actually the file name. Similarly, it would be great if I could get the scraper to automatically add the movie "Crash" when that's the name of the file, instead of picking a lesser-known film that only has the word "Crash" somewhere in the title.

The other problems I have are content-related, and I'd be happy to get some advice how I'm supposed to name/organize my content so scraping works flawlessly.

#1 I have several copies from DVDs that are not actual movies, but (animated) short film collections. Would I have to separate those from my movies and put them into an entirely different folder so they can be treated as TV shows, or is there a better way to handle them? If I had the choice, I'd prefer to keep them in the movie section (with a folder icon I can click on to access the episodes on the next screen), but if that's not possible, I understand.

#2 The biggest problems come from my "DOCUMENTARIES" folder that is currently a sub-folder of "MOVIES". That folder has categories like "Sports" or "Nature" with a vast array of films sitting underneath (sometimes with sub-categories). Some of those are full-feature documentaries, and I kinda like that they show up among the (other) movies, but there's also a bunch of rare documentaries inbetween that are not even in the movie database. That wouldn't be a problem if the scraper would just admit it didn't find anything (possibly just adding the original title and a screenshot to the database?), but since the scraper is just so generous to find SOMETHING, I end up with lots of wrong entries in the db (i.e. a huge mess).

So if there's some advice how I should handle this, I'd be really grateful. I'd rather not end up doing lots of manual edits to the database (I've been there before, but lost it all soon after as I had to delete the db) - anything that would survive a re-scrape/reinstall would be preferrable.

Many thanks in advance!
Reply
#2
Which scraper are you using? If you are not already, then switch to this one... https://kodi.wiki/view/Add-on:The_Movie_Database_Python

(2020-12-29, 06:24)kodo Wrote: the default scraper chooses other movies that I never heard of before
Try with the scraper above. If no luck then call up the Information Page for the movie and select Refresh. Pick the correct title from the offered list. If it is still not offered, then use the iMDB ID or TMDB ID as detailed in that wiki page above.

(2020-12-29, 06:24)kodo Wrote: I have several copies from DVDs that are not actual movies, but (animated) short film collections. Would I have to separate those from my movies and put them into an entirely different folder so they can be treated as TV shows, or is there a better way to handle them? If I had the choice, I'd prefer to keep them in the movie section (with a folder icon I can click on to access the episodes on the next screen), but if that's not possible, I understand.
If you don't want them scraped into the library, then add them to an Extras (wiki) folder, but I don't think this is ideal.

If you want to keep them in Movies (I personally would make a tv show out of them), then move them to their own Source in Kodi. Don't mix them in with your regular movies. Scrape the ones that are scrapable. The others, use a NFO file.
NFO templates are available here... https://kodi.wiki/view/NFO_files/Movies

(2020-12-29, 06:24)kodo Wrote: The biggest problems come from my "DOCUMENTARIES" folder that is currently a sub-folder of "MOVIES". That folder has categories like "Sports" or "Nature" with a vast array of films sitting underneath (sometimes with sub-categories). Some of those are full-feature documentaries, and I kinda like that they show up among the (other) movies, but there's also a bunch of rare documentaries inbetween that are not even in the movie database. That wouldn't be a problem if the scraper would just admit it didn't find anything (possibly just adding the original title and a screenshot to the database?), but since the scraper is just so generous to find SOMETHING, I end up with lots of wrong entries in the db (i.e. a huge mess).
Nope, don't use sub-folders. This will confuse the scraper, and it will try to scrape a movie called "Documentaries", "Sports" and "Nature" which probably accounts for some of your wrong entries.

Move those three folders out of Movies so they are all on the same level in your folder structure. Then add each as a separate Source in Kodi. Then scrape. Use the NFO template for ones that don't have entries.

Use Custom home items (wiki) to separate these into their own menu item.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#3
Thanks for the quick and helpful advice. For starters, I've changed the overall setup to a three-folder structure (Movies/Documentaries/Series) - that should help me get my movies straight (that's the bulk anyway) and then I'll go from there. And the recommended scraper is fantastic - 100% accurate so far.

I also wasn't aware that the NFO can be used for material that isn't in the database (thought it's just to point the scraper in the right direction) - that should help a great deal.

I had already started looking into Extras folders, but couldn't get it to work out of the box. Seems like I will have to edit Advancedsettings.xml for that?
https://kodi.wiki/view/Add-on:Extras#Pre...To_Library

Just to make sure I'm not missing anything: Since I have to do this manually anyway, is there any benefit at all in using the name "Extras" or could I just go with any folder name(s) I like?
Reply
#4
(2020-12-29, 13:50)kodo Wrote: I also wasn't aware that the NFO can be used for material that isn't in the database (thought it's just to point the scraper in the right direction)
That is one use of them. Read the NFO files (wiki) page to discover how they are used.

(2020-12-29, 13:50)kodo Wrote: Seems like I will have to edit Advancedsettings.xml for that?
https://kodi.wiki/view/Add-on:Extras#Pre...To_Library
Yes, that is what the wiki says.

(2020-12-29, 13:50)kodo Wrote: is there any benefit at all in using the name "Extras" or could I just go with any folder name(s) I like?
As the wiki says (which I am sure you have read) you can use only one name.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#5
I did read the wiki, but it seems my human scraping is more like the default MovieDB scraper. If I could upgrade myself to Python, I definitely would. Rofl

Having said that, I thought I had read about Extras in the core wiki as well, but guess I was just tired that night. Now that I understand it's all just up to the addon, everything makes sense (including the need to edit the advanced settings file).
Reply
#6
(2020-12-29, 23:59)kodo Wrote: If I could upgrade myself to Python, I definitely would.
?? Well what is stopping you?
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#7
I'm afraid I do need some more help...

In my folders, I've singled out three types of media: Movies, TV Shows and Documentaries.

All I really want is match those three folder categories with three buttons in the home menu, and I'm done.

- I've added the movies folder to the movies button (scraped those as movies). Check.
- I've added the TV shows folder to the TV shows button (scraped those as TV shows). Check.

As for the documentaries, I've literally spent hours looking into nodes, custom home items and whatnot. I've installed the Estuary mod, but whatever I do, things just get more complicated. Technically, my documentaries are movies, but once I scrape them (even if I do it from a different node), they end up with my other movies. If I spend a couple more hours, I will probably have figured out how to set up a filtered node that will be able to recognize the documentaries' file paths and connect that to a custom menu item on the home screen. But that still leaves me with a movies section which is now cluttered with movies AND documentaries. So should I try to hide the original movies button and set up a fake movies button instead with a different file path filter? It If something that could be super-easy becomes super-complicated, it's usually the user's fault for not doing it the right way. But if I'm getting it all wrong, what is the right way?
Reply
#8
Create a Smart playlists (wiki) for your Documentaries. Add this documentary playlist to the new button on your main menu.

Create another Smart playlist for all your movies excluding Documentaries. Replace the main Movie button with this new smart playlist.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#9
(2020-12-31, 05:42)Karellen Wrote: Create a Smart playlists (wiki) for your Documentaries. Add this documentary playlist to the new button on your main menu.

Create another Smart playlist for all your movies excluding Documentaries. Replace the main Movie button with this new smart playlist.

Yep - except when you do that, you have to rebuild the main screens for both the Film and the Documentary playlists from scratch. Because if you got with the default widgets, you'll end up on an unfiltered list of films, not the playlist you're designing it for. And once you've put in all the effort, you discover that it's actually broken, because only the top widget ever becomes clickable, while the others are not.

Next step: Look for a different skin. I picked Aura. "Excellent" choice, because it's a minimal skin, and after deactivating all the menu items in the previous skin, there's no way to enter the configuration screen and change things back.
Next step: Do research on how to change the skin back by editing configuration files. According to superuser.com you edit guisettings.xml - except when I do that, it gets overwritten and I end up with Aura anyway.
And so on. At the end of the day maybe I have to reinstall Kodi. Which is why I always try to avoid extensive configuration via the GUI. It's all gone so fast and really cripples your options when things go wrong.

And I'm not ranting, just reiterating what I wrote in my previous post: If something that should take five minutes takes a day instead, it really gives you the feeling that you've misunderstood the program and you're doing something you weren't supposed to. Except that I still haven't understood how Kodi wants me to organize my material.

Happy New Year!
Reply
#10
(2021-01-01, 01:57)kodo Wrote: and after deactivating all the menu items in the previous skin, there's no way to enter the configuration screen and change things back.
Next step: Do research on how to change the skin back by editing configuration files.
Exit Kodi. Open guisettings.xml and look for <setting id="lookandfeel.skin" default="true">skin name in here</setting> and change it to <setting id="lookandfeel.skin" default="true">skin.estuary</setting>

(2021-01-01, 01:57)kodo Wrote: Yep - except when you do that, you have to rebuild the main screens for both the Film and the Documentary playlists from scratch. Because if you got with the default widgets, you'll end up on an unfiltered list of films, not the playlist you're designing it for
You never mentioned Widgets. You will need to ask in the skin thread for whichever skin you are using.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#11
As a footnote for anyone running into similar issues, if you're on a raspberry pi, to edit guisettings.xml, you will need to ssh to the machine first and shut down kodi (systemctl stop kodi) - then it'll work.
Reply
#12
(2021-01-01, 02:18)Karellen Wrote:
(2021-01-01, 01:57)kodo Wrote:  
You never mentioned Widgets. You will need to ask in the skin thread for whichever skin you are using.

I didn't know I had to. Hell, I didn't even know that Kodi has widgets. Like I said, all I ever wanted was three buttons matching three categories. But if the default movies categories shows you thumbnails and the faux movies playlist category you just set up just shows a blatantly empty screen, something seems broken and then you look into it.
Reply
#13
(2021-01-01, 02:26)kodo Wrote: But if the default movies categories shows you thumbnails and the faux movies playlist category you just set up just shows a blatantly empty screen, something seems broken and then you look into it.
Post some screenshots so I can have a clearer understanding of where the problem lies. Are you still using Aura or another skin?
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#14
Thanks for offering your help. Smile

I managed to overcome my frustration, and I think I'm all set now. Personal guide/recommendations for anyone with similar problems:

- If you're looking for a clean skin with vertical menus for a non-HD resolution, the Estuary Mod is the way to go. Couldn't find anything comparable.
- If you want custom menu items, the widgets are useless/broken. Unless you're somehow addicted, just get rid of them.
- Best way to configure the content for each menu item is to use nodes. For new custom menu items, it doesn't make a huge difference whether you go with nodes or playlists, but the default menu items use nodes as well, so once you get a hang of nodes, you can tweak those as well and don't have to replace what's already there.

These are fairly decent instructions (written for a different skin, but the node editor is the same either way).
https://forum.kodi.tv/showthread.php?tid...pid2717986

Hope this helps someone!
Reply

Logout Mark Read Team Forum Stats Members Help
Scraping advice needed0