managing multiple scrapers
#1
I use the simple file structure at the moment

Code:
TV Shows                  <<Source folder, Content: TV shows>>
   |----TV Show 1
   |       |----Season #
   |            |--Files
   |----TV Show 2 (year)
   |       |--Files
   |----TV Show 3
   |       |----2008
   |            |--Files

I have to use multiple scrapers, since not all TV shows are available/complete in a single database. For each scraper I have to define a directory path. It can either be the directory of multiple TV shows or of a single TV show. If I choose "TV Shows" (path in simple file structure) for each scraper, they come into conflict with each other. Configuring the scraper for each TV show directory ("TV Show 1", "TV Show 2 (year)", "TV Show 3", etc. in simple file structure) in the UI on my TV is cumbersome. I could create a root directory for each scraper (configuration)

Code:
TV Shows
   |----thetvdb                  <<Source folder, Content: TV shows, Scraper: thetvdb.com - default settings>>
       |----TV Show 1
       |       |----Season #
       |            |--Files
       |----TV Show 2 (year)
       |       |--Files
   |----thetvdb 2                <<Source folder, Content: TV shows, Scraper: thetvdb.com - custom settings>>
       |----TV Show 3
       ...
   |----themoviedb               <<Source folder, Content: TV shows, Scraper: themoviedb.org - default settings>>
       |----TV Show 4
       ...

but I don't feel confident of that approach. How do you manage multiple scrapers?
Reply
#2
Just stating the obvious, but I hope you are not using the same root folder for both movies and tv shows?
Because Kodi cannot tell those apart. Better use different starting points for both movies and tv shows.

/root/Movies/
|----Movie1
|----Movie2

/root/TV Shows
|----TV Show 1
| |----Season #
| |----Season #
|----TV Show 2
| |----Season #

If necessary, you can (re)set the content for each tv show that doesn't get the metadata using the default scraper.
Reply
#3
I use different directories for TV shows and movies like in your example. Resetting the content saves some manual work, but it gets cumbersome if the number of TV shows that is not matched by the default scraper increases. I also would prefer to document which scraper (configuration) is used for which TV show. Thereby, I can better recover my scraper settings or share it with someone else. I'd appreciate if this documentation could be processed automatically (less work).
Reply
#4
@catpult,

Your proposed directory structure shown in your first post is perfect. There is no issue with having multiple TV Show sources, and each of those sources can have separate scrapers set up, so there is no concern there. In fact it is actively encouraged to have multiple sources, rather than one for larger libraries.

I do question what tv shows are in one scraper site, but not in another. Can you provide a few examples of which shows are not in the TVDB but are listed at TMDB? Maybe it is a language related listing?

As these scraper sites are free to use, why don't you contribute back by updating the missing data? A lot of members do that... and it is just a small gesture of thanks to those that are providing this service at no cost.

Quote:Resetting the content saves some manual work
I don't understand this statement. What are you doing here?

Quote:but it gets cumbersome if the number of TV shows that is not matched by the default scraper increases.
The normal procedure is to check your tv show listing at the scraper first. This ensures you have the correct name applied to your files.

Quote:I also would prefer to document which scraper (configuration) is used for which TV show
Why would you want to do this? Seems like an incredible waste of time. Why not just check at the scraper site?

Quote:]Thereby, I can better recover my scraper settings or share it with someone else
What do you mean "recover scraper settings". Again, why share this when it is already at the scraper site.

Quote:I'd appreciate if this documentation could be processed automatically (less work).
What documentation?

Please feel free to clarify anything I have misunderstood from your posts Smile
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#5
(2017-07-08, 22:34)Karellen Wrote: I do question what tv shows are in one scraper site, but not in another. Can you provide a few examples of which shows are not in the TVDB but are listed at TMDB? Maybe it is a language related listing?

For example, I'm missing the second season of fate/zero in the TVTB.

(2017-07-08, 22:34)Karellen Wrote: As these scraper sites are free to use, why don't you contribute back by updating the missing data?

I'm willing to contribute. May I copy the missing data of fate/zero from TMBD to the TVDB or does this violate the terms of service?

(2017-07-08, 22:34)Karellen Wrote:
Quote:Resetting the content saves some manual work
I don't understand this statement. What are you doing here?

I do what @Klojum suggested:

(2017-07-08, 20:04)Klojum Wrote: If necessary, you can (re)set the content for each tv show that doesn't get the metadata using the default scraper.

First, I define a scraper for my "TV shows" directory. To reset the content of a particular TV show I select the directory of the TV show in the file list, open the context menu and change the content or more specifically the scraper (configuration). I have do this for all TV shows that do not match the default scraper. This is less work compared to setting each TV show directory as a source, but more work than creating a root directory for each scraper (configuration). At least, if I have several TV shows that do not match the default scraper. In return, I can keep the simple file structure if I reset the scraper.

(2017-07-08, 22:34)Karellen Wrote:
Quote:I also would prefer to document which scraper (configuration) is used for which TV show
Why would you want to do this? Seems like an incredible waste of time. Why not just check at the scraper site?

If I connect my media source to another Kodi machine, I have to configure all scrapers (sources) again for that machine. I have to remember which configuration to use if I don't write it down. I consider to share my media source (for example a NAS) between multiple devices. In that case, I have to keep the scraper configuration on each device up to date. What is the best way to do this? If I create a root directory for each scraper I don't have to remember (since it is documented/written down) which scraper to use. However, if I don't use the default settings, I have to remember the settings. I could write down those settings in a simple text file to look them up, but I have to remember to update the scrapers on all devices if I change the settings, since my simple text file is not processed automatically by Kodi.

@Klojum @Karellen Thanks so far. I really appreciate your help.
Reply
#6
(2017-07-09, 10:10)catapult Wrote:
(2017-07-08, 22:34)Karellen Wrote: I do question what tv shows are in one scraper site, but not in another. Can you provide a few examples of which shows are not in the TVDB but are listed at TMDB? Maybe it is a language related listing?

For example, I'm missing the second season of fate/zero in the TVTB.
Actually, all episodes are there. At TMDB, it is split into two seasons. At TVDB, it is one season. Have a look

Quote:
(2017-07-08, 22:34)Karellen Wrote: As these scraper sites are free to use, why don't you contribute back by updating the missing data?

I'm willing to contribute. May I copy the missing data of fate/zero from TMBD to the TVDB or does this violate the terms of service?
I don't know, you will have to check. I would imagine that you could change the wording a little bit, just so it doesn't look like a straight copy and paste.


Quote:
(2017-07-08, 22:34)Karellen Wrote:
Quote:Resetting the content saves some manual work
I don't understand this statement. What are you doing here?

I do what @Klojum suggested:
ok

Quote:
(2017-07-08, 20:04)Klojum Wrote: If necessary, you can (re)set the content for each tv show that doesn't get the metadata using the default scraper.

First, I define a scraper for my "TV shows" directory. To reset the content of a particular TV show I select the directory of the TV show in the file list, open the context menu and change the content or more specifically the scraper (configuration). I have do this for all TV shows that do not match the default scraper. This is less work compared to setting each TV show directory as a source, but more work than creating a root directory for each scraper (configuration). At least, if I have several TV shows that do not match the default scraper. In return, I can keep the simple file structure if I reset the scraper.
I just think you are creating too much work for yourself. Set up two source directories as follows:
f:\TV Shows-TVDB\
f:\TV Shows-TMDB\
You set the scrapers just once.

You then check your show at both those scraper sites. Whichever site you believe has the better info, then place the show in that proper source.

If you just do a tiny amount of research before adding things, honestly it will save you all these problems.


Quote:
(2017-07-08, 22:34)Karellen Wrote:
Quote:I also would prefer to document which scraper (configuration) is used for which TV show
Why would you want to do this? Seems like an incredible waste of time. Why not just check at the scraper site?

If I connect my media source to another Kodi machine, I have to configure all scrapers (sources) again for that machine. I have to remember which configuration to use if I don't write it down. I consider to share my media source (for example a NAS) between multiple devices. In that case, I have to keep the scraper configuration on each device up to date. What is the best way to do this? If I create a root directory for each scraper I don't have to remember (since it is documented/written down) which scraper to use. However, if I don't use the default settings, I have to remember the settings. I could write down those settings in a simple text file to look them up, but I have to remember to update the scrapers on all devices if I change the settings, since my simple text file is not processed automatically by Kodi.
Ok, I understand now what you mean. I thought you wanted to share your nfo files or metadata.

Again it is not that hard. How many machines are you setting up? Two or three? You have your sources defined as above. When you add a new Kodi, just point at your two sources. They are named so that at a glance you can see which scraper site it uses. There is no need to edit individual shows.

But another solution would be this... http://kodi.wiki/view/Import-Export_library

Once you have your library set up exactly as you want it, then export the library to separate nfo files. When you add a new Kodi, you don't have to rely on the scraper site. You can just set Kodi to scan the local information and it will reproduce the library.



Quote:@Klojum @Karellen Thanks so far. I really appreciate your help.
No problems Smile
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#7
(2017-07-09, 10:49)Karellen Wrote:
(2017-07-09, 10:10)catapult Wrote: For example, I'm missing the second season of fate/zero in the TVTB.
Actually, all episodes are there. At TMDB, it is split into two seasons. At TVDB, it is one season. Have a look

Thanks, I've overlooked that.

(2017-07-09, 10:49)Karellen Wrote: How many machines are you setting up? Two or three? You have your sources defined as above. When you add a new Kodi, just point at your two sources. They are named so that at a glance you can see which scraper site it uses. There is no need to edit individual shows.

Three and I think you are right. I feel more confident of the approach to create a root directory for each scraper. I guess that is simple enough.

By the way, what about TV shows that I cann't find in any database. For example, I haven't found "DragonBall Z Abridged" yet, which is a quite popular fan-made parody. Is there a scraper that only uses the file name/path as source of information (no database)

Code:
TV Shows (TVDB)                    <<Source folder, Content: TV shows, Scraper: TVDB >>
   |...
TV Shows (TMDB)                    <<Source folder, Content: TV shows, Scraper: TMDB >>
   |...
TV Shows (unknown)                  <<Source folder, Content: TV shows, Scraper: File Name Scraper >>
   |----DragonBall Z Abridged
   |       |----Season 1
   |            |--S01E01.mkv
   |            |--S01E02.mkv
   |            |...
   |       |----Season 2
   |            |--S02E01.mkv
   |            |--S02E02.mkv
   |            |...

I just like to get the show listed. I'm okay with episode numbers as episode name. I don't need any description. Quick and dirty is better than nothing.
Reply
#8
Looks like you have a bit of Anime in your collection. Might be worth looking at this... https://forum.kodi.tv/showthread.php?tid=142835

Regarding your DragonBall Z Abridged... http://www.thetvdb.com/?tab=series&id=248580&lid=7

Have a look at the TVDB forum discussion here... https://forums.thetvdb.com/viewtopic.php...ll#p109176

Also don't forget you can create your own nfo files for Kodi to scrape into the library... http://kodi.wiki/view/Nfo There are programs around that can do these for you.

And remember to name the files correctly. The easiest naming scheme is Firefly S01E01

---Edit---
In your diagram above, how do you intend to get TV Shows (unknown) to scrape?
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#9
Thanks, I'm surprised that I haven't found DragonBall Z Abridged using the search.

I have set the scraper TVDB for the directory "TV Shows" and reset the subdirectory "TV Shows/Fate Zero" to TMDB. How can I remove this resetting such that the scraper of the parent directory is used again? I could change the content (re-set the scraper) to TVDB, but I guess that still overwrites the setting of the parent directory scraper even if it is the same setting. I like to switch to the new file structure. Nevertheless, I'd like to undo this setting before to keep my settings clean.
Reply
#10
Yes, sometimes the searches can be a bit funny at TVDB.

I don't know how far along you now are and how many shows you have scraped into the library. If you haven't done too many, I would completely remove all your TV Show sources and let the Clean Library run. Then re-add your TV Show sources. This will clear all the sub-directory settings. Personally, if this is all a major change. I would remove the sources. Do all your organising in Explorer (if you are using windows), then once all the shows are moved to the correct sources, add the sources back to Kodi.

If you have done too much work already, then you will need to reset them individually.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#11
Thanks, the resetting worked like you described.

(2017-07-09, 12:25)Karellen Wrote: In your diagram above, how do you intend to get TV Shows (unknown) to scrape?

Similar to any other scraper, scan "TV Shows (unknown)" for directories. For each directory: Add a TV show with the directory name as title to the library. Don't look up in a database if this TV show exists. Deeply scan (i.e. including subdirectories like "Season 1") each of those directories for files (e.g. "S01E01.mkv"). Add an episode to the library using the season and episode number of the (parsed) filename. Use the filename as title of the episode. Like before, don't look up in a database if the season or episode exists. Complete missing/required data with dummy data.
Reply

Logout Mark Read Team Forum Stats Members Help
managing multiple scrapers0