[RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies

  Thread Rating:
  • 6 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
scudlee Offline
Team-Kodi Member
Posts: 839
Joined: Jul 2011
Reputation: 51
Post: #46
I imagine most of the shows and episodes themselves are on AniDB, it's just the specific files which aren't.

If that's the case, then basically no automatic re-namer will work (if it relies on an AniDB file look-up - which I'm guessing filebot does). You have to go in there and manually fix the names, there's no other way around that.

@hentai23 Check the first post for a fool-proof renaming scheme using the AniDB client, that will automatically deal with OPs, EDs, movies, OVAs , etc. so they will be picked up by XBMC.

But still, if the files aren't on AniDB, manual renaming is your only option.
find quote
hentai23 Offline
Fan
Posts: 688
Joined: Sep 2010
Reputation: 0
Location: another universe ..it would seem xD
Post: #47
thanks shall check it out
find quote
ZERO <ibis> Offline
Senior Member
Posts: 268
Joined: May 2010
Reputation: 1
Post: #48
Had been using the WIP] AniDB.net for a long time now. I have switched over to this and currently am having it redo my library on auto update from scratch. This is well over 500 series so I will be posting back in a bit to report any problems I ran across, so far things look good!
So it managed to auto scrap 255, so that leaves about 300+ to manually add Sad

Edit: looks like a lot of folders did not get the set content from the parent folder and that is likely the cause of the problem. Trying again lol
(This post was last modified: 2013-01-26 10:41 by ZERO <ibis>.)
find quote
ZERO <ibis> Offline
Senior Member
Posts: 268
Joined: May 2010
Reputation: 1
Post: #49
100+ left sleep for now
find quote
ZERO <ibis> Offline
Senior Member
Posts: 268
Joined: May 2010
Reputation: 1
Post: #50
Feature suggestion.

To aid in preforming a full db update from scratch without needing any additional human input I propose the detection of a .txt file or something that can hard link a folder to a particular series.

For example a series like K instead of just relying on the title of the folder the scrapper can look for a file called anidb.txt and read the first line for a9043

The scrapper would then know that the data in this folder is for a9043 making scrapping 100% successful at finding the correct series.

Is there any way to implement such a feature where as we would have the "adi" declared from our file system directly as to ensure that the scrapper always gets it right 100% of the time?
find quote
scudlee Offline
Team-Kodi Member
Posts: 839
Joined: Jul 2011
Reputation: 51
Post: #51
This is already possible. Just create a file called tvshow.nfo (exactly that, don't replace "tvshow" with the title of the show) and place in it "aid=9043". Technically, you should include the entire URL to the page on AniDB.net, but only the "aid=..." part is searched for. That will short-cut the title search and jump straight to getting the details for that show, regardless of the folder name.

Check the wiki for details.
find quote
ZERO <ibis> Offline
Senior Member
Posts: 268
Joined: May 2010
Reputation: 1
Post: #52
Cool, I will try that method in the future
find quote
Tastyfreeze Offline
Junior Member
Posts: 1
Joined: Jan 2013
Reputation: 0
Post: #53
Actually that reminds me of something along those lines that could be an interesting feature to add. AniDB gives you the ability to export your personal list in rather exhaustive detail. Furthermore it's customizable and that customization can be added to site for use by everyone. The idea here is to let the scraper use that first before going to anidb. It would speed things up slightly and ease the load off the site.
find quote
Mihara Offline
Junior Member
Posts: 34
Joined: Mar 2012
Reputation: 0
Post: #54
Apparently, the current version of the scraper can cause you to get banned from AniDB. I was refreshing information for all episodes based on series IDs in NFO files, (i.e. skipping title search, but re-requesting episode information) for multiple series in quick succession, (I have a lot of them and many of them needed updating artwork as I switched to Frodo RC3, which was less troublesome to accomplish by refreshing the entire series) and eventually bumped into a ban. While I intend to be more careful in the future, it seems to me the scraper should be careful to avoid this sort of thing programmatically. Smile

Can anyone tell me how long does it typically take to get unbanned and is there a way to find out if you are without triggering an extension of the ban, by the way?
find quote
twig Offline
Junior Member
Posts: 15
Joined: Oct 2012
Reputation: 0
Post: #55
Dude, I'd like to say a big thank you for getting all of my anime working, properly!

The original anidb.net scanner didn't do a very good job at finding the OVA/specials, along with the insane naming convention of Gintama` Enchousen.
I even took a stab at entering some data in myself but to no luck.

Then I installed AniDB.net[mod] and it worked great!
Thanks!

Could you please explain a little bit more regarding the post below? The wiki page doesn't provide much information for a person who hasn't looked at the XBMC code.

Does that mean:
  1. Create a text file in the "K" folder called "tvshow.nfo"
  2. Paste down "aid=9043" and save
  3. Or does it mean I need the full URL to the AniDB page such as: http://anidb.net/perl-bin/animedb.pl?sho...e&aid=9043 ?


(2013-01-27 00:09)scudlee Wrote:  This is already possible. Just create a file called tvshow.nfo (exactly that, don't replace "tvshow" with the title of the show) and place in it "aid=9043". Technically, you should include the entire URL to the page on AniDB.net, but only the "aid=..." part is searched for. That will short-cut the title search and jump straight to getting the details for that show, regardless of the folder name.

Check the wiki for details.
find quote
Mihara Offline
Junior Member
Posts: 34
Joined: Mar 2012
Reputation: 0
Post: #56
(2013-01-28 15:08)twig Wrote:  Does that mean:
  1. Create a text file in the "K" folder called "tvshow.nfo"
  2. Paste down "aid=9043" and save
  3. Or does it mean I need the full URL to the AniDB page such as: http://anidb.net/perl-bin/animedb.pl?sho...e&aid=9043 ?

Both work. I typically paste the full URL as it's quicker to do when you have a lot of series.
find quote
scudlee Offline
Team-Kodi Member
Posts: 839
Joined: Jul 2011
Reputation: 51
Post: #57
(2013-01-28 15:11)Mihara Wrote:  Both work. I typically paste the full URL as it's quicker to do when you have a lot of series.

Yeah, you're supposed to put the full URL, but the scraper only bothers to check for the "aid=..." part at the end of it, so in practical terms, you can put whatever you like, as long as you include "aid=..." somewhere.


In terms of the banning situation, I'll look at porting over the delay function from the movie scraper and have it run after the episode list has been fetched. I'll have it default to something low, so only those affected need crank it up. That's probably the best I can do.

Bans typically only last at most 24 hours (and it's usually less), but because the scraper caches everything for 48 hours, it may appear longer. If you know you're unbanned, you can delete your cache and continue.

(On Windows the cache is in %APPDATA%\xbmc\cache\scrapers\metadata.tvshows.anidb.net.mod (or metadata.movies.anidb.net.mod). Not too sure for other OSes, but it shouldn't be too hard to find.)
find quote
Mihara Offline
Junior Member
Posts: 34
Joined: Mar 2012
Reputation: 0
Post: #58
(2013-01-28 15:47)scudlee Wrote:  Yeah, you're supposed to put the full URL, but the scraper only bothers to check for the "aid=..." part at the end of it, so in practical terms, you can put whatever you like, as long as you include "aid=..." somewhere.

IIRC the code, it is now supposed to actually use the full URL to determine which scraper to use for this particular media entry, (it didn't before even though the wiki said otherwise), so yeah, supposed to be full URL. Scraper selection happens earlier than a scraper gets a chance to see the "aid=".

(2013-01-28 15:47)scudlee Wrote:  Bans typically only last at most 24 hours (and it's usually less), but because the scraper caches everything for 48 hours, it may appear longer. If you know you're unbanned, you can delete your cache and continue.

How do I know if I am without checking though? Word on the net is, checking increases your ban time. Smile
find quote
scudlee Offline
Team-Kodi Member
Posts: 839
Joined: Jul 2011
Reputation: 51
Post: #59
As I understand it, XBMC will use the scraper set for that folder to check the tvshow.nfo for a URL first, and only if that doesn't find anything does it check using the other installed scrapers (and if one of them gets a hit, it will use that scraper).
The actual URL you need to use is entirely dependent on the regular expression(s) in the NfoUrl function inside the scraper (and for anidb.net that regex is just aid=(\d+) ).

In terms of knowing when you're unbanned...

¯\(°_o)/¯

I got banned from the website not long ago, and basically just sat on my hands for 24 hours. I could still access the api, though, so they must have separate banning systems.

The only tactic would be to just dip your toe when you think you might be unbanned, and if you are still banned, sit tight for the full 24 (starting from when you dipped your toe).
It's pretty bad that they don't at least have some form of "Am I banned?" page you can check without increasing your ban (unless you hammer it).
find quote
Mihara Offline
Junior Member
Posts: 34
Joined: Mar 2012
Reputation: 0
Post: #60
To be more precise, the current Frodo code should use the set scraper if there is no NFO or NFO matches no URL, otherwise the scraper that matches the URL. See https://github.com/xbmc/xbmc/pull/1121 for details -- got merged in quite a while ago.

I found out specifically because I tried to put all my movies into one directory and give a different scraper for some of them, which wasn't working. Smile
(This post was last modified: 2013-01-28 22:08 by Mihara.)
find quote
Post Reply