Kodi Community Forum
[WIP] AniDB.net Anime Video Scraper - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Metadata scrapers (https://forum.kodi.tv/forumdisplay.php?fid=147)
+---- Thread: [WIP] AniDB.net Anime Video Scraper (/showthread.php?tid=64587)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37


- bambi73 - 2010-05-03

Looks like it's only some kind of addon/plugin registration info, i was bit afraid that there will be some changes inside scraper.
Thank you for this info/work, it could be useful for other "early-adopters", because when i checked your screenshots i found you are using Confluence and realized that my modified Alaska most probably will not work correcly, so another reason why not migrate right now Smile.
And of course any feedback is welcome.


- konti - 2010-05-14

For people that don't want to manually rename there anime collection, or don't know how to deal with it. I'll share my solution that works okay for me.

Using xbmc live (ubuntu)
I use deluge as my torrent client (transmission works also). I use the plugin/extra program flexget to auto download anime episodes for favorite shows. It reads a RSS feed every 15 minutes and if found it sends to torrent to the torrent client.

It also renames the files (this is the reason i'm posting this here) from e.a. [HorribleSubs] Naruto Shippuuden - 160 [720p].​mkv to Naruto Shippuuden - S01E160 - 720p.mkv. This should make it suitable for scraping as mentioned in this topic. Haven't tested this yet though.

Flexget is a little complicated at first to setup. But eventually you get trough it.
With my setup i have a basic config file with a couple of rules where i define what anime episodes i want. The part where i define wich anime shows i want looks something like this:
....
720p_anime:
- Naruto Shippuuden
- Fairy Tail
- Fullmetal Alchemist Brotherhood:
from_group: Eclipse
- Bleach:
from_group: DB

If someone want's the complete config just ask me.


- ZERO <ibis> - 2010-05-17

How do I get prequels to work? For example I have the prequel to Air as seen as T1 @ http://anidb.net/perl-bin/animedb.pl?show=anime&aid=2002

Is there any way to get it to show up in my list? Also if so can I get other things to work such as a promotion clip for example?

Now even though I have not changed anything I am losing data, for example I used to have Amatsuki show up as a folder with the info and everything. Then I edited the files for it just like everything else and it gets removed from the library. Now it will not load any info even though it shows all the right info for the animedb lookup. Any reason why it suddenly broke?

(It lists everything as Not available even though it has the right info and did so in the past)

As a test I did the following:
created a folder called GA
manually selected it and hit the tv show information button
it then gave me the anime db list
I select Galaxy Angel and I get the same problem
everything shows up as not available

*anime news network works fine...


- Zarbis - 2010-05-17

After a using this scraper for a while, i've noticed that scraper fails to get info about titles with "!" sign: Angel Beats! Baccano! Basquash! Durarara!! Kaichou wa Maid-sama! K-ON! Working !! All that titles are failed to be scraped... at least with google search enabled.


- ZERO <ibis> - 2010-05-17

But technically I accounted for that by setting GA and trying Galaxy Angel...

However if it is still working for everyone else it could just be something on my system. So this is good news b/c I plan to rebuild the library from scratch on a different system.


- bambi73 - 2010-05-17

Hello there,

ZERO <ibis> Wrote:How do I get prequels to work? For example I have the prequel to Air as seen as T1 @ http://anidb.net/perl-bin/animedb.pl?show=anime&aid=2002

Is there any way to get it to show up in my list? Also if so can I get other things to work such as a promotion clip for example?
I think you misunderstand what i mean by "prequel" lookup.
Prequel(s) titles are used for thetvdb.com anime lookup in case actual title is not found. It's because anidb.net and thetvdb.com use different approach for showing animes with multiple seasons, anidb.net uses new entry for each season (linked as prequel <-> sequel), on the other hand thetvdb.com stores all "seasons" under one id. So for example in case of Code Geass you have two anime entries on anidb.net (Code Geass: Lelouch of the Rebellion and Code Geass: Lelouch of the Rebellion R2) but only one entry on thetvdb.com (Code Geass: Lelouch of the Rebellion) with two seasons. When you try to scrape second season (Code Geass: Lelouch of the Rebellion R2) it will fail because such show isn't listed on thetvdb.com and it's place where prequel lookup is used. When scraper fails to find anime entry on thetvdb.com it check if prequel exist on anidb.net and then use its name (Code Geass: Lelouch of the Rebellion in our case) for another search ...
These T(trailer)/O(other)/C(opening/ending) has nothing to do with prequels/sequels. In case of Air T01 is some kind of prelude but it's more likely coincidence, because normaly real trailers are listed under T. Unfortunately it's kind of impossible to add these to library without some nasty tricks. But it's more problem of XBMC than scraper. XBMC recognize only specials (season 0) and regular episodes (season 1-n).

ZERO <ibis> Wrote:Now even though I have not changed anything I am losing data, for example I used to have Amatsuki show up as a folder with the info and everything. Then I edited the files for it just like everything else and it gets removed from the library. Now it will not load any info even though it shows all the right info for the animedb lookup. Any reason why it suddenly broke?

(It lists everything as Not available even though it has the right info and did so in the past)

As a test I did the following:
created a folder called GA
manually selected it and hit the tv show information button
it then gave me the anime db list
I select Galaxy Angel and I get the same problem
everything shows up as not available

*anime news network works fine...
From you description i'm not sure what's problem. Can you check it again with debug on and save log somewhere? Without debug log it's almost impossible to tell what's wrong.


- bambi73 - 2010-05-17

Zarbis Wrote:After a using this scraper for a while, i've noticed that scraper fails to get info about titles with "!" sign: Angel Beats! Baccano! Basquash! Durarara!! Kaichou wa Maid-sama! K-ON! Working !! All that titles are failed to be scraped... at least with google search enabled.
Hmm, another relict of poor testing of not used features (using anidb search). Replace
Code:
<RegExp input="$$1" output="+\1" dest="4">
  <expression clear="yes" repeat="yes">(?i)(?:%[a-f0-9]{2})*([a-z0-9]+)</expression>
</RegExp-->

in CreateSearchUrl function with

Code:
<RegExp input="$$1%20" output="+\1" dest="4">
  <expression clear="yes" repeat="yes">(?i)([a-z0-9]*?)(%[a-f0-9]{2})+</expression>
</RegExp>

It should work now with some special character at the end of directory name.
Anyway i checked mess what Google return for example for K-On! and really don't know what to do with it :confused2:


- ZERO &lt;ibis&gt; - 2010-05-18

It started working again. Now the only issue I have sometimes is that if there is something like 1024x576 in the file name it will not show up in the library unless I remove that from the name and refresh the data.


- bambi73 - 2010-05-19

Episodes scanning and listing has nothing to do with scraper, it's done by XBMC itself and depends on your episodes naming convention and your <tvshowmatching> setting in advancedsettings.xml (description on Wiki).

If you changed something in advancedsettings.xml you probably know what you did.
If you changed nothing then best will be to post some example what episode filename works for you and some what doesn't. I guess that your resolution is captured as part of season X episode pattern (e.g. as 1x01), but without example i can't help you much.


- ZERO &lt;ibis&gt; - 2010-05-19

I simply removed the res part of the file name and fixed it but for reference here is an example:

Works: Baka to Test to Shokanju s01e01 [D64D0665]
Fails: Baka to Test to Shokanju s01e01 [1280x720][D64D0665]

Also with the auto functions and with help from the Google look up I was able to get correct data for over 300 series. Great job on this scrapper!

I wanted to update this post with a suggestion:
When you run the scraper via the refresh option from inside the tv information view and you have google search option selected it will search only with that option. It would be best if it just used anidb and google so that you do not need to keep toggling it on and off when you can not find what you are looking for.


- salival - 2010-05-31

I have a small problem.

When I scan a folder for the first time, it finds all episode in a series.
When I scan the folder for new content it doesn't find the newly added files.
It gives me the following error in the log file:
Code:
21:12:56 T:3132 M:787607552  NOTICE: My Videos: Scanning for video info using worker thread, operation took 00:11
21:17:51 T:2532 M:782712832 WARNING: XFILE::CFileCurl::CReadState::FillBuffer: curl failed with code 3
21:17:51 T:2532 M:782712832   ERROR: CFileCurl::CReadState::Open, didn't get any data from stream.
21:17:51 T:2532 M:782712832   ERROR: CIMDB::InternalGetEpisodeList: Unable to retrieve web site

21:19:21 T:3136 M:805322752 WARNING: XFILE::CFileCurl::CReadState::FillBuffer: curl failed with code 3
21:19:21 T:3136 M:805322752   ERROR: CFileCurl::CReadState::Open, didn't get any data from stream.
21:19:21 T:3136 M:805322752   ERROR: CIMDB::InternalGetEpisodeList: Unable to retrieve web site
21:19:21 T:3136 M:804753408  NOTICE: My Videos: Scanning for video info using worker thread, operation took 00:00
(the first few lines are from when I call the episode info, the last few lines are when I scan the folder for new content.)

xbmc log (not entirely the same log as above, but the same error)

Besides this it's a great scraper.

on another note:
It does find more anime than the ann scraper, but the ann scraper does seem to find the cast info more easily. But that's aa minor gripe I can live without. Big Grin


- bambi73 - 2010-06-06

I'm aware of this bug, or at least i think it's same problem what i solved month ago. Unfortunately even if i release it right now you will need to go thru your whole library and rescan all your anime because there is wrong/partial episode list link in your database (if you still interested PM me).
Right now i'm waiting for some 10.05 beta/rc to release AniDB scraper under new addon structure.


- gates - 2010-06-06

Hi,

First, thx Bambi for your great job.

I have the same bug ("scan for new content" doesnt get the new files)
What can I do to fix it ?
maybe, the simplest way is to wait your next scrapper version (with 10.05).

PS : apologise my poor english
/a french guy Smile


- ZERO &lt;ibis&gt; - 2010-06-07

Also it would be cool if there was a way for it to detect an nfo file with the page id for any particular anime we are scraping as to save time with look ups or if the database is recreated.


- chojin - 2010-06-18

Just bumped into this scraper. Finaly an AniDB scraper, and it seems it is even a very good one! Besides the above mentioned bug, it works great!

As I am a long time anidb user myself it would be a great feature if the scraper could sync the watched status of my anime in the library and the 'seen' state in my personal mylist. But I assume a scraper is not really fit to make changes on a web-service?

Anyway, keep up the good work!