Kodi Community Forum
Release [MOD] AniDB.net scrapers for TV shows and Movies - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: TV Show Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=305)
+----- Thread: Release [MOD] AniDB.net scrapers for TV shows and Movies (/showthread.php?tid=142835)

Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - scudlee - 2013-11-07

"E01", etc. is not one of the default patterns, those files won't be enumerated without a custom tvshowmatching regexp. If you're willing to rename, you could use "Ep01" or S01E01" - both are default patterns and will get found without using custom regexps.

Similarly with the Dragon Ball file name, if you haven't got an advancedsettings.xml (wiki) with custom regexps, it won't work.

This is an advancedsettings.xml for the file names you're using:
Code:
<advancedsettings>
  <tvshowmatching action="prepend">
    <regexp> - ()(\d+)((?:-\d+)*)(?:v\d+)? - [^\\/]*$</regexp>
    <regexp defaultseason="0"> - ()s(\d+)((?:-\d+)*)(?:v\d+)? - [^\\/]*$</regexp>
    <regexp> ()e(\d+)((?:-\d+)*)(?:v\d+)? '[^\\/]*$</regexp>
  </tvshowmatching>
</advancedsettings>

The first two regexps are from the first post (and match the Dragon Ball format), the third matches against " E01 '" - that's space-E<EpNo>-space-apostrophe (the Outlaw Star format)

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - LoveJoy - 2013-11-07

@outlaw Mantis

See if this http://forum.xbmc.org/showthread.php?tid=64587&pid=1522837#pid1522837
or this http://forum.xbmc.org/showthread.php?tid=142835&pid=1522912#pid1522912
can help you any way.

Else I know that my XBMC can not find files with E# for episode number. I have my anidb client to name my files ep##. With that and the use of the tvshov.nfo file in all my anime series folders, even the original anidb scrapper found everything without the use of advancesettings.xml.

I did however one time experience that my XBMC stopped finding anything new on my NAS, but still was able to work with all I had scrapped already. After looking in a lot of wrong places I found that XBMC had lost its sources, and it worked with old files because of there path saved in the database.

I got an anime collection long time before I found XBMC. So when I started with XBMC I had a lot of cleaning up to do in my files. It took a long time and a lot of work to get it all up and running, but it was a time worth spending.
Most of my collection I left to anidb to sort and rename, as I also found anidb.net at the same time as XBMC. The lot that anidb client could not recognize I had to rename manually. There are renaming software out there, but in the end I found my logitech G15 keyboard the most useful. As I just record a macro of me manually renaming the first file, and then just used the macro key for the rest. (remember to record Return and Arrow down as the end of the macro)
Now all my anime is in a named folder and the files have either ep# or S##E## as mask. The scrapper is set to store anime ID from TheTVDB.com, as I find it helps a lot for getting fanart. And the tedious work of making a tvshow.nfo file for every folder is over.

But be warned! Anime is addictive Smile

P.S. And yes, I just realized that I again overlooked the last page, and that Scudlee have already answered. But I am going to press Post Reply button anyway, as it was all written in good intention Smile

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Outlaw Mantis - 2013-11-07

Thanks for your replies, but over the past few hours I tried using something that I never noticed in the thread before: one of scudlee's recommended programs, FileBot. At first, it appeared to be a lifesaver. On occasions, it didn't work, but when using it with Advanced Renamer, I was able to rename episodes en masse and then have them renamed to a proper name that FileBot and AniDB would recognise. So, I renamed the episodes of 53 anime series to the near-exact AniDB names (obviously having to remove question marks and the like), and a few movies too. I waited around an hour for the complete rebuilding of my library to complete. Then...

Image

Seriously, it had only recognised the episodes of about three series. Fifty unrecognised! This is driving me insane. I know you've posted some regex fixes in the last couple of posts and I will use them, but are they really necessary when my episodes are renamed to almost the exact names on AniDB? It scrapes the actual series okay, but not the episodes!

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - scudlee - 2013-11-08

Just to clarify, I never particularly recommended Filebot, I only stated an awareness that it uses (or can use) anidb in some fashion.
I seriously don't know enough about any file renaming tool (other than the anidb client) to be able to recommend one, or to offer support for it.

That aside, the most important point to make is that your files have to have the season and episode numbers be recognized by XBMC in order for them to get added.
If you don't want to have an advancedsettings.xml that means having one of S01E01, 1x01, or ep01 somewhere in your file name (there are a few others, but those are the most sensible options).

If your file renaming tool failed to put one of those in your file name, there's a problem somewhere.

Using any other format will require an advancedsettings.xml with custom tvshowmatching regexps.


If you can use the anidb client, but don't want to use an advancedsettings.xml, you can adapt the rules I linked to in my first post by changing the lines:
Code:
Regular:=""
Special:="S"
Separator:=" - "
to:
Code:
Regular:="S01E"
Special:="S00E"
Separator:=" "
or:
Code:
Regular:="1x"
Special:="0x"
Separator:=" "
(If you want to use the ep01 format, change the Regular to "ep", and change the Special to either "S00E" or "0x".)

That will (as you can hopefully deduce) replace the default " - 01 - " format with one of the XBMC-compatible " S01E01 " or " 1x01 " formats.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Kuenaimaku - 2013-11-08

Hello there, newbie XBMC user here. I'm having trouble actually having anime episodes appear under the TV show. I have followed the OP's direction to the letter, and have converted my library using the tagging system. I have scanned my library using default settings, along with trying using the official english names. At first glance, it works because the series are shown as TV shows, but the episodes are missing.

That is, except for one series. One series out of my library shows up with episodes, the others do not. Here's a screenshot of what I mean:

watch gallery


I've been looking into this problem and I'm at my wit's end. I'd love to be pointed in the right direction!



Some relevant information on my setup is below:


This folder is the only one that is recognized to have episodes.
Code:
Folder: J:\Anime\TV Shows\Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_

Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_ - 01 - I Couldn't Be a Hero, so I'm Working the Register [HorribleSubs][www][
Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_ - 02 - The Demon Lord's Daughter Is Going to Learn How to Speak to Cust… [Hor
Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_ - 03 - We Should Be Careful When Handling Weird Products Sent to Us by … [Hor
Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_ - 04 - The Daughter of the Demon Lord Will Be Working at the Convenienc… [Hor
Yuusha ni Narenakatta Ore wa Shibushibu Shuushoku o Ketsui Shimashita_ - 05 - I Couldn't Be a Hero so I Snuck into a Rival Store with the Daug… [Hor

filetype: .mkv

Here's a folder that has a series with the same formatting, yet no episodes.

Code:
Folder: J:\Anime\TV Shows\Nagi no Asukara

Nagi no Asukara - 01v3 - In Between the Sea and the Land [Underwater-Vivid][HDTV][1280x720][h264][4546E4F2]
Nagi no Asukara - 02 - The Chilly Desert [Underwater-Vivid][HDTV][1280x720][h264][475FE354]
Nagi no Asukara - 03 - The Legend of the Sea [Underwater-Vivid][HDTV][1280x720][h264][FB964F8F]
Nagi no Asukara - 04 - Because We're Friends [Underwater-Vivid][HDTV][1280x720][h264][29E13057]
Nagi no Asukara - 05 - Hey, Sea Slug [Underwater-Vivid][HDTV][1280x720][h264][9779256F]

filetype: .mkv

My advancedsettings.xml:

Code:
<advancedsettings>
  <tvshowmatching action="prepend">
    <regexp> - ()(\d+)((?:-\d+)*)(?:v\d+)? - [^\\/]*$</regexp>
    <regexp defaultseason="0"> - ()s(\d+)((?:-\d+)*)(?:v\d+)? - [^\\/]*$</regexp>
  </tvshowmatching>
</advancedsettings>


RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Outlaw Mantis - 2013-11-09

(2013-11-08, 20:29)scudlee Wrote: Just to clarify, I never particularly recommended Filebot, I only stated an awareness that it uses (or can use) anidb in some fashion.
I seriously don't know enough about any file renaming tool (other than the anidb client) to be able to recommend one, or to offer support for it.

That aside, the most important point to make is that your files have to have the season and episode numbers be recognized by XBMC in order for them to get added.
If you don't want to have an advancedsettings.xml that means having one of S01E01, 1x01, or ep01 somewhere in your file name (there are a few others, but those are the most sensible options).

If your file renaming tool failed to put one of those in your file name, there's a problem somewhere.

Using any other format will require an advancedsettings.xml with custom tvshowmatching regexps.


If you can use the anidb client, but don't want to use an advancedsettings.xml, you can adapt the rules I linked to in my first post by changing the lines:
Code:
Regular:=""
Special:="S"
Separator:=" - "
to:
Code:
Regular:="S01E"
Special:="S00E"
Separator:=" "
or:
Code:
Regular:="1x"
Special:="0x"
Separator:=" "
(If you want to use the ep01 format, change the Regular to "ep", and change the Special to either "S00E" or "0x".)

That will (as you can hopefully deduce) replace the default " - 01 - " format with one of the XBMC-compatible " S01E01 " or " 1x01 " formats.

Well, for a few examples of how FileBot has renamed my series, visit this Pastebin link. So, I gather from what you're saying, FileBot does actually do its job correctly, but the problem lies in the fact that XBMC does not play nice this simply because of the way XBMC has been coded. I'm a bit confused though. I am using an advancedsettings.xml with some of the recommended code from others. I have some TV shows with the craziest file names that I haven't had to go through any of these processes with.

Sorry for the dumb question, but what action should I take from here? I know you said in your past post, but that was on the basis I wasn't using advancedsettings.xml. I notice what you've said about the "anidb client", but when I searched for it, I was told that many clients exist. I guess I should keep searching.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - scudlee - 2013-11-09

@Kuenaimaku: I'll need to see a debug log (wiki), I think.

If it had been zero shows with episodes, I would have said you'd put your advancedsettings.xml in the wrong place (e.g. you'd placed it in the program folder, not your userdata folder (wiki)), or named it wrong (e.g. it has a hidden .txt extension).

But the fact that one show worked, though, confuses me...

So,
1. Turn on debug logging. (Important!)
2. Refresh a series that has no episodes (press 'i', select refresh).
3. Also refresh the series that worked (just to see why).
4. Upload the log to pastebin.com or xbmclogs.com and post the link.


@outlaw Mantis: By anidb client, I'm specifically referring to the "official" client (which I do link to in the first post). Many other clients do exist, but that one still remains for me the most configurable.

If you use the client, with the rules provided in the first post (but inserting your own file paths, naturally) and the advancedsettings.xml (or make the alterations to the rules as described in my previous post), you should be getting ~99%* success rate. Only a handful of shows need some form of manual intervention. This is speaking from personal experience.

This is the method I recommend, precisely because it is the one I use personally. I know it works.


*Totally made up figure

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Kuenaimaku - 2013-11-09

(2013-11-09, 01:14)scudlee Wrote: @Kuenaimaku: I'll need to see a debug log (wiki), I think.

If it had been zero shows with episodes, I would have said you'd put your advancedsettings.xml in the wrong place (e.g. you'd placed it in the program folder, not your userdata folder (wiki)), or named it wrong (e.g. it has a hidden .txt extension).

But the fact that one show worked, though, confuses me...

So,
1. Turn on debug logging. (Important!)
2. Refresh a series that has no episodes (press 'i', select refresh).
3. Also refresh the series that worked (just to see why).
4. Upload the log to pastebin.com or xbmclogs.com and post the link.

I had no idea that you could refresh a series using the I key! that was useful.

I tried a "hail mary" of sorts and re-installed the repo, and this time it actually worked, with a couple of them missing episodes. refreshing the series fixed that in the end though!

Now I just have to wait for a couple, some of them have the desc. "You have been banned from anidb.net for some time".

I really appreciate your help though Scudlee.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - DiMag - 2013-11-09

@scudlee
Let me express thanks and appreciation for his work. The treasure trove that is AniDB.net was foreclosed to me until I started using your scraper.

I herewith ask for help with an issue, and post two feature requests:

ISSUE: One-pass refresh of tvshow(URL).nfo
Just to be sure (and speed up scraping) I put a tvshow.nfo with the aid from anidb at the root of each anime I want to have scraped. The problem with this method is that when you ask XBMC to export its library to nfo files the content of the tvshow.nfo doesn't get refreshed unless you answer the question "Overwrite old nfos?" with "Yes". But you don't want to overwrite all old nfos, just this one in this particular folder.
The only method to achieve this I came up with is to do the initial scraping, then export the database, then rename the tvshow.nfo to anything other than tvshow.nfo, and then (and finally) do a second library export. This time XBMC exports its library content to tvshow.nfo without asking "Overwrite old nfos?" - there is none to overwrite.
Do you have any suggestion how this could be done in one pass?

FEATURE REQUEST, 1: Anime Characters
AniDB.net maintains for each (and all) anime a database of characters, but XBMC knows nothing about characters instead of whom it indexes actors. Evidently this makes little sense in the context of anime - here actors are nothing and characters are everything. It would be lovele to have for each anime a characters database.
Can this extra feature be added to the scraper?
If not, is there a script-based workaround (say, something like the VideoExtras script)?

FEATURE REQUEST, 2: Anime Relationships
AniDB.net maintains for each (and all) anime a database of anime-to-anime relationships in the sense of parent-child (sequel/prequel/alternative version) and similarity of content. It would be very useful to have this sort of relationships displayed in XBMC.
Can this feature be implemented?
If not, is there a workaround? (I do not mean including a note in plot.)



RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - LoveJoy - 2013-11-09

@Kuenaimaku

If it is the AniDB.net[MOD] you are using, then try and set the Delay Parameter to 125.
I have been banned with it at 100, but at 125 I have not noticed any ban so far.

It is under the scrappers settings, General tap.

@DiMag.

Issue:
Would it not be easier to just
-do the initial scraping,
-then rename the tvshow.nfo to anything other than tvshow.nfo,
-then export the database,
all done

Feature Request 1:
Totally disagree on -"here actors are nothing and characters are everything."
That might be because that I am a fan of a few voice actors and like to find there other works trough XBMC
I use AniDB or the manga for Character backgrounds.

Feature Request 2:
Totally agree -"It would be very useful to have this sort of relationships displayed in XBMC. "
I think our best chance would be if there could be something like the SET feature the movie scrappers have.But I have no knowledge if that exist in a tvshow scrapper.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - bass.exe21 - 2013-11-09

Don't know where else to post this, but the thetvdbid for "Choujigen Game Neptune The Animation" is wrong in the anime-list.xml. It's currently 270067 when it should be 267705.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - scudlee - 2013-11-09

(2013-11-09, 21:51)bass.exe21 Wrote: Don't know where else to post this, but the thetvdbid for "Choujigen Game Neptune The Animation" is wrong in the anime-list.xml. It's currently 270067 when it should be 267705.

Fixed. Thanks!

(I'm going to guess it was a duplicate that got deleted.)

You may want to delete your cache to see the change immediately, otherwise it could take up to 48hrs.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Outlaw Mantis - 2013-11-10

(2013-11-09, 01:14)scudlee Wrote: @outlaw Mantis: By anidb client, I'm specifically referring to the "official" client (which I do link to in the first post). Many other clients do exist, but that one still remains for me the most configurable.

If you use the client, with the rules provided in the first post (but inserting your own file paths, naturally) and the advancedsettings.xml (or make the alterations to the rules as described in my previous post), you should be getting ~99%* success rate. Only a handful of shows need some form of manual intervention. This is speaking from personal experience.

This is the method I recommend, precisely because it is the one I use personally. I know it works.


*Totally made up figure

It's processing all files and folders. Unfortunately I left it processing for a couple of hours. I guess I'll need to do that again, because it has renamed and moved everything to have Japanese names. Is there any way to make the client rename to English names instead, or is this really the only option for this to work? It's good for XBMC if it works, because your scraper will retrieve the English names, but if by chance I want to watch a couple of episodes on my computer, I won't have a clue where they are. Tongue

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - Outlaw Mantis - 2013-11-10

I've got an update for you. Check this pastebin link. A few things aren't being renamed, despite the fact I've had all of them renamed to (apparently) AniDB's standards using FileBot beforehand. First of all is Cat Planet Cuties. (Please do not ask why I have this.) FileBot renamed it fine. Why is the official client flat out refusing to rename it? It confuses me. Then there's Fullmetal Alchemist Brotherhood. For the most part (despite the Japanese names which irks me, which is Hagane no Renkinjutsushi), they've renamed fine, except episodes 47-51. There's no logical reason for these to not be renamed. They all had the same episode names as in AniDB (exactly, because of FileBot) and there's no deviation from the other episodes. Finally, there's Mushishi, which has a similar problem to FMA.

Edit: There's another one, Mezzo DSA episode four: Shell of Lies (Episode Posi). No logical reason why it won't rename and move.

RE: [RELEASE] [MOD] AniDB.net scrapers for TV shows and Movies - scudlee - 2013-11-10

If you want English titles, change the line:
Code:
ShowTitle:=[%ATr%, %ATe%, %ATk%]
to:
Code:
ShowTitle:=[%ATe%, %ATr%, %ATk%]

(e = English, r = romaji, k = kanji. The order determines the priority.)


The files are recognized based on their file hashes (specifically the ed2k hash, I believe). Unlike other file renaming tools, it doesn't rely on parsing the file name to determine the show and episode number, it just matches the hash against the one stored in the database.
That makes it a lot more reliable for renaming... Provided, that is, the file is in the database...

If the files are corrupt, or are "unofficial" remuxes, or just plain aren't fansubs (i.e. they're personal rips) they won't be in the database, so they won't get renamed.

Considering some shows, like Mushi-shi and FMA had most of their episodes recognized, I'd be inclined to lean towards the "corrupt file" option in your case, assuming all the episodes were from the same fansub source. You might want to look into the CRC values of the unrenamed files and compare them to an "official" source (such as their original file name).


Another possibility is that the client found the file but was for some reason unable to rename or move it. You can tell by the [P][F][M][R] in the client window: If the [F] is green, the file was found in the database, if the [M] and [R] are red it was unable to move and rename. The most likely cause in that case is that some other process is locking the file (e.g. BitTorrent).