• 1
  • 62
  • 63
  • 64(current)
  • 65
  • 66
  • 71
Release [MOD] AniDB.net scrapers for TV shows and Movies
(2020-01-07, 21:42)Uatschitchun Wrote: Hi, tried that scraper for the first time, works good, but I do get lots of these in logs:
Code:
2020-01-07 20:22:57.770 T:139952436414208   ERROR: Parse: Could not find scraper function GetArt
2020-01-07 20:22:57.770 T:139952436414208   ERROR: Run: Unable to parse web site
It seems there's something wrong with the URL in the nfos?
Same problem (Kodi 16.1 – Jarvis):

16:56:11 T:3172   ERROR: CScraperParser:Tonguearse: Could not find scraper function GetArt
16:56:11 T:3172   ERROR: ADDON::CScraper::Run: Unable to parse web site
Reply
Is there a way to disable scraping ratings? I don't want episode ratings and stuff.
Reply
(2020-01-21, 23:15)fighuass Wrote: Is there a way to disable scraping ratings? I don't want episode ratings and stuff.

Only way is to get to the scapper xml and remove it - metadata.tvshows.anidb.net.mod\anidb.xml
Then remove the call to "GetRatingByIDChain". This prevents the ratings from being emitted to Kodi.
Reply
(2020-01-22, 00:39)Sesshoumaru Wrote:
(2020-01-21, 23:15)fighuass Wrote: Is there a way to disable scraping ratings? I don't want episode ratings and stuff.

Only way is to get to the scapper xml and remove it - metadata.tvshows.anidb.net.mod\anidb.xml
Then remove the call to "GetRatingByIDChain". This prevents the ratings from being emitted to Kodi.  
Hm, didn't do anything for me, rescaped everything and episode ratings are still there.

All fanart/background images in my TV show library are also missing, it doesn't grab anything except posters...

Honestly, is this scraper even worth sticking with? It seems broken as hell.
Reply
(2020-01-22, 12:34)fighuass Wrote:
(2020-01-22, 00:39)Sesshoumaru Wrote:
(2020-01-21, 23:15)fighuass Wrote: Is there a way to disable scraping ratings? I don't want episode ratings and stuff.

Only way is to get to the scapper xml and remove it - metadata.tvshows.anidb.net.mod\anidb.xml
Then remove the call to "GetRatingByIDChain". This prevents the ratings from being emitted to Kodi.   
Hm, didn't do anything for me, rescaped everything and episode ratings are still there.

All fanart/background images in my TV show library are also missing, it doesn't grab anything except posters...

Honestly, is this scraper even worth sticking with? It seems broken as hell. 
I don't know another one which access the information from AniDB and combine it with nice images.
But its true, I repaired it a couple times for myself. You also never should forget cleaning the cache, otherwise you get the old stuff back. And the recent changes on TVDB at least have broken the zip download (see above)... so you need more than just script adaptions. I think there is a new phyon based scapper in the make for TVDB... but nothing for anime fans and AniDB users. Sadly.
Reply
Is there a fix for getting episodes thumbnails and info from thetvdb?
Reply
Star 
(2020-01-23, 05:12)stylus7 Wrote: Is there a fix for getting episodes thumbnails and info from thetvdb?
You need this: Zip-Fix
And this: #945

Then it works again. At least that is my personal experience...
Reply
(2020-01-22, 22:13)Sesshoumaru Wrote:
(2020-01-22, 12:34)fighuass Wrote:
(2020-01-22, 00:39)Sesshoumaru Wrote: Only way is to get to the scapper xml and remove it - metadata.tvshows.anidb.net.mod\anidb.xml
Then remove the call to "GetRatingByIDChain". This prevents the ratings from being emitted to Kodi.   
Hm, didn't do anything for me, rescaped everything and episode ratings are still there.

All fanart/background images in my TV show library are also missing, it doesn't grab anything except posters...

Honestly, is this scraper even worth sticking with? It seems broken as hell.   
I don't know another one which access the information from AniDB and combine it with nice images.
But its true, I repaired it a couple times for myself. You also never should forget cleaning the cache, otherwise you get the old stuff back. And the recent changes on TVDB at least have broken the zip download (see above)... so you need more than just script adaptions. I think there is a new phyon based scapper in the make for TVDB... but nothing for anime fans and AniDB users. Sadly.  
Tried deleting the cache and rescraping everything, but the cursed numbers are still there..

Also, is it me or do the series descriptions kinda suck? Almost every anime I scrape has a description like "new anime based on an ... manga by ... and broadcasted in early 2012.", with literally nothing actually talking about the anime itself.

For example, with Dr. Stone I get "Based on an adventure sci-fi shounen manga series written by Inagaki Riichirou and illustrated by Boichi.". Whereas the full summary is "* Based on an adventure sci-fi shounen manga series written by Inagaki Riichirou and illustrated by Boichi.

One fateful day, all of humanity was petrified by a blinding flash of light. After several millennia, high schooler Taiju awakens and finds himself lost in a world of statues. However, he’s not alone! His science-loving friend Senkuu’s been up and running for a few months and he's got a grand plan in mind—to kickstart civilization with the power of science!".

I suppose I can deal with the lack of fanart though, guess I'll just have to manually download it and add it to Kodi.
Reply
Depends, when the entry is new the description is pretty much like that. If you scape it again after the anime has aired, then you most certainly get a more comprehensive description.
It probably would be best to have not a scapper which used multiple sources, like AniDB for Info (Descrption, Seiyuu, etc) and embedded TVDB for fanart. Something like a high level information aggregator, using multiple scappers to get information about a single entry, would be better (one has just to map the Ids).
Reply
So...  Having woken somewhat from my slumber, and seeing a lot of complaints in this thread, I decided to have a look at trying to fix the scraper.

I originally quietly abandoned the scraper because the v1 tvdb api was supposedly going to be discontinued, and I didn't have a good way to integrate the v2 api without completely rewriting everything, so I was expecting the scraper to just kind of die a death.  Instead, of course, the v1 api never stopped, and the scraper kind of muddled on, but by then my interest (in a lot of things) had passed.  A while ago though, I did do a complete rewrite of the tvdb scraper (as it turns out, purely for my own entertainment, not public consumption) which made the whole thing a lot more modular, and it turned out it was surprisingly easy to bodge it into the anidb scraper and get working.

At the moment it's not in a fit state to push to the repo - really, the code is a mess, tabs and spaces colliding, functions just copy-pasted willy-nilly, little bits of debugging here and there, etc. (plus I really need a minute to remember what I need to do to publish it.)
It also needs testing, anyway, so rather than wait until I get it looking all pretty, I figured I'd just upload the zip files straight away, and let you have at it.

[metadata.common.anidb.net] v1.2.1 (temp version number for testing)
[metadata.tvshows.anidb.net.mod] v2.4.1 (temp version number for testing)
No update for the movie scraper.  All it's important stuff is in the common library anyway.

(Install the common library first, as the tv scraper now depends on the updated version.)

Fixes:
  • TVDB/TMDB API keys updated to current ones.  I'm assuming this might have been an issue.
  • TVDB artwork restored - posters, banners, fanart (using V2 api - so no zip errors)
  • TVDB episode details restored - episode plots, thumbnails, uniqueids
  • Some tweaks to the plot filtering to fix the somewhat overzealous editing <-- CHECK THIS PLEASE
  • Decoupled the language selection from the Official/main name selection.  You can select "main" name and still have a different language for episode plots and titles (where available - English is the automatic fallback).  The series/movie plot is still from the anidb description, so only in English, that's just how that goes.
  • GetArt function added - I see some people pointing out these "errors" in the log.  It's not really an error, the GetArt function is a way for Kodi to update the artwork automatically, independent of a scan, it's primary purpose was so the user didn't have to refresh a series every time a new season was added in order to get the new season posters.  
    Since anidb shows are all one season, it didn't seem that necessary, plus the function would need the anidb id number to work, but at the time it was possible to switch the scraper to use the tvdb id (for compatibility with extrafanart addons and such), but there is no way for the function to reasonably distinguish which id it is given, since either way it's just some random number.  The scraper now ignores the tvdbid setting, and uses named uniqueids for both anidb and tvdb, and always defaults to anidb (the more up-to-date addons should look for the tvdb uniqueid now, so it doesn't matter if it's not the default), but if you were using the tvdbid before, some oddities may arise... possibly?  Refreshing should put the right uniqueids in, and fix anything.
  • There are other fixes/tweaks I idly made for myself in the past, I think, mainly in the genres and tagging areas.  General tagging is still off the table, but location, source, and audience tagging should work again (if they didn't(?)).
    Also if you see anything appear as a genre that definitely shouldn't be one, report it and it can be filtered.
  • By necessity for the tvdb api, the cachepersistence has been lowered from 48 hours to 24.  This shouldn't cause issues with anidb, 48 hours was always a little overkill, but... be aware.

My plan is to fix everything up the week after next, when I have some time, and push it around the 15th(ish).
You have until then to report any errors in the zip files or bugfixes I've missed, and to request any reasonable/achievable new features to consider.
A "reasonable/achievable" new feature would be e.g. adding a "disable ratings" setting  (consider that on the todo list).  Any tvdb-related features might also be doable with the new setup.
Unreasonable new features would be anything involving major rewrites, e.g. integrating themoviedb tv information, or just saying "make thing work better" without clearly stating what "better" is, or stuff that is outside the scraper's control.
Reply
(2020-02-01, 19:37)scudlee Wrote: So...  Having woken somewhat from my slumber, and seeing a lot of complaints in this thread, I decided to have a look at trying to fix the scraper.

I originally quietly abandoned the scraper because the v1 tvdb api was supposedly going to be discontinued, and I didn't have a good way to integrate the v2 api without completely rewriting everything, so I was expecting the scraper to just kind of die a death.  Instead, of course, the v1 api never stopped, and the scraper kind of muddled on, but by then my interest (in a lot of things) had passed.  A while ago though, I did do a complete rewrite of the tvdb scraper (as it turns out, purely for my own entertainment, not public consumption) which made the whole thing a lot more modular, and it turned out it was surprisingly easy to bodge it into the anidb scraper and get working.

At the moment it's not in a fit state to push to the repo - really, the code is a mess, tabs and spaces colliding, functions just copy-pasted willy-nilly, little bits of debugging here and there, etc. (plus I really need a minute to remember what I need to do to publish it.)
It also needs testing, anyway, so rather than wait until I get it looking all pretty, I figured I'd just upload the zip files straight away, and let you have at it.

[metadata.common.anidb.net] v1.2.1 (temp version number for testing)
[metadata.tvshows.anidb.net.mod] v2.4.1 (temp version number for testing)
No update for the movie scraper.  All it's important stuff is in the common library anyway.

(Install the common library first, as the tv scraper now depends on the updated version.)

Fixes:
  • TVDB/TMDB API keys updated to current ones.  I'm assuming this might have been an issue.
  • TVDB artwork restored - posters, banners, fanart (using V2 api - so no zip errors)
  • TVDB episode details restored - episode plots, thumbnails, uniqueids
  • Some tweaks to the plot filtering to fix the somewhat overzealous editing <-- CHECK THIS PLEASE
  • Decoupled the language selection from the Official/main name selection.  You can select "main" name and still have a different language for episode plots and titles (where available - English is the automatic fallback).  The series/movie plot is still from the anidb description, so only in English, that's just how that goes.
  • GetArt function added - I see some people pointing out these "errors" in the log.  It's not really an error, the GetArt function is a way for Kodi to update the artwork automatically, independent of a scan, it's primary purpose was so the user didn't have to refresh a series every time a new season was added in order to get the new season posters.  
    Since anidb shows are all one season, it didn't seem that necessary, plus the function would need the anidb id number to work, but at the time it was possible to switch the scraper to use the tvdb id (for compatibility with extrafanart addons and such), but there is no way for the function to reasonably distinguish which id it is given, since either way it's just some random number.  The scraper now ignores the tvdbid setting, and uses named uniqueids for both anidb and tvdb, and always defaults to anidb (the more up-to-date addons should look for the tvdb uniqueid now, so it doesn't matter if it's not the default), but if you were using the tvdbid before, some oddities may arise... possibly?  Refreshing should put the right uniqueids in, and fix anything.
  • There are other fixes/tweaks I idly made for myself in the past, I think, mainly in the genres and tagging areas.  General tagging is still off the table, but location, source, and audience tagging should work again (if they didn't(?)).
    Also if you see anything appear as a genre that definitely shouldn't be one, report it and it can be filtered.
  • By necessity for the tvdb api, the cachepersistence has been lowered from 48 hours to 24.  This shouldn't cause issues with anidb, 48 hours was always a little overkill, but... be aware.

My plan is to fix everything up the week after next, when I have some time, and push it around the 15th(ish).
You have until then to report any errors in the zip files or bugfixes I've missed, and to request any reasonable/achievable new features to consider.
A "reasonable/achievable" new feature would be e.g. adding a "disable ratings" setting  (consider that on the todo list).  Any tvdb-related features might also be doable with the new setup.
Unreasonable new features would be anything involving major rewrites, e.g. integrating themoviedb tv information, or just saying "make thing work better" without clearly stating what "better" is, or stuff that is outside the scraper's control. 
Awesome, welcome back. I'll check it out.
Reply
Okay, so my observations after rescraping my existing library:

-Plots now scrape correctly, without the "blablabla adaptation made in 2011 and broadcasted by blablabla" stuff.
-Fanart and posters scrape correctly.
-Episode titles scrape correctly.

Seems to work just fine for me, added a new anime out of curiosity (Hellsing Ultimate) and that one worked fine as well.

But yes, if I have to suggest a function: please make ratings optional. They bother me a lot and turning them off would make Kodi perfect for me.
Reply
(2020-02-01, 19:37)scudlee Wrote: So...  Having woken somewhat from my slumber, and seeing a lot of complaints in this thread, I decided to have a look at trying to fix the scraper.

I originally quietly abandoned the scraper because the v1 tvdb api was supposedly going to be discontinued, and I didn't have a good way to integrate the v2 api without completely rewriting everything, so I was expecting the scraper to just kind of die a death.  Instead, of course, the v1 api never stopped, and the scraper kind of muddled on, but by then my interest (in a lot of things) had passed.  A while ago though, I did do a complete rewrite of the tvdb scraper (as it turns out, purely for my own entertainment, not public consumption) which made the whole thing a lot more modular, and it turned out it was surprisingly easy to bodge it into the anidb scraper and get working.

At the moment it's not in a fit state to push to the repo - really, the code is a mess, tabs and spaces colliding, functions just copy-pasted willy-nilly, little bits of debugging here and there, etc. (plus I really need a minute to remember what I need to do to publish it.)
It also needs testing, anyway, so rather than wait until I get it looking all pretty, I figured I'd just upload the zip files straight away, and let you have at it.

[metadata.common.anidb.net] v1.2.1 (temp version number for testing)
[metadata.tvshows.anidb.net.mod] v2.4.1 (temp version number for testing)
No update for the movie scraper.  All it's important stuff is in the common library anyway.

(Install the common library first, as the tv scraper now depends on the updated version.)

Fixes:
  • TVDB/TMDB API keys updated to current ones.  I'm assuming this might have been an issue.
  • TVDB artwork restored - posters, banners, fanart (using V2 api - so no zip errors)
  • TVDB episode details restored - episode plots, thumbnails, uniqueids
  • Some tweaks to the plot filtering to fix the somewhat overzealous editing <-- CHECK THIS PLEASE
  • Decoupled the language selection from the Official/main name selection.  You can select "main" name and still have a different language for episode plots and titles (where available - English is the automatic fallback).  The series/movie plot is still from the anidb description, so only in English, that's just how that goes.
  • GetArt function added - I see some people pointing out these "errors" in the log.  It's not really an error, the GetArt function is a way for Kodi to update the artwork automatically, independent of a scan, it's primary purpose was so the user didn't have to refresh a series every time a new season was added in order to get the new season posters.  
    Since anidb shows are all one season, it didn't seem that necessary, plus the function would need the anidb id number to work, but at the time it was possible to switch the scraper to use the tvdb id (for compatibility with extrafanart addons and such), but there is no way for the function to reasonably distinguish which id it is given, since either way it's just some random number.  The scraper now ignores the tvdbid setting, and uses named uniqueids for both anidb and tvdb, and always defaults to anidb (the more up-to-date addons should look for the tvdb uniqueid now, so it doesn't matter if it's not the default), but if you were using the tvdbid before, some oddities may arise... possibly?  Refreshing should put the right uniqueids in, and fix anything.
  • There are other fixes/tweaks I idly made for myself in the past, I think, mainly in the genres and tagging areas.  General tagging is still off the table, but location, source, and audience tagging should work again (if they didn't(?)).
    Also if you see anything appear as a genre that definitely shouldn't be one, report it and it can be filtered.
  • By necessity for the tvdb api, the cachepersistence has been lowered from 48 hours to 24.  This shouldn't cause issues with anidb, 48 hours was always a little overkill, but... be aware.

My plan is to fix everything up the week after next, when I have some time, and push it around the 15th(ish).
You have until then to report any errors in the zip files or bugfixes I've missed, and to request any reasonable/achievable new features to consider.
A "reasonable/achievable" new feature would be e.g. adding a "disable ratings" setting  (consider that on the todo list).  Any tvdb-related features might also be doable with the new setup.
Unreasonable new features would be anything involving major rewrites, e.g. integrating themoviedb tv information, or just saying "make thing work better" without clearly stating what "better" is, or stuff that is outside the scraper's control. 
Thanks for the correction! Works, but I have a problem... I don't see 'ratings' now (for anime/single episodes) before applying it worked correctly. 

It is the only problem I find for now.
Reply
@NemRe 
What version of Kodi are you using?  I've changed the ratings code to use the more up-to-date format that allows for multiple named ratings.
It's almost certainly not backwards compatible.
Reply
(2020-02-02, 16:19)scudlee Wrote: @NemRe 
What version of Kodi are you using?  I've changed the ratings code to use the more up-to-date format that allows for multiple named ratings.
It's almost certainly not backwards compatible.

Kodi 16.1 – Jarvis (because it allows me to zoom correctly -in pictures- with a console controller).
Reply
  • 1
  • 62
  • 63
  • 64(current)
  • 65
  • 66
  • 71

Logout Mark Read Team Forum Stats Members Help
[MOD] AniDB.net scrapers for TV shows and Movies8