[scudlee]

Why have you done this?
In November 2017 the TVDB updated it's API to v2.0. Kodi needed to update the TVDB scraper to continue to access their database.


Do I need to refresh my library?
Short answer, YES. The reason for this is because Kodi stores an <episodeguide> URL from TVDB for each show, which it uses when adding new episodes. This stored URL is now wrong.

The exception to this requirement is if you have complete TV Shows to which no more episodes will be added, there is no need to refresh as the shows are not affected.

Note: If you have exported your library, and at a later stage decide to rebuild a fresh library, those exports will still hold the v1.0 guides, and you will need to refresh them. You can...

a. Use one of the methods described below, or
b. Before importing your new library, open the tvshow.nfo file and delete the <episodeguide>...</episodeguide> tags, and everything in-between those two tags. Save the nfo file and import. Useful if you want to preserve any personalised information as well as playcounts, resume points and watched status. The drawback with this method is you can no longer add new episodes to the show, so use this method on complete shows only.

How do I refresh?
1. To Refresh individual shows...

1. Highlight the tvshow
   
2. Press 'i' to bring up the info screen
   
3. Select the Refresh button
   

Note: Only the show details need updating, you don't need to update every episode if you don't want.


2. To refresh the entire library...

1. Go to your source folder(s), bring up the context menu ('c' or long-press enter)
   
2. Select Change content.
   
3. Change the content type to 'None' and agree to remove the contents from the library when prompted.  You will not lose watched/unwatched status on the files!
   
4. Repeat, but this time setting the content back to TV Shows.  Make sure the information provider is The TVDB, and check the settings!  
   
5. If you are not using English as your default language you may need to turn on the new Fallback language setting.
   

What if I'm using NFO files?

Full NFO Files - From Media Managers or Kodi Export
If you scraped before Nov 2017 using an External media manager or used Kodi to scrape and exported to NFO Files, you most likely have the wrong <episodeguide> URL.
1. Check your tvshow.nfo file and see which <episodeguide> URL is in use.

The v1.0 guide looks like this... <url cache="((ID)).xml">http://www.thetvdb.com/api/1D62F2F90030C444/series/80379/all/en.zip</url>
The v2.0 guide looks like this... <url post="yes" cache="auth.json">https://api.thetvdb.com/login?{&quot;apikey&quot;:&quot;439DFEBA9D3059C6&quot;,&quot;id&quot;:80379}|Content-Type=application/json</url>

2. If the tvshow.nfo file has the v1.0 API, they will no longer scan into the library. Simply replace the v1.0 guide with the v2.0 guide by copying the above line, ensuring you replace the TV Show ID with the correct ID as shown in red above.
3. Refresh the TV Show from your NFO Files. No need to refresh the episodes.


Parsing NFO Files
If you're just using URL NFO files (Parsing NFO) (that is, they only contain the TVDB website URL of the TV show), you still need to refresh as the wrong API is stored in the database.


What if I really don't want to refresh my library?
Warning: This is not a recommended procedure for the novice user. Undertaking this procedure is at your own risk.

It may be possible to manually update the database to directly replace the old episodeguide URLs with the new.

The old URLs look like:

Code:

<episodeguide><url cache="((ID)).xml">http://www.thetvdb.com/api/((API-KEY))/series/((ID))/all/en.zip</url></episodeguide>

The new URLs look like:

Code:

<episodeguide><url post="yes" cache="auth.json">https://api.thetvdb.com/login?{&quot;apikey&quot;:&quot;((API-KEY))&quot;,&quot;id&quot;:((ID))}|Content-Type=application/json</url></episodeguide>

These are located in the C10 column of the tvshow table of MyVideosXXX.db (where XXX is the highest number in your database directory).  
This will require a SQLite browser/editor (or MySQL) and knowledge of SQL.  It can't be done with a text editor.

It may also be possible to do this with a Python script via the JSON-RPC API, although someone with more knowledge on that would have to comment.

Caution (and back-ups) should be taken before attempting any direct manipulation of the database.


Will my artwork change?
If you don't have locally saved artwork, then refreshing may indeed pull different artwork from the TVDB.
This has to do with the fact that the scraper pulls the art with the highest voted-on average. Averages can change as more people vote; what was popular when you originally added the show to your library may no longer be so.
This would be just as true if you refreshed with the old scraper, and has little to do with the change in API.
If you manually chose a different artwork than the one pulled by the scraper then refreshing WILL undo that.


TVDB- No Longer the Default Scraper
In Nov 2018, Kodi's default scraper for TV Shows was changed from TVDB to TheMovieDB-TV Shows.

• For established libraries there will be no difference.
  
• If you are rebuilding your library from NFO Files which were exported from Kodi, and these TV Shows were scraped with the original TVDB scraper, then you will need to change the active scraper back to TVDB for Kodi to recognise the exported NFO Files. This will also be required if your media manager uses TVDB to create your NFO Files.
  

Note: The original content of this post by scudlee has been rewritten and updated numerous times by Karellen.

[jjd-uk]

New features
The first thing to note is that there was nothing particularly wrong with the old version of the scraper, provided you stick to the defaults - Aired order, Folders named exactly according TVDB, etc.
Once you started veering away from the defaults was when you may have run into issues.  Using DVD or Absolute order, or naming folders not quite as the show appears on TVDB, could all lead to unexpected outcomes.
This new version tries to address some of those issues, while hopefully still behaving the same in the default configuration.

Search improvements
The search has been improved in a couple of ways.

Firstly the year (if present) is no longer attached to the title before running the search.  Instead the year is scraped from the search results and used to aid in matching.  This is similar to how movie scrapers handle it (sometimes the year is included as part of the search query, but that's slightly different).
If you included the year as part of your folder name when it isn't needed, the old scraper would return no results.  This was apparently so bad that extra code was added to Kodi core to automatically re-run the scraper with the year ignored if there were no results - which hid the issue in most cases.
However, take the case of a US remake of another country's show, for example Being Human and Being Human (US) - If you had instead named the folder for the remake "Being Human (2011)", on the first pass the scraper would have searched for "Being Human (2011)" and found no matches.  On the second pass it would have searched for "Being Human", returned both versions of the show, and crucially, matched the earlier version over the remake - the year would not factor in at all.

Comparison of search results between the old scraper (top), the XSLT version (middle), and the new v2 API version (bottom).
All searches performed on the folder "Being Human (2011)".

Secondly, alternative/alias titles are now also returned as separate search results.  The TVDB API has for quite a while now returned search results that match against alias titles, however the old scraper would only ever return the "official" titles from the results.
This can lead to a situation where an exact match to an alias for one show is ignored for a partial match to a different show.
For example, "CSI" is an alias for the original CSI: Crime Scene Analysis, but Kodi would favor CSI: NY from the search results because it's the closest match from the official titles (since it has the fewest extra characters).
The alias titles are returned after the official titles, so an exact match to an official title will still be preferred over an exact match to an alias.

Comparison of search results between the old scraper (top), the XSLT version (middle), and the new v2 API version (bottom).
All searches performed on the folder "CSI".

(Note also the massively improved search in the new API, and increased number of aliases.  Handy, no doubt, if you use "CSI" as shorthand for Variety Show of Mr. Con & Ms. Csi (Result #14).)

Of course, the natural response to both those cases is, "You named your folder wrong", but that doesn't mean it needn't work.

Absolute order
The Absolute Episode order setting was only usable if the absolute order was defined on TVDB, if it wasn't, then the episodeguide the scraper returned would contain no episode or season numbers at all, and no episodes would ever found.  There was no fallback (unlike DVD order).
In the new version, if there isn't a defined absolute order, the scraper will generate one itself, based on the aired order.
The formula used is:
(aired) Season X, Episode Y -> (absolute) Season 1, Episode [Y+(#(episodes sAeB: X > A > 0))]  (Note: This won't be accurate if there are episodes missing on TVDB.)

Turns out the 200th episode of Stargate SG-1 is "200".  Phew!
Note the lack of episode numbers in the Absolute Order listing.
(Files are named S01E001E002.disc, S01E003.disc, ..., S01E214.disc.)

Another issue fixed, which I don't think I've ever seen brought up, but definitely happened, is that the placement of special episodes only ever used the aired order, even if absolute order was chosen.  This meant that specials that should appear within the second, or higher, (aired) season would always appear after the single (absolute) season.
The new version maps the placement back to the absolute position (using the generated value if necessary).

The first two One Piece specials appear after every other episode using the old scraper.
(S25 is really (absolute) episode 590 - This was an actual bug.)

In the new scraper, the specials appear before the correct "Season 2" episodes.
(Episode 590 shows up correctly too.)

DVD order
The most practical new feature is in the handling of DVD order.
One of the biggest issues many users have with DVD order is in the handling of double episodes - in DVD order TVDB has a habit of renumbering the two halves to be two sub-parts of a single episode, e.g. S01E01.1 and S01E01.2.
If you had the sub-parts in a single file it used to be the case that you could number that file SE01E01.1E01.2 (or equivalent) and both parts would be found.  This currently doesn't work due to a bug, but even then this had issues - both parts would display the same episode number and would be sorted alphabetically, not by sub-part.

The new version offers an alternative - by creating a single merged episode from all the sub-parts.
So instead of SE01E01.1E01.2, you number the file simply S01E01, and the details of the sub-parts are merged together.

It's probably easiest to just see this in action.

Here is the first season of Stargate SG-1 using Aired Order:

(Files are named S01E01E02.disc, S01E03.disc, ..., S01E22.disc.)

And here it is again, using DVD Order:

(Files are named S01E01.disc, S01E02.disc, ..., S01E21.disc.)

As you can see, the two-part opener is now listed as a single episode - the titles have been merged and shrunk (because they're identical apart from the bracketed number), the plots have been concatenated, even the ratings have been re-averaged (an IMDb rating will only be the rating for the first part, though).  All the other, unseen, details are merged as well.

The first season of Adventure Time using Aired Order:

(Files are named S01E01E02.disc, S01E03E04.disc, ..., S01E25E26.disc.)

And again, using DVD Order:

(Files are named S01E01.disc, S01E02.disc, ..., S01E13.disc.)

This shows what it looks like when the titles of the sub-parts are different.

The sub-parts are still available to use individually, if you have the episode split into separate files, but as noted above, the bug prevents them from working in a single file.  The only way that will work currently is as a merged episode.

Here is Adventure Time, again with DVD Order, but this time with the parts separated:

(Files are named S01E01.1.disc, S01E01.2.disc, ..., S01E13.2.disc.)

Notice how some of the pairs of episodes are swapped, due to being sorted alphabetically.

Although the examples here only have two sub-parts, any number of parts will be merged.  

Other minor DVD order fixes include:

• The old scraper didn't actually use the DVD season and DVD episode numbers directly, instead it used the "Combined" values, which are defined as being the DVD numbers if they exist, otherwise the Aired numbers.  In rare cases, though, only the DVD season might be defined (typically for a special episode) - leaving the combined values as the DVD season/Aired episode, potentially duplicating (and blocking) an actual DVD Season/DVD episode pairing.  The new scraper uses the DVD numbers directly, and then only if both are defined.
  
  
• In even rarer cases, an aired episode may not have any DVD numbers set at all, but its aired numbers still coincide with the DVD numbers of another episode (this used to be the case for Fringe S02E11 which was really a bonus season 1 episode and was left off the DVD release - it's since been given the DVD number S01E21).  To stop this blocking the actual DVD episode, but still let it be scanned, the aired episode will be remapped as the first available sub-part of that episode, so in the case of Fringe, it would have been accessible as S02E11.1  (if the DVD episode was a double episode it would be .3, etc. - it won't show up as part of the merged episode).  (This might be still be broken - VERY low priority fix needed.  Almost certainly zero shows affected.)
  
  
• As with Absolute order, the placement of special episodes is remapped to being before the correct DVD order episodes.
  

[jjd-uk]

@scudlee hope you don't mind but took the New Features part of your 1st post from the XSLT scraper testing thread and included it here so all information on new scraper is in this thread, edit as appropriate if you've got moderator powers, if not just let me know if anything requires adjustment.

[maxtherabbit]

What if I really don't want to refresh my library?
Warning: This is not a recommended procedure for the novice user. Undertaking this procedure is at your own risk.

It may be possible to manually update the database to directly replace the old episodeguide URLs with the new.

The old URLs look like:

Code:

<episodeguide><url cache="((ID)).xml">http://www.thetvdb.com/api/((API-KEY))/series/((ID))/all/en.zip</url></episodeguide>

The new URLs look like:

Code:

<episodeguide><url post="yes" cache="auth.json">https://api.thetvdb.com/login?{&quot;apikey&quot;:&quot;((API-KEY))&quot;,&quot;id&quot;:((ID))}|Content-Type=application/json</url></episodeguide>

These are located in the C10 column of the tvshow table of MyVideosXXX.db (where XXX is the highest number in your database directory).  
This will require a SQLite browser/editor (or MySQL) and knowledge of SQL.  It can't be done with a text editor.

It may also be possible to do this with a Python script via the JSON-RPC API, although someone with more knowledge on that would have to comment.

Caution (and back-ups) should be taken before attempting any direct manipulation of the database.

Is it really necessary to look for ids from the SQL database? I did some succesful testing with a couple of nfo files and I replaced the ID number with the numbers found inside <id></id> tags. Same numbers can also be found if you go to thetvdb.com, open a tv show info page and look at the page url.

[scudlee]

@maxtherabbit  That section is specifically talking about editing the database to directly replace the old URLs with the new ones, so that you don't have to refresh your library.

Editing NFO files does not get you out of having to refresh, because the only way to get the information from the NFO file to the database is by refreshing.

Unless I've misunderstood your meaning.


I wrote out the old and new URLs, and where to find them, to hopefully make it easier for someone to write a SQL command to find and replace all the URLs in the database in one go.

[maxtherabbit]

@maxtherabbit  That section is specifically talking about editing the database to directly replace the old URLs with the new ones, so that you don't have to refresh your library.

Editing NFO files does not get you out of having to refresh, because the only way to get the information from the NFO file to the database is by refreshing.

Unless I've misunderstood your meaning.


I wrote out the old and new URLs, and where to find them, to hopefully make it easier for someone to write a SQL command to find and replace all the URLs in the database in one go.

I think I misunderstood the part about having to edit the database in your guide.

I replaced the old urls in my tvshow.nfo files with the new urls you provided. Then I refreshed the shows in Kodi and it seems to get info for all old episodes and the ones that I added after the refresh.

[scudlee]

That is ABSOLUTELY a valid way to deal with the issue, though.

I deliberately left that method out of the section on NFO files because I thought it was a bit more technically involved than converting to a mixed NFO file (which just requires you to copy-paste the website URL and delete the old episodeguide URL from the file).
No fiddling about with some weird template and pasting in IDs and API keys.  If you make a mistake, you're equally stuffed.

[ponkotsu]

@scudlee Thank you for the thorough explanation of the new capabilities the new scraper and how it works. I'll be refreshing my existing shows as time allows; hopefully I finish before TVDB fully retires v1.

Not to throw more potential work at you while you're in the midst of (or from the looks of it, pretty much completed) the transition... With the search returning aliases for shows, would it be possible in the future to have the selected alias be used for the show title as stored in Kodi's database? In my particular usage case, when I scrape an anime TV show, the scraper will often return both the English-translated and the English-romanized title. Even if I select the romanized title, the scraper will populate my library with the translated title, assumedly since I have English set as my language in the scraper settings. I usually go to the show in my library afterwards and edit the title manually to the romanized title. Since the alternate title is available and selectable from the search result, would it be feasible to use the selected title instead of the primary one? I've never coded in C (or any of its variants) or Python, so I can't say how difficult or easy (or even possible) it may be. Just an item on the wishlist.

Thanks again for the quick fixes and for the great explanation!

[kaminix]

Hi Scudlee, I am using Kodi 17.3 and TVDB 2.0.3  My question pertains to your statement that everyone will need to refresh once TVDB version 1 closes, does that mean for people who have never had a library? Meaning, I have never clicked on the 'add to library' icon button when viewing the information page such as for a particular tv show. In fact the Movie and Tv show folders in the Library Folder have always been empty. Also I have never ever had a 'Refresh' icon button even on the 'information' page. I have always seen the 'Play' 'Trailer' 'Show Fanart' 'Same Director' and 'Choose Art' but never a Refresh icon, which I never understood why not. If I never use the 'Library' will I still be able to continue seeing the Tvshow and Movie Thumbnails and get the Information as I do know even after TVDB version 1 closes?

[Karellen]

@Kamix

The above requirements for refreshing your library do not apply in the following cases:

1. You access your TV Shows using File Mode and they have never been scraped into the library
   
2. Your TV Show is complete and there are no new episodes to add.
   

Although what you state seems to be contradictory. You state In fact the Movie and Tv show folders in the Library Folder have always been empty and then state . Also I have never ever had a 'Refresh' icon button even on the 'information' page. I have always seen the 'Play' 'Trailer' 'Show Fanart' ...

That is not possible. If you are viewing the Information page, then you have Library entries. Please check your system to confirm this.

[scudlee]

@runningnaked That is such a surprisingly simple thing to do, that I'm just going to tell you how.
(...Because (a) I'm not fully convinced, (b) to do it fully would mean also making it optional because not everyone would want it, and © I'd have to come up with a pithy way of saying what it is that option actually does for the settings screen, which I'm struggling with.)

Anyway, find the tvdb.xml in the metadata.tvdb.com scraper folder (NOT the one in the program folder).
Look for this line:

PHP Code:

<id><xsl:value-of select="id"/></id> 


This should be inside a template named "entity" (it's on line 142 currently).  This template is what generates each search result for each title and alias.  Now, the nice thing about the id is that it gets carried forward to the GetDetails function (while the title doesn't).  So what we can do is hide the search result title here, and then retrieve it later in GetDetails!

So replace that line with:

PHP Code:

<id><xsl:value-of select="id"/>|<xsl:value-of select="$title"/></id> 


(See we've just added the code from the <title> tags a few lines above and separated it with a pipe.  Technically we could have replaced the entire contents with the title, since we don't use this id at all in GetDetails.  We're not breaking anything.

Now move down into GetDetails, find:

PHP Code:

<RegExp input="$$1" output="&lt;title&gt;\1&lt;/title&gt;" dest="4+">
    <expression fixchars="1">"seriesName": "(.*)",\s+"aliases"</expression>
</RegExp>
<RegExp conditional="usefallbacklanguage" input="$$1" output="missingtitle|" dest="14">
    <expression clear="yes">"seriesName": null,\s+"aliases"</expression>
</RegExp> 


(Lines 165-170)  This is where the title is added to the series.
We no longer to worry about fallbacks, so delete the bottom 3 lines completely, they can only interfere.  We just need to replace the top 3 lines with code to recapture the title from the id, which is stored in buffer $$2:

PHP Code:

<RegExp input="$$2" output="&lt;title&gt;\1&lt;/title&gt;" dest="4+">
    <expression fixchars="1">^\d+\|(.+)$</expression>
</RegExp> 


The code there captures everything after the pipe (after a bunch of numbers).  If you'd replaced the whole id you could literally leave the expression blank (no spaces).

And that's it!  The matching title from the search results should now be the one used as the actual title.  How easy was that?

Obviously, assuming I don't put something like this in, every update will remove your fix, and you'll have to reapply it, but hopefully that's still easier than having to type in every title manually.

[scudlee]

That is not possible. If you are viewing the Information page, then you have Library entries. Please check your system to confirm this.

There is one way.  The clue was the 'Add to Library' button.  And it relates to the second link in your signature.

[Karellen]

Thanks. I assumed normal, local usage.

[ponkotsu]

{super-clear, amazing instructions here}

And that's it!  The matching title from the search results should now be the one used as the actual title.  How easy was that?

Obviously, assuming I don't put something like this in, every update will remove your fix, and you'll have to reapply it, but hopefully that's still easier than having to type in every title manually.

@scudlee  Thank you so much!  That does look easy, especially with the step-by-step you laid out for me!  It probably would have taken me days, if not weeks (for hospital visits due to smashing my head against the desk) if I were to try to figure it out myself lol.  Come to think of it, you also saved me tons of time doing the edits during my refresh of the TV library (380-something shows -- I'm a packrat).

I'm going to bookmark your reply so that I can refer to it for future updates of the scraper.

Thanks again for everything!  Going to make the modifications now, and start rescraping once I get home.

PS: I also agree I don't know how to word it if it were an option, so it is definitely safer to keep as it is for mass-consumption.

[kaminix]

Thank you for the replies Karellen and Scudlee, I presumed by the reference to my signature that I just got thrown into the pre-loaded box categoty LOL. Actually I use an Android box just because it's more conveniant to connect to our Tv than using my Windows Pc's and because each Pc is dedicated to a specific use such as one for Pro Tools recording and one for Video recording and I don't want other programs slowing them down. I delete as much as I can from the Android and use it just like I would one of my Pc's. It does not have preloaded anything, I load everything myself including the real Kodi 17.3 not a minix-xbmc apk and I write scripts and edit scripts all as a hobby like most of you. Thank you for the clarification Karellen about not needing to do a refresh if I only access tv shows using File Mode instead of Library and yes I did recheck and my Library folder is empty. However your statement: 'Your TV Show is complete and there are no new episodes to add', does this mean that after TVDB version 1 closes down that any new episodes won't show up anymore in my system like they currently do? And my last couple of questions are: Why do I not see a 'Refresh' icon button on my information page like you show in the picture? Is it because I use the Android arm version of Kodi or is it because I don't use the library feature? If I used the Library feature would the 'Refresh' icon show up? Thanks again.