v20 - TheSportsDB Python Scraper (for Nexus and Later)

v20 TheSportsDB Python Scraper (for Nexus and Later) - Printable Version

+- Kodi Community Forum (https://forum.kodi.tv)
+-- Forum: Support (https://forum.kodi.tv/forumdisplay.php?fid=33)
+--- Forum: Add-on Support (https://forum.kodi.tv/forumdisplay.php?fid=27)
+---- Forum: Information Providers (scrapers) (https://forum.kodi.tv/forumdisplay.php?fid=147)
+----- Forum: TV Show Scrapers (https://forum.kodi.tv/forumdisplay.php?fid=305)
+----- Thread: v20 TheSportsDB Python Scraper (for Nexus and Later) (/showthread.php?tid=368208)

Pages: 1 2 3

TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-05-11

With a little urging, I've created a Python version of the TV show scraper for TheSportsDB called TheSportsDB TV Shows. This is available in the Kodi Nexus add-on repo. This is a Nexus (and later) only scraper, as the only way to uniquely name games for leagues where there are multiple games in a day was to use some features only available in Nexus.

Features:

brings in Leagues on theSportsDB as a TV show and then adds the games/matches/events for that league broken down by season and episode. See below for naming conventions for leagues that have more than one game in a day.
Includes fanart and thumbs for leagues and games.
For team sports, the team members of both the home and away team are listed as cast for each episode.
Includes the option to select the default language for the league and game descriptions. Falls back to English if the language isn't available for a given league or game.
The network broadcasting the league is listed as the TV studio

File Naming
File have to be named in a format Kodi recognizes as a TV Show episode and in a way that will uniquely identify them. For more specific information on file naming, please see:

https://kodi.wiki/view/Naming_video_files/Episodes#Single_Episode_Files

Using Dates
Kodi recognizes files that have the date in a few specific formats. You can only use this option if the sport only has one game on a given day. For example:

Formula.1.2022-03-20.Bahrain.Grand.Prix.mp4

Using Game Name
Starting with Kodi 20 (Nexus), Kodi will use the filename to try and match the title of an epsiode if the name ends with .special before the file extension. Because this is most likely to be used for sports that have multiple games in a day and teams could play each other multiple times, you need to include the date. But it needs to be numbers only, or Kodi will use the date matching above, which will cause problems. In this case the entire file name is used as a match, so it must be in the format of <league>.<date>.<name>.special.ext. For example:

English Premier League.20210813.Brentford.vs.Arsenal.special.mp4

Kodi will do fuzzy searching, so you can use periods or underscores instead of spaces, although you should still separate the items with a period. Whens scraper, Kodi will only display the actual game name (i.e. Brentford vs Arsenal). The rest of the file is just used for matching purposes.

Using Season/Episode Numbers
Kodi reconizes files with the SxxExx format. The season has to be a number, so for seasons on The SportsDB that are multi-year (i.e. 2020-2021) use the first year. The SportsDB doesn't provide episode numbers, so the scraper numbers each event in a season as it loads them. To find the episode number you'll have to count from the first game in the season (starting at 1). The file name can contain any other information you want, but Kodi will only match on the season and epsiode number. For example:

NFL.S2021E02.Atlanta.Falcons.vs.Philadelphia.Eagles.mp4

Parsing NFO Files for Leagues
This scraper does support parsing nfo files to identify a league even if the folder name does not match the league name. For more specific information on parsing nfo files, please see:

NFO_files/Parsing (wiki)

The SportsDB TV Show scraper uses the URL for the league page as the parsing URL. So, for instance, the parsing URL for MLS (American Major League Soccer) would be:

https://www.thesportsdb.com/league/4346-American-Major-League-Soccer
or
https://www.thesportsdb.com/league/4346

RE: TheSportsDB Python Scraper (for Nexus and Later) - docwra - 2022-05-18

Real World File naming examples

By Date (Use this for events that are on a specific date)

php:
C:\Sports\UFC\Season 2010\UFC.2010-10-23.UFC.121.Lesnar.vs.Velasquez.mkv

By Round (Corresponds to the round number on TheSportsDB.com)

php:
C:\Sports\UFC\Season 2022\UFC S2022E06.mkv

By Filename (Use when multiple events are on the same date and round)

php:
C:\Sports\English Premier League\2021-2022\English.Premier.League.20210815.Tottenham.vs.Man.City.special.mp4

Screenshots

RE: TheSportsDB Python Scraper (for Nexus and Later) - yhuss - 2022-06-03

Hello. Where can it be obtained. I can not find it. I'm sorry for the inconvenience.

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-06-03

(2022-06-03, 00:06)yhuss Wrote: Hello. Where can it be obtained. I can not find it. I'm sorry for the inconvenience.

I'm still waiting on it to be reviewed by the Kodi Team for the scraper repo. Once it's approved, you'll be able to get it by going to the addon area and downloading it from the main Kodi repo. In the meantime, you can get it here:

https://github.com/pkscout/metadata.tvshows.thesportsdb.python

Under the CODE button is a link to download a zip. That should install in Kodi just fine. Please remember that this scraper is Nexus only, so you need to be running the Nexus alpha to use this.

RE: TheSportsDB Python Scraper (for Nexus and Later) - zag - 2022-06-07

Out of interest what are the reasons for Nexus only requirement.

Is it something to do with the python scraping system?

Its hard to get feedback as the userbase of nexus is very low currently, i'm not even running it yet Smile

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-06-08

(2022-06-07, 13:50)zag Wrote: Out of interest what are the reasons for Nexus only requirement.

Is it something to do with the python scraping system?

Its hard to get feedback as the userbase of nexus is very low currently, i'm not even running it yet

The big one is that Nexus introduces the "specials" title scraping that actually allows the scraper to differentiate between games in a given league on the same day. Without that we would have had to add season and episode numbers to every game and then make sure people named them correctly.

The other reason is that Nexus changed the Python calls to get information into the database. I'd already done all that work for the TMDb TV Shows scraper, so when I started this one I used that as a template.

RE: TheSportsDB Python Scraper (for Nexus and Later) - zag - 2022-06-09

Awesome thanks for the clarification.

I'll try to get my main HTPC's all upgraded to Nexus nightlies for some proper testing.

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-12-08

ALERT!!!!!!!

If you are using an Kodi Nexus nightly build after about Dec 2, this scraper is now broken. The Kodi team introduced last minute breaking changes to some new Python calls for v20 that the scraper uses. I'm going to try and do the needed updates tomorrow, but there is going to be a rough transition to the next beta (or RC version). Once I make these changes and they are pushed to the repo, the scraper will not work with the current Beta1 release. You will have to update to either a nightly or the current beta/RC release of Nexus. If you want better control over when you update your Nexus beta, I would encourage you to turn off auto updates for this scraper for the time being. After you upgrade to Beta2/RC1, you can do the update and then turn auto updates back on.

I apologize for this inconvenience. The team made a decision to introduce these breaking changes to make things more stable for the v21 branch, but that means some pain right now. On top of that, I'm not going to be very available for the next month, so I'm going to be able to do only very limited support after I push these mandatory changes out.

RE: TheSportsDB Python Scraper (for Nexus and Later) - zag - 2022-12-08

(2022-12-08, 10:11)pkscout Wrote: ALERT!!!!!!!

If you are using an Kodi Nexus nightly build after about Dec 2, this scraper is now broken. The Kodi team introduced last minute breaking changes to some new Python calls for v20 that the scraper uses. I'm going to try and do the needed updates tomorrow, but there is going to be a rough transition to the next beta (or RC version). Once I make these changes and they are pushed to the repo, the scraper will not work with the current Beta1 release. You will have to update to either a nightly or the current beta/RC release of Nexus. If you want better control over when you update your Nexus beta, I would encourage you to turn off auto updates for this scraper for the time being. After you upgrade to Beta2/RC1, you can do the update and then turn auto updates back on.

I apologize for this inconvenience. The team made a decision to introduce these breaking changes to make things more stable for the v21 branch, but that means some pain right now. On top of that, I'm not going to be very available for the next month, so I'm going to be able to do only very limited support after I push these mandatory changes out.

No problems, I guess the pain is worth it in the end Smile

Thanks for keeping on top of this.

The scraper has been working great in testing, but not important until the final release really.

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-12-11

Kodi Nexus RC1 has been released, including changes that break the version of this scraper in the repo. I have submitted an updated version, but it has not yet been through code review, and there is no ETA on when that might happen. Yu can download an update for this scraper that will work with Kodi Nexus RC1 and later from:

https://github.com/pkscout/metadata.tvshows.thesportsdb.python/releases

Download the zip and then manually install from that.

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2022-12-23

(2022-12-11, 04:41)pkscout Wrote: Kodi Nexus RC1 has been released, including changes that break the version of this scraper in the repo. I have submitted an updated version, but it has not yet been through code review, and there is no ETA on when that might happen. Yu can download an update for this scraper that will work with Kodi Nexus RC1 and later from:

https://github.com/pkscout/metadata.tvshows.thesportsdb.python/releases

Download the zip and then manually install from that.

This update is in the repo now, so anyone on Nexus RC1 and later should be fine. If you're on a beta or a nightly older than RC1, you should probably upgrade to the current RC2 release.

RE: TheSportsDB Python Scraper (for Nexus and Later) - Bakadori - 2023-04-29

Hello! New user for both Kodi and the TSDB. Hopefully I'm asking in the right place something that hasn't been already answered 10 million times.

I'd like to ask about naming convention for MotoGP, more precisely about the Moto2 and Moto3 series. Do each series need to be in their separate folder from the beginning, or is there a way to keep a structure like

C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\MotoGP\Properlynamedvideofile.mp4
C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\Moto2\Properlynamedvideofile.mp4
C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\Moto2\Properlynamedvideofile.mp4

Also, I understand that the database doesn't offer info for Practice or qualifying sessions either for this or F1, but is there a way to let those rogue files to show within the series? Or do I need to switch to file browser for those?

Thanks!

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2023-04-29

(2023-04-29, 02:20)Bakadori Wrote: I'd like to ask about naming convention for MotoGP, more precisely about the Moto2 and Moto3 series. Do each series need to be in their separate folder from the beginning, or is there a way to keep a structure like

C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\MotoGP\Properlynamedvideofile.mp4
C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\Moto2\Properlynamedvideofile.mp4
C:\Sports\MotoGP-2-3\Season 2022\R01 Grand Prix of Qatar\Moto2\Properlynamedvideofile.mp4

I am not a sports person, but I did write this scraper as a favor to the person who manages TheSportsDB. Based on what I'm looking at on TheSportDB web site, I don't think that folder structure will ever work. It needs to be something like:

Code:

C:\Sports\MotoGP\Season 2022\Properlynamedvideofile.mp4

C:\Sports\Moto2\Season 2022\Properlynamedvideofile.mp4

C:\Sports\Moto3\Season 2022\Properlynamedvideofile.mp4

Each of those motorsport leagues is treated as the TV show, each year is a season, and each race in that year is an episode.

Quote:Also, I understand that the database doesn't offer info for Practice or qualifying sessions either for this or F1, but is there a way to let those rogue files to show within the series? Or do I need to switch to file browser for those?

You could do them as special episodes, and you'd have to create a complete NFO file for each one of those. This is not a novice level thing, but it is possible:

https://kodi.wiki/view/NFO_files/Episodes

RE: TheSportsDB Python Scraper (for Nexus and Later) - Bakadori - 2023-04-30

Thanks a lot for your answer! And for confirming the folder structure requirement.

Also thanks for the NFO suggestion. Looks like a big task, but I'll give it a whirl. I knew one could edit them with any text editor but I see now that there are dedicated ones. Can you recommend / suggest one that allows to batch edit fields in multiple files?

RE: TheSportsDB Python Scraper (for Nexus and Later) - pkscout - 2023-05-01

(2023-04-30, 23:42)Bakadori Wrote: Thanks a lot for your answer! And for confirming the folder structure requirement.

Also thanks for the NFO suggestion. Looks like a big task, but I'll give it a whirl. I knew one could edit them with any text editor but I see now that there are dedicated ones. Can you recommend / suggest one that allows to batch edit fields in multiple files?

Sorry, no. What few NFO files I do, I do by hand. I've never tried any of the NFO generator tools.