Looking for a method to scrape by filename
#1
Greetings,

I am trying to use the intelligent playlists in KODI as a way to play a filtered part of my videos. This failed so far.

I understand that the thing i want is not the KODI-way of life. Playlists work only in cooperation with the media library. The media library wants its information either from a .nfo or the internet. I do not want either. The Internet has no data for my Salsa Dance Videos. And creating .nfo with just the Filename seems stupid.

My solution so far was to try and write or adapt my own scraper. This also failed. The instructions I found for addon-development did not help. Error messages in KODI made me just try/guess/fail/retry. I do have a background in amateur programming. I think I understand the basics of what is happening between the xml and KODI.

My questions:
Is there a scraper out there simplified enough to build the library with just DB.title = <filename> for every file? OR
Can someone offer help in understanding the mechanics of a scraper? e. g.: Does a scraper have to do trigger a request to a website or can one have KODI omit this step?

Thanks in advance
Reply
#2
A long time ago I wrote a very simplified movie scraper that did just what you're asking, but then someone else wrote a near-identical version and posted it before I did anything with mine, so I just left it.

You can probably find theirs if you search the forum for something like "just use the filename", or something along those lines.  It might still work.

However, this is mine: metadata.justthefilename

A scraper does need to connect to an external website, first to get the search results, and then again to get the actual movie details. 
This scraper uses Google search as it handily will return back the search term in a tidy fashion in the title of the page. (The actual Google search results are completely ignored.)
It just pings Google with the filename (cleaned up by Kodi) as search term, returns it back directly as the only search result, then uses the same page (cached) to return the title as the only detail.

I think the other version used a different search engine, but was otherwise functionally the same.  It might even be possible to do it with an arbitrary website, but it would be a little more complicated.
Reply
#3
Thanks, your scraper worked for me and solved my problem.

I could not find the other solution you mentioned (just did a quick search). But beforehand google showed me some other users with my situation. They had no success afaik. So, your scraper could be helpful for other users as well.

Using google as a proxy to feed the filename back to KODI is still an odd thing and a workaround (privacy issues ignored). Maybe there will be a better solution in the future.
Reply
#4
The other scraper was written by kib. When I tried using it earlier this year it did not work on v17 and a quick search for it now shows it is no longer available... https://kodi.wiki/view/Custom_video_entries

@scudlee, maybe you should start a new thread announcing this scraper so it can be pinned and questions can be directed to it rather than random new threads.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#5
Greetings,

the Scraper that scudlee kindly provided to me stopped working. It takes a very long time for KODI to process the files but the video library does not get any updates.
At the moment I am trying to understand and debug what is happening.

The scraper uses google as an echo service for the filename.

When I use wget to send a request I get "403 Forbidden". Maybe google changed the behavior of their service?

Can someone please look into this and confirm whether this is a general problem or not?
Reply
#6
I solved my problem. It is only a local solution (PHP-Script on a server in my network that echoes the filename and a change to scudlee's scraper).
Reply
#7
(2018-12-31, 11:47)Eruydike Wrote: I solved my problem. It is only a local solution (PHP-Script on a server in my network that echoes the filename and a change to scudlee's scraper).

Hi, can you explain what you did? Would a temporary server (on a laptop) work?
Reply
#8
(2018-10-01, 19:25)scudlee Wrote: A long time ago I wrote a very simplified movie scraper that did just what you're asking, but then someone else wrote a near-identical version and posted it before I did anything with mine, so I just left it.

You can probably find theirs if you search the forum for something like "just use the filename", or something along those lines.  It might still work.

However, this is mine: metadata.justthefilename

A scraper does need to connect to an external website, first to get the search results, and then again to get the actual movie details. 
This scraper uses Google search as it handily will return back the search term in a tidy fashion in the title of the page. (The actual Google search results are completely ignored.)
It just pings Google with the filename (cleaned up by Kodi) as search term, returns it back directly as the only search result, then uses the same page (cached) to return the title as the only detail.

I think the other version used a different search engine, but was otherwise functionally the same.  It might even be possible to do it with an arbitrary website, but it would be a little more complicated.
Holy crap this has made my life so much easier. Thank you for this. Why Kodi just doesn't do this automatically when I cannot find any info to scrape is beyond me.
Reply
#9
Most folks have some material that can't be scanned online -- family videos, favorite scenes, whatever it may be. I've never really understood why it should be necessary to add these to the Kodi Library at all, but from reading the forum, that seemed to be the only way. After trying all the methods described above, I took a step back and have been happy with my videos ever since. It's really pretty simple: Make sure your videos are stored in a sensible, easy to understand structure. Under Videos --> Files, add a new source pointing to the top level of your folder structure. For type, select "None". Make sure that "Exclude from scans" is marked.  If you wish, you can make additional sources pointing to specific folders within that structure, or on different drives.

If you're using a skin that allows broad customization of the menu (Aeon Nox Silvo for me), it's simple to make as many menu items or submenu items as you want, by just pointing to Video Library --> Sources --> navigate to whatever source, or source sub-folder, you want. Choose "Create link to here --- in Files mode." If you have Kodi set to extract thumbnails from videos, you'll have a description (your filename) and a thumbnail. What more do you need? All it takes is the ability to change some folder and file names and arrange your files sensibly.
Reply
#10
(2018-12-31, 11:47)Eruydike Wrote: I solved my problem. It is only a local solution (PHP-Script on a server in my network that echoes the filename and a change to scudlee's scraper).

Any chance you could share this php script with us please?
Reply
#11
(2018-10-01, 19:25)scudlee Wrote: A long time ago I wrote a very simplified movie scraper that did just what you're asking, but then someone else wrote a near-identical version and posted it before I did anything with mine, so I just left it.

You can probably find theirs if you search the forum for something like "just use the filename", or something along those lines.  It might still work.

However, this is mine: metadata.justthefilename

A scraper does need to connect to an external website, first to get the search results, and then again to get the actual movie details. 
This scraper uses Google search as it handily will return back the search term in a tidy fashion in the title of the page. (The actual Google search results are completely ignored.)
It just pings Google with the filename (cleaned up by Kodi) as search term, returns it back directly as the only search result, then uses the same page (cached) to return the title as the only detail.

I think the other version used a different search engine, but was otherwise functionally the same.  It might even be possible to do it with an arbitrary website, but it would be a little more complicated.

Hi when i use this is works somewhat but I keep getting cannot connect to remote server a lot is there a reason that might be happening?
Reply
#12
(2019-07-31, 06:51)lrddread Wrote: Hi when i use this is works somewhat but I keep getting cannot connect to remote server a lot is there a reason that might be happening?
It is broken.

Look at the Important Threads section for the other filename scraper.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#13
(2019-07-31, 06:54)Karellen Wrote:
(2019-07-31, 06:51)lrddread Wrote: Hi when i use this is works somewhat but I keep getting cannot connect to remote server a lot is there a reason that might be happening?
It is broken.

Look at the Important Threads section for the other filename scraper. 

Thanks tried that one as well same thing happens ie cannot connect to remote server although not quite as often as I get it with the one i asked about originally. Hmmm could it be that its just my wifi connection causing the problem with this? Might have to try it with wired connection and see. Only real dif between the 2 is one uses google and the other stupid one uses yahoo instead of google really only dif i can see or pretty much and only thing i can think of that might make it give my error is to many searches to close together?
Reply
#14
(2019-07-31, 21:13)lrddread Wrote: i can think of that might make it give my error is to many searches to close together?
Possibly is an issue. Spam/DoS attack prevention by those sites? A Debug Log might show something.
My Signature
Links to : Official:Forum rules (wiki) | Official:Forum rules/Banned add-ons (wiki) | Debug Log (wiki)
Links to : HOW-TO:Create Music Library (wiki) | HOW-TO:Create_Video_Library (wiki)  ||  Artwork (wiki) | Basic controls (wiki) | Import-export library (wiki) | Movie sets (wiki) | Movie universe (wiki) | NFO files (wiki) | Quick start guide (wiki)
Reply
#15
hey guys,

I coded a simple python-based scraper based on the example python scraper code here: https://github.com/xbmc/xbmc/tree/master...emo.movies
Advantage with a python-based scraper is that it does not query any website so it works offline.

Here is my plugin to scrape file-names only in a trivial way: https://github.com/chaosgrid/kodi-filenameScraper

Just download the repo as zip and install as a plugin and you should have a filename-only scraper. If you look at the code, you could implement some more logic here https://github.com/chaosgrid/kodi-filena...ame.py#L37 so that based on the filename you set more options for the movie metadata.
Reply

Logout Mark Read Team Forum Stats Members Help
Looking for a method to scrape by filename0