[RELEASE] Data18.com Web Content Scraper - Adult Movie Web Downloads

  Thread Rating:
  • 2 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
DoctorD Offline
Member
Posts: 56
Joined: Apr 2013
Reputation: 3
Post: #1
This scraper uses data18.com to pull in information for downloaded adult web videos from sites like Brazzers, Naughty America, etc. For the complete list of sites supported by this scraper, you can visit the list of sites and networks on data18.com. The scraper will also sometimes work with split scene files from an adult DVD movie if the information is in Data18. The data for split scenes on Data18 is only average, so there's no guarantee that your split scene will be there. This scraper is not intended for use in feature length adult movies that come out on DVDs. For that, use the other existing Data18 scraper or the AdultDVDEmpire or ExcaliburFilms scraper (available from the XBMC Adult 18+ Repository, see installation instructions below for more detail). For Japanese Adult Videos (JAV), check out my new standalone program I wrote: https://github.com/DoctorD1501/JAVMovieScraper[/url]. The standalone program also supports data18 movies and data18 web content now!

The scraper works by doing a google search of the filename, so it's fairly lenient on what you call the file. It generally works OK if you have the actor and website name in the filename, but since this can be one of several scenes if the actor has been in multiple scenes for that website, it's also best to include the name of the scene in the file name, if you know it. If you don't know the scene name, you can usually just leave it out and go through each of the results looking for the correct scene.

Alternatively, you can go into the configuration options for the scraper and use data18's own search. Data18's search requires that the file be named exactly the same as it appears on the site, which can require a lot of work when renaming all your files, but as a plus you shouldn't get blocked like you do when you use Google to perform a large number of searches in succession.

All posters and fanarts are generated from the list of gallery images (or trailer image if there aren't any gallery images) on data18 and by default you will get image 1 set as both the poster and the fanart. You will probably want to go into "Choose Art" after the movie has been scraped and pick an image for each that is the correct aspect ratio (tall for posters, wide for fanart).

Please let me know if something doesn't appear to be scraping correctly or if you have suggestions for improvement. Thank you!

Installation Instructions
Download the file from the download link below and extract the folder into your xbmc\addons folder. After extraction, there should be a new folder in addons called "metadata.movie.data18.content.com". You should then be able to select the scraper from the movies category of scrapers when setting up a video source.

Or, add the XBMC Adult 18+ Repository to receive updates automatically. This post has the details (download the github version):

[url=http://forum.xbmc.org/showthread.php?tid=94942&pid=1545041#pid1545041]XBMC Adult 18+ Repository Release Thread

Updates
1.5 - 2013-11-26 - Fixed problems with actors not scraping properly. Added in NfoUrl section so that you can create a .nfo alongside your file containing the URL of the page on data18's content section you want the file scraped as. Renamed scraper to metadata.movie.data18.content.com to conform to rename done when it was uploaded to xbmc-adult.

1.5.1 - 2013-11-28 - Fixed genres not scraping.

1.5.2 - 2013-12-07 - Added configurable option to use either Google search (default option) or data18's own search. Data18's search requires that the file be named exactly the same as it appears on the site, which can require a lot of work on your work when renaming, but as a plus you shouldn't get blocked like you do when you use google.

1.5.3 - 2014-02-09 - Fixed problem with cast photo no longer working.

1.5.4 - 2014-09-10 - Fixed problem with scraping years due to new updated layout on data18.

1.5.5 - 2014-09-11 - Fix for <releasedate> tag. While this tag isn't supported for movies right now in XBMC, my hope is that someday it will be.

Download Link (Mediafire)
(This post was last modified: 2014-09-13 00:23 by DoctorD.)
find quote
AiWaBR Offline
Junior Member
Posts: 37
Joined: Oct 2010
Reputation: 0
Post: #2
Thanks friend worked with videos from brazzers I have more like this with the name that I have downloaded to look one by one, I will test with other sites to see, but I'm sure I'll work with videos from BangBros as it has the name the actress or the scene of the videos downloaded from the site.
It would be nice if it was also done to show as manually done manually as a sitcom, because videos are released in sequence.

Thank you and congratulations for the work.
find quote
Paule Offline
Junior Member
Posts: 1
Joined: May 2013
Reputation: 0
Post: #3
Great work! Thank you!

I have a lot of work to do now Wink
find quote
DoctorD Offline
Member
Posts: 56
Joined: Apr 2013
Reputation: 3
Post: #4
I'm not sure what you mean by "show as manually done manually as a sitcom"? Do you mean to scrape in the files in as tv shows instead of as movies? I had thought about doing that for a little while, but I decided against it because when searching for actors XBMC seems to keep the actors in TV shows separate than the actors in movies. What I really would like is for XBMC to support more than just scraping a year for movies, because data18 does have information on the actual release date (day, month, and year) of each particular file.
(This post was last modified: 2013-05-23 19:34 by DoctorD.)
find quote
Toobs Offline
Junior Member
Posts: 7
Joined: Jun 2011
Reputation: 0
Video  RE: [RELEASE] Data18.com Scraper Mod - Adult Movie Web Downloads
Post: #5
Nice work DoctorD... I always avoided putting this sort of content in my main pr0n library because it would make a mess of things!
Now if only there was a decent site we could scrape for data on JAV... Xonair used to have the more a decent index of the more popular titles but then they dropped off the face of the earth!

I have merged your modified scraper into the xbmc-adult repo for safe-keeping.
https://code.google.com/p/xbmc-adult/sou...tail?r=234

It's listed as "Data 18 Web Content"
https://code.google.com/p/xbmc-adult/sou...ontent.com
find quote
AiWaBR Offline
Junior Member
Posts: 37
Joined: Oct 2010
Reputation: 0
Post: #6
It seems that is not working, tried with several scenes and it did not work.
find quote
DoctorD Offline
Member
Posts: 56
Joined: Apr 2013
Reputation: 3
Post: #7
Thanks Toobs. Good idea on the rename and I appreciate putting it on the main site for me. As far as JAV content, we're probably going off topic here, but I've written a partially working scraper for that for dmm.co.jp. I haven't shared it with anyone because it isn't ready for release yet. The problem is the site is entirely in Japanese, so I've been using google translate on some of the fields (plot, title, director, actors, genre) to get English descriptions of them. It actually works, sorta, at least in Ember Media Manager, but in XBMC the text returned by google translate looks garbled, like it doesn't know how to decode and display UTF-8 text or something. Another problem I'm having is with calling a function in a function to google translate the actor names. I haven't gotten that to work yet (I can get their names fine as either hiragana or the name from the file thumbnail, but the file thumbnail name isn't always accurate in spelling). Maybe you can help me finish the scraper or you know someone who can?

AiWaBR, please PM me the names of the scenes that aren't working. Also, first check to see if you can even find them on data18.com manually. Sometimes they just don't have some scene from a particular website even though they have the rest from that website. And unfortunately, unlike many of the other movie database sites, it's not like we can submit any missing data.
find quote
AiWaBR Offline
Junior Member
Posts: 37
Joined: Oct 2010
Reputation: 0
Post: #8
It's a shame the same can not add new contents as in thetvdb, would be good to have a website just like to add adult content because there are many people who would like to add your videos and adult movies too.
I had a blog in wordpress that adds all the data from these sites and movies, but do not know how to shut it scrapper.
I'll send PM with some scenes that are not working.
But once again congratulations for the work DoctorD.
find quote
Toobs Offline
Junior Member
Posts: 7
Joined: Jun 2011
Reputation: 0
Post: #9
I find that Data18 takes a few days to get new releases for this kind of stuff... if you are grabbing 0day content you will probably have to wait a bit for it to show on Data18.

Re: dmm.co.jp I get intermittently region blocked whenever I try to visit that site... also the releases are listed in Japanese whereas the content I get my hands on has the title translated in English... to top things off the covers on dmm.co.jp are typically full DVD case covers (front, spine and rear in one image) which doesn't work so well with XBMC.

PS for any bugs etc. feel free to report bugs on issues tracker of the main xbmc-adult repo and we will fix them... though typically any bugs for this scraper will likely affect the standard scraper and will get fixed up that way.
find quote
skinemax Offline
Junior Member
Posts: 1
Joined: Aug 2013
Reputation: 0
Post: #10
This scraper is amazing, except that I've been having issues with google blocking it, as they don't like automated searches. Is anyone else having an issue like this? The internal search on data18 is crap it seems and can't find what google can.
find quote
AiWaBR Offline
Junior Member
Posts: 37
Joined: Oct 2010
Reputation: 0
Post: #11
Skinemax also have this problem with google by doing many searches they block access and you can not make new search.
It would be nice if this to do a search directly on the site or use other data18 quest to test whether also block access.

Thanks again for the great job DoctorD.
find quote
AiWaBR Offline
Junior Member
Posts: 37
Joined: Oct 2010
Reputation: 0
Post: #12
I think data18 changed something, is no longer appearing more actors in the scene. Tested in various scenes, including re-did scrape scenes he had done, and not add more scraper
find quote
theplastickid Offline
Junior Member
Posts: 6
Joined: Nov 2013
Reputation: 0
Post: #13
I am new to xbmc. I downloaded and installed this from the .zip file.

It says installed but I am not sure where to go from here? How do I access the files? I have Navi-X installed.
find quote
DoctorD Offline
Member
Posts: 56
Joined: Apr 2013
Reputation: 3
Post: #14
Hi All,

I've released a new version. Please use the new version from the direct download link in the first post of the thread until it gets merged into xbmc-adult. It fixes issues with actors not showing up. I've also noticed some of the fanart issues seem to have gone away when I was doing development. Anyone else still having issues with fanart in this scraper?

Recently it seems Data18 has vastly improved their search for content in that you can actually search their content database natively. I still think google is going to be the most accurate way of finding the scene as there's no standard naming convention for files and google can sometimes sort it out anyways, but it might be possible now to allow the scraper not to use google when searching. This would stop the problem with the scraper sometimes not working due to being blocked temporarily by google. If I can figure how settings work in scrapers and I have some free time, I may try to make this part of the scraper so you can optionally use this. If anyone else knows of another scraper that has a Settings feature in it that allows you to pick between two different search functions, that could greatly speed along development since I can examine how it was done there. XBMC scraper documentation is pretty out of date Sad.
find quote
DoctorD Offline
Member
Posts: 56
Joined: Apr 2013
Reputation: 3
Post: #15
I made another quick update to the scraper to fix the problems with genres not scraping. Nobody had complained it was broken, but it was something I noticed myself. Smile

You can grab the new version in the original post.
find quote
Post Reply