Thread Rating:
  • 1 Vote(s) - 5 Average
AdultDVDEmpire Scraper
#1
After helping artik with his Excalibur Films scraper, I learned a lot more about regexp coding and scrapers in general so I was able to finish my AdultDVDEmpire scraper. This scraper will retrieve the following info:

- Film Title along with box cover.
- Production year and film studio.
- Rating (which should always be XXX, but I set it to pull anyway).
- Film Director
- Film Genres/Categories (All categories are pulled if a film fits into multiple ones).
- Film Actors/Actress along with thumbnails for each star if available.
- Film runtime.
- Film plot/tagline. *

* About the plot and tagline. Some films have just plots, some have just taglines, and some have both. Since the taglines come first in the code, I've set it to try and pull just a tagline first then it tries to pull just a plot, then tries to pull a plot if it has a tagline before it in the code. This should work most of the time but their are a few films which it fails on. Another thing is it will fail to pull the complete plot if the plot itself has a '<' bracket in it. For instance: "This here is a plot about an <b>AWESOME</b> movie!" There are a few plots like that and in that case it will pull everything up to 'an'. I tried to figure out a way around this but couldn't and finally settled with just having it pull as much as it can if it has a case like that. If it fails otherwise, it's most likely due to that particular movie having weird coding (which I've also run into during testing).

This script pulled a good majority of my collection on first try. I do have a few low budget films that it couldn't find but it was hard to even find those using google, so I'm happy either way. Here's some shots taken off my xbox with the MediaStream skin and the script:

Image Image
Image Image
Image Image



Code:
<scraper name="Adult DVD Empire" content="movies" thumb="adultdvdempire.jpg">
<NfoUrl dest="3">
<RegExp input="$$1" output="&lt;url&gt;http://www.adultdvdempire.com/itempage.aspx?item_id=\1&lt;/url&gt;" dest="3">
<expression noclean="1">adultdvdempire.com/itempage.aspx?item_id=([0-9]*)</expression>
</RegExp>
</NfoUrl>

<CreateSearchUrl dest="3">
<RegExp input="$$1" output="&lt;url&gt;http://www.adultdvdempire.com/SearchTitlesPage.aspx?SearchString=\1&lt;/url&gt;" dest="3">
<expression noclean="1"></expression>
</RegExp>

</CreateSearchUrl>

<GetSearchResults dest="6">
<RegExp input="$$5" output="&lt;?xml version=&quot;1.0&quot; encoding=&quot;iso-8859-1&quot; standalone=&quot;yes&quot;?&gt;&lt;results&gt;\1&lt;/results&gt;" dest="6">
<RegExp input="$$1" output="\1" dest="4">
<expression>&lt;a href=&quot;itempage.aspx?item_id=([0-9]*)[^&gt;]&gt;</expression>
</RegExp>
<RegExp input="$$1" output="&lt;entity&gt;&lt;title&gt;\2&lt;/title&gt;&lt;url&gt;http://www.adultdvdempire.com/itempage.aspx?item_id=\1&lt;/url&gt;&lt;/entity&gt;" dest="5">
<expression repeat="yes">ListItem_ItemTitle&quot;&gt;&lt;a href=[^=]*=([0-9]*)[^&gt;]*&gt;([^&lt;]*)</expression>
</RegExp>
<expression noclean="1"></expression>
</RegExp>
</GetSearchResults>

<GetDetails dest="3">
<RegExp input="$$5" output="&lt;details&gt;\1&lt;/details&gt;" dest="3">
<RegExp input="$$1" output="&lt;thumb&gt;http://images2.dvdempire.com/res/movies/\1h.jpg&lt;/thumb&gt;" dest="5">
<expression>BoxCover_Container&quot;&gt;[^&gt;]*&gt;&lt;img src=&quot;http://images2.dvdempire.com/res/movies/([^m]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;title&gt;\1&lt;/title&gt;" dest="5+">
<expression>Item_Title&quot;&gt;([^&lt;]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;studio&gt;\1&lt;/studio&gt;" dest="5+">
<expression>StudioProductionRating&quot;&gt;([^&lt;]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;year&gt;\1&lt;/year&gt;" dest="5+">

<expression>Year: ([0-9]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;tagline&gt;\1&lt;/tagline&gt;" dest="5+">
<expression>InfoTagLine&quot;&gt;([^&lt;]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;plot&gt;\1&lt;/plot&gt;" dest="7">
<expression clear="yes">Item_InfoContainer&quot;&gt;[^ ]*([^&lt;]*)&lt;</expression>
</RegExp>

<RegExp input="$$1" output="&lt;plot&gt;\1&lt;/plot&gt;" dest="5+">
<expression>Item_InfoContainer&quot;&gt;[^&gt;]*&gt;[^&lt;]*&lt;/span&gt;[^ ]*([^&lt;]*)&lt;</expression>
</RegExp>

<RegExp input="$$1" output="&lt;actor&gt;&lt;name&gt;\2&lt;/name&gt;&lt;thumb&gt;http://images.dvdempire.com/pornstar/actors/\1.jpg&lt;/thumb&gt;&lt;/actor&gt;" dest="5+">
<expression repeat="yes">cast_id=([0-9]*)[^t]*type=1&quot;[^&gt;]*&gt;([^&lt;]*)</expression>
</RegExp>


<RegExp input="$$1" output="&lt;genre&gt;\1&lt;/genre&gt;" dest="5+">
<expression repeat="yes">media_id=[^i]*item_id=[^&gt;]*&gt;([^&lt;]*)</expression>
</RegExp>


<RegExp input="$$1" output="&lt;runtime&gt;\1&lt;/runtime&gt;" dest="5+">
<expression>&gt;Length: ([^&lt;]*)&lt;</expression>
</RegExp>

<RegExp input="$$1" output="&lt;mpaa&gt;\1&lt;/mpaa&gt;" dest="5+">
<expression>&gt;Rating: ([^&lt;]*)</expression>
</RegExp>

<RegExp input="$$1" output="&lt;director&gt;\1&lt;/director&gt;" dest="5+">
<expression repeat="yes">type=4&quot;&gt;([^&lt;]*)</expression>
</RegExp>
<expression noclean="1"></expression>
</RegExp>
</GetDetails>
</scraper>

To spiff, if you read this: I submitted a ticket for this already.
Reply
#2
Nice!
Reply
#3
Thanks man! JadedVideo scraper lacked plots the last time I used it. And yes, plots/summaries can be a very useful feature of an adult library (though not as cool as cross-referencing by actress). Big Grin

NS
Reply
#4
I don´t get any Infos from this scraper, No Images, No Plot, No Actors, nothing...
Reply
#5
Thumbs Up 
Anacotic Wrote:I don´t get any Infos from this scraper, No Images, No Plot, No Actors, nothing...

I found the same as well. I've fixed the issue on my end and submitted a patch[1]. Hopefully this will make it to SVN soon!

[1] http://trac.xbmc.org/ticket/6047
Reply
#6
gongloo Wrote:I found the same as well. I've fixed the issue on my end and submitted a patch[1]. Hopefully this will make it to SVN soon!

[1] http://trac.xbmc.org/ticket/6047

It's in r18354. Cheers!
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#7
gongloo Wrote:I found the same as well. I've fixed the issue on my end and submitted a patch[1]. Hopefully this will make it to SVN soon!

[1] http://trac.xbmc.org/ticket/6047

ADE must have altered their layout a bit and I didn't notice (not to mention I haven't been on the forums for a while). Thanks for the fix, gongloo.
Reply
#8
I noticed after gongloo fixed the script for those changes ADE made, it screwed up the plot detection. Been messing around with it all night and changed up a few things:
- The script still pulls the same info as before. Title, front cover, now pulls the back cover as well, production year, studio, director, all actresses and actors as well as all genres and categories (great for sorting by star or style of porn), runtime, and plot + tagline.

As before, not every film has both a plot and tagline but the way ADE coded it was a little difficult to scrape. I've now got it to pull both tagline and plot if a film has both or pull just the plot if the flim doesn't have a tagline and only a plot. I haven't run across any films that have ONLY a tagline. I've just seen both, just plot, or nothing. If you run across one that has only a tagline, nothing will be pulled. Seems rare though if not non-existent. Also, just like before, plots will be scraped completely unless the plot uses HTML tags. It will then be scraped up until the first HTML tag it hits. Taglines should be complete all the time.

http://trac.xbmc.org/ticket/6215

Image Image
Reply
#9
broken - box covers

looks like url might have changed to:

Code:
http://images2.dvdempire.com/res/movies/1/

checked all r/w permissions.

might want to check it out.

r21936 / all skins.

If you rescrape it loses cover as well.
KODI1: Intel Haswell i5 NUC on Win10, Asus Chromebox on Openelec Nvidia Shield (3 Kodi Boxes) AVR1: Denon AVRX7200WA DISP1: Sammy UN78JS9500/3D *UHD*
KODI2: Arctic MC10001 AVR2: Marantz NR-1603 DISP2:Sammy UN46H7150/3D
KODI3: Zidoo X9S/Sheild 16GB 2015 AVR3: Denon AVR-X3300 DISP3: LG OLED 65G6P/3D
*UHD/4K/HDR/DV*





Reply
#10
see the sticky in this very forum
Reply
#11
i just saw it , but im don't know how to fix it. can you update the xml for us Smile

i did try however.just not that good at all this.

is there anyway i can erase my name from this post lol.
KODI1: Intel Haswell i5 NUC on Win10, Asus Chromebox on Openelec Nvidia Shield (3 Kodi Boxes) AVR1: Denon AVRX7200WA DISP1: Sammy UN78JS9500/3D *UHD*
KODI2: Arctic MC10001 AVR2: Marantz NR-1603 DISP2:Sammy UN46H7150/3D
KODI3: Zidoo X9S/Sheild 16GB 2015 AVR3: Denon AVR-X3300 DISP3: LG OLED 65G6P/3D
*UHD/4K/HDR/DV*





Reply
#12
Fixed in SVN r22012
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#13
broke again for covers.

Any idea?

Back on windows platform. Confirm same on ATV>

Everything else scrapes.
KODI1: Intel Haswell i5 NUC on Win10, Asus Chromebox on Openelec Nvidia Shield (3 Kodi Boxes) AVR1: Denon AVRX7200WA DISP1: Sammy UN78JS9500/3D *UHD*
KODI2: Arctic MC10001 AVR2: Marantz NR-1603 DISP2:Sammy UN46H7150/3D
KODI3: Zidoo X9S/Sheild 16GB 2015 AVR3: Denon AVR-X3300 DISP3: LG OLED 65G6P/3D
*UHD/4K/HDR/DV*





Reply
#14
Please create a new bug report at xbmc.org/trac (with debug log attached). Thanks.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply
#15
Fixed in SVN r22999.
Always read the online manual (wiki), FAQ (wiki) and search the forum before posting.
Do not PM or e-mail Team-Kodi members directly asking for support. Read/follow the forum rules (wiki).
Please read the pages on troubleshooting (wiki) and bug reporting (wiki) before reporting issues.
Reply



Logout Mark Read Team Forum Stats Members Help
AdultDVDEmpire Scraper51