2021-02-03, 23:29
Hi Guys,
I've started to develop a new scraper for Adult Movies and Videos. Any remarks or suggestions are very welcome.
Implemented sources: AdultDVDEmpire, TMDB, Brazzers.com, BangBros.com, DogfartNetwork.com, Kink.com, NaughtyAmerica.com, RealityKings.com. The full list can be found here. I'm working on implementing more, and you also can ask for your favorite.
If you found my software useful feel free to buy me a beer, or two.
IMPORTANT: When you update for a new version DO NOT override the Profile folder!
UPDATE 2022.04.14:
I've released the v0.10.2 version with some hotfixes you asked for.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.27:
v0.10.1 version is now available with lots of improvements in scraping and matching titles and also implemented three more scrapers for sites.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.21:
v0.10.0 version is now stable!
I have implemented the possibility to use "meta-scrapers" like ThePornDB or Data18 for scraping video info, bypassing Cloudflare DDOS protection to scrape from secured sites, and adding in-memory cache possibility to speed up the scraping. New scrapers for 18 new sites are also implemented.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.17:
A new BETA version is available with some small fixes! - v0.10.0-BETA.3
I've found a way to bypass Cloudflare's bot protection. Also, there were lots of changes in the structure of sites I had to follow up. Bang.com scraper has been added.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.10:
A new BETA version is available! - v0.10.0-BETA.2
I've found a way to bypass Cloudflare's bot protection. Also, there were lots of changes in the structure of sites I had to follow up. Bang.com scraper has been added.
For more details and to download visit the release page on GitHub.
UPDATE 2022.02.14:
A new BETA version is available! - v0.10.0-BETA.1
I have implemented the possibility to use "meta-scrapers" like ThePornDB or Data18 for scraping video info. It required a lot of code refactoring, so please, be patient if you found an error.
Also added in-memory cache possibility to speed up the scraping and 18 new sites.
For the details and to download visit the release page on GitHub.
UPDATE 2022.02.02:
A new version is available! - v0.9.0
The most important change is that AdultScraper now supports WebP images and secured images. Also, I implemented 24 new site scrapers and lots of smaller changes and bug fixes.
For the details and to download visit the release page on GitHub.
UPDATE 2022.01.29:
A new BETA version is available! - v0.9.0-BETA.2
UPDATE 2022.01.24:
Happy New Year, Guys! EMM still does not support the WebP format, although more and more sites use it. I figured out that I convert the WebP images to JPG format and then upload them to ImgBox.com and use ImgBox's URL. If you have a better idea, please let me know.
So a new BETA version with this feature is available! - v0.9.0-BETA.1
UPDATE 2021.12.13:
The stable v0.8.0 version has been released.
Changes since v0.8.0-BETA.4:
UPDATE 2021.11.28:
A new BETA version is available! - v0.8.0-BETA.4
Guys, it was a big challenge, but (I hope,) I finally found a way to use search engines through Tor network to avoid too many request errors. It's not a perfect solution, so sometimes "too many request" errors can occur, but I have no better idea, so I'll release it as stable if no new problem is found in beta 4. Also, I implemented one of the most requested features to download actors' images from TMDB. So I released it in a BETA version. Any error report is very welcome. Thanks
You can download it from Github: https://github.com/adultmm/AdultMediaMan...8.0-BETA.4
v0.8.0-BETA.4
v0.8.0-BETA.3
...and some other changes and fine-tuning under the hood.
v0.8.0-BETA.2
...and code cleaning, unit tests, and a lot of changes under the hood.
v0.8.0-BETA.1
UPDATE 2021.09.02:
A new version is available! - v0.7.8
You can download it from Github: https://github.com/adultmm/AdultMediaManager/releases
Enjoy!
UPDATE 2021.07.19:
A new version is available! - v0.7.7
Enjoy!
UPDATE 2021.06.28:
A new version is available! - v0.7.6
Enjoy!
UPDATE 2021.06.06:
A new version is available! - v0.7.5
Enjoy!
UPDATE 2021.06.04:
A new version is available! - v0.7.4
UPDATE 2021.05.31:
A new version is available! - v0.7.3
UPDATE 2021.05.23:
A new version is available! - v0.7.2
UPDATE 2021.05.20:
A new version is available! - v0.7.1
UPDATE 2021.05.16:
A new version is available! - v0.7
I've done a huge refactoring on Scrapers to handle some special cases on different sites, improved error handling, made testing easier, and also possible parallel processing in further situations. I also implemented a new ScraperSelector dialog to make it easier to select the appropriate scrapers from the huge list. And the most requested DDF Network Scraper is now available (with some others). The whole change list can be found here: https://github.com/adultmm/AdultMediaManager/releases
UPDATE 2021.04.15:
A new version is available!
The new v0.6 version supports scraping Fanarts, multiple Posters, and Screenshots from sites. You can control this behavior under Images / AdultScraper settings. I made some processing parallel, so scraping can be faster, improved auto-matching, and much more. Also added support for more than 130 sites/subsites.
From this version I distribute the application on GitHub: https://github.com/adultmm/AdultMediaManager/releases
v0.5.12
EMM Plugin version v0.5.12 x64 - Extract it under EmberMediaManager/Module folder, then enable AdultScraper under Edit/Settings/Movies. (Compatible with x64 versions only.)
v0.5.11
v0.5.10
v0.5.9
v0.5.8
v0.5.7
v0.5.6
I've started to develop a new scraper for Adult Movies and Videos. Any remarks or suggestions are very welcome.
Implemented sources: AdultDVDEmpire, TMDB, Brazzers.com, BangBros.com, DogfartNetwork.com, Kink.com, NaughtyAmerica.com, RealityKings.com. The full list can be found here. I'm working on implementing more, and you also can ask for your favorite.
If you found my software useful feel free to buy me a beer, or two.
IMPORTANT: When you update for a new version DO NOT override the Profile folder!
UPDATE 2022.04.14:
I've released the v0.10.2 version with some hotfixes you asked for.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.27:
v0.10.1 version is now available with lots of improvements in scraping and matching titles and also implemented three more scrapers for sites.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.21:
v0.10.0 version is now stable!
I have implemented the possibility to use "meta-scrapers" like ThePornDB or Data18 for scraping video info, bypassing Cloudflare DDOS protection to scrape from secured sites, and adding in-memory cache possibility to speed up the scraping. New scrapers for 18 new sites are also implemented.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.17:
A new BETA version is available with some small fixes! - v0.10.0-BETA.3
I've found a way to bypass Cloudflare's bot protection. Also, there were lots of changes in the structure of sites I had to follow up. Bang.com scraper has been added.
For more details and to download visit the release page on GitHub.
UPDATE 2022.03.10:
A new BETA version is available! - v0.10.0-BETA.2
I've found a way to bypass Cloudflare's bot protection. Also, there were lots of changes in the structure of sites I had to follow up. Bang.com scraper has been added.
For more details and to download visit the release page on GitHub.
UPDATE 2022.02.14:
A new BETA version is available! - v0.10.0-BETA.1
I have implemented the possibility to use "meta-scrapers" like ThePornDB or Data18 for scraping video info. It required a lot of code refactoring, so please, be patient if you found an error.
Also added in-memory cache possibility to speed up the scraping and 18 new sites.For the details and to download visit the release page on GitHub.
UPDATE 2022.02.02:
A new version is available! - v0.9.0
The most important change is that AdultScraper now supports WebP images and secured images. Also, I implemented 24 new site scrapers and lots of smaller changes and bug fixes.
For the details and to download visit the release page on GitHub.
UPDATE 2022.01.29:
A new BETA version is available! - v0.9.0-BETA.2
- ADD: caching ImgBox URLs
- ADD: AstraDB database as remote storage
- MOD: GoogleSearcher now tries to handle 403 error ("detected unusual traffic from your computer", reCAPTCHA page)
- MOD: add the requested URL to the error message of WebException
- FIX: resolved Kink.com "403 Forbidden" error
UPDATE 2022.01.24:
Happy New Year, Guys! EMM still does not support the WebP format, although more and more sites use it. I figured out that I convert the WebP images to JPG format and then upload them to ImgBox.com and use ImgBox's URL. If you have a better idea, please let me know.
So a new BETA version with this feature is available! - v0.9.0-BETA.1
- ADD: possibility to convert WebP images to JPG
- ADD: implement ImgBox.com upload to support image formats not supported by EMM
- ADD: implement ImgBox.com upload to support secured images
- ADD: LegalPorno.net scraper
- ADD: Private.com Network and 11 subsites (Anal Introductions.com, RussianFakeAgent.com, TightAndTeen.com and more)
- ADD: PrivateCastings.com scraper
- ADD: PrivateBlack.com scraper
- ADD: ExploitedCollegeGirls.com scraper
- ADD: CherryPimps.com scraper (also supports WildOnCam.com, CherrySpot.com)
- MOD: VixenGroup scraper (Blacked, Tushy, Vixen, Deeper, BlackedRaw, TushyRaw) now supports WebP format via ImgBox.com image proxy
- MOD: BackroomCastingCouch Scraper now supports plots with multiple paragraphs
UPDATE 2021.12.13:
The stable v0.8.0 version has been released.
Changes since v0.8.0-BETA.4:
- ADD: FilthyKings.com scraper (GammaEntertainment network)
- ADD: MyPervyFamily.com scraper (GammaEntertainment network)
- ADD: add SKU of AdultDVDEmpire to the plot (asked by Johnson_Stgt on the Kodi Forum). To enable put the next line into AdvancedSettings.xml under your EMM profile directory.
<Setting Section="AdultScraper.Shared" Content="Movie" Name="AddADEIdToPlotPattern" DefaultValue=""> SKU:%%ID%%</Setting>- MOD: WebClient now handles Request Timeout errors
- FIX: AnalVids scraper request timeout errors
- FIX: Yahoo searcher url decode problem
- FIX: RateLimiter over-indexing bug
UPDATE 2021.11.28:
A new BETA version is available! - v0.8.0-BETA.4
Guys, it was a big challenge, but (I hope,) I finally found a way to use search engines through Tor network to avoid too many request errors. It's not a perfect solution, so sometimes "too many request" errors can occur, but I have no better idea, so I'll release it as stable if no new problem is found in beta 4. Also, I implemented one of the most requested features to download actors' images from TMDB. So I released it in a BETA version. Any error report is very welcome. Thanks
You can download it from Github: https://github.com/adultmm/AdultMediaMan...8.0-BETA.4
v0.8.0-BETA.4
- ADD: PornWorld.com scraper
- ADD: BellaPass.com
- ADD: TugPass.com scraper (Petite18, Teen Tugs, Shady Spa, and 12 more sites)
- ADD: MYLKED.com scraper
- ADD: Dogfart Network now supports subsites (23 subsites, WeFuckBlackGirls.com, InterracialBlowbang.com, CumBang.com, and more)
- ADD: AnalVids.com Scraper now supports RefNo (e.g., VK000, SZ0000, etc.) in the file path or title for searching and scraper selection
- MOD: RealityKings search improved
- MOD: GenderXFilms removed GenderX.com domain (GenderXFilms.com used only)
- MOD: Movie matching now handles the "Movie name" vs. "Movie name 1" case
- MOD: removed the line break from the end of polts of some scrapers
- MOD: DDG search improvements
- MOD: Google search improvements
- MOD: do not reuse any existing Tor proxy (it caused some strange errors)
- MOD: Tor proxy change circuit improvements
- FIX: actor scraping NullReferenceException
- FIX: AnalVids scraper now avoids NullReferenceException when the search method was called without moviePath
- DEL: DDFNetworkScraper removed (DDFNetwork.com closed, merged into PornWorld.com)
v0.8.0-BETA.3
- ADD: Search Actors' images on TMDB (must be enabled under Edit > Settings > Scrapers - Data > AdultScraper)
- FIX: AdultScraper didn't handle the scraping cases without search, e.g. (Re)Scrape Movie.
- FIX: AdultScraper could clear the earlier selected external ids when a scraping was canceled.
- FIX: DorcelVision site structure changed
- FIX: Devil's Film video URL changed
- MOD: FuckYouCashSites better error handling
- MOD: GoogleSearcher cookie handling
- MOD: DDGSearcher improvements, better error handling, etc
- DEL: DDFNetworkScraper removed (DDFNetwork.com closed, merged into PornWorld.com)
...and some other changes and fine-tuning under the hood.
v0.8.0-BETA.2
- ADD: AnalVids.com now supports studio information
- ADD: WebClient tries to terminate or reuse existing Tor process after unhandled software exit
- FIX: Brazzers.com - the case when search engines don't see the valid title of the video
- FIX: Brazzers.com search "length cannot be less than zero" error
...and code cleaning, unit tests, and a lot of changes under the hood.
v0.8.0-BETA.1
- ADD: Possibility to use Tor network for searching to avoid too many request errors.
- ADD: OCR-support to read information from images.
- ADD: AnalVids.com (earlier LegalPorno.com) Scraper
- FIX: 429 or 418 errors occurred when too many requests were sent to search engines
- FIX: German users can't access some Google results because of German youth protection laws
- FIX: YahooSearcher - result page has changed
- FIX: NubileFilmsNetwork sites don't find some videos
- FIX: Project1Sercvice based Scrapers (Mofos, Babes, PropertySex, Twistys, and a lot more) don't find new videos
- FIX: MaxHardcore scraper timezone handling
UPDATE 2021.09.02:
A new version is available! - v0.7.8
- ADD: MaxHardcore scraper
- ADD: Nubiles.net support
- MOD: NubilesFilmsNetwork scraper now handles the case when the search result contains the actress' name
- FIX: NaughtAmerica search sometimes didn't find the video
- FIX: NubileFilms Network search sometimes didn't find the video
- FIX: DDG Searcher sometimes didn't find results when the URL contained - (minus) sign
- FIX: Wankz video url changed
- FIX: RoccoSiffredi.com video url changed
- FIX: Brazzers Scraper errors retrieving specific video details (https://github.com/adultmm/AdultMediaManager/issues/13)
You can download it from Github: https://github.com/adultmm/AdultMediaManager/releases
Enjoy!
UPDATE 2021.07.19:
A new version is available! - v0.7.7
- FIX: GoogleSearcher 409 error when you used it too much. Now we use multiple search engines for Scrapers: Brazzers, FuckYouCashSites, NaughtyAmerica, NewSensations, NubileFilmsNetwork, RealityKings, TeamSkeetNetworkNoSearch, XEmpire.
- FIX: video scoring problem with requested year
- FIX: runtime scraping of Movies (it was buggy)
- FIX: LetsDoeIt scraper search failed
- FIX: FuckYouCashSites supports posters on pages without player
- FIX: now GammaEntertainment Scraper supports videos with empty title (eg: https://www.evilangel.com/en/video/-/122027 )
- MOD: NubileFilmsNetwork - cut the title after episode info
- ADD: RateLimiter improvements
- ADD: UnderTheBed.com subsite to PureTaboo.com
- ADD: GiveMeTeens.com subsite to Devil's Film scraper
Enjoy!
UPDATE 2021.06.28:
A new version is available! - v0.7.6
- ADD: more than 60 websites
- and some bugfix
- see the details here: https://github.com/adultmm/AdultMediaMan...tag/v0.7.6
Enjoy!
UPDATE 2021.06.06:
A new version is available! - v0.7.5
- FIX: VixenGroup has changed the URL of video pages (Blacked, Tushy, Vixen, Deeper, BlackedRaw, TushyRaw)
Enjoy!
UPDATE 2021.06.04:
A new version is available! - v0.7.4
- FIX: VixenGroup search had been changed (Blacked, Tushy, Vixen, Deeper, BlackedRaw, TushyRaw)
- FIX: XEmpire scraper releasedate format (HardX, AllBlackX, EroticaX, DarkX, LesbianX)
- FIX: NewSensations scraper releasedate format
- FIX: DDFNetwork scraper releasedate format
UPDATE 2021.05.31:
A new version is available! - v0.7.3
- ADD: XEmpire Network's sites (HardX, AllBlackX, EroticaX, DarkX, LesbianX)
- MOD: use Google as the first search engine for search on Brazzers (so we can support year for results)
- FIX: Brazzers.com changed the DOM, so we couldn't find tags
- FIX: NewSensations Scraper updated to the new site structure (Title and Plot)
UPDATE 2021.05.23:
A new version is available! - v0.7.2
- FIX: Brazzers didn't find some videos, so we now use multiple search engines
- MOD: NewSensations now uses multiple search engines
- MOD: TeamSkeetNetwork now uses multiple search engines
UPDATE 2021.05.20:
A new version is available! - v0.7.1
- FIX: DorcelVision scraper language selector
- FIX: NewSensations scraper didn't find some video
- FIX: LetsDoeIt changed the structure of the site
- FIX: HotBabesPlus.com has become part of VIPSexVault.com
- MOD: BackroomCastingCouch scraper now supports runtime
UPDATE 2021.05.16:
A new version is available! - v0.7
I've done a huge refactoring on Scrapers to handle some special cases on different sites, improved error handling, made testing easier, and also possible parallel processing in further situations. I also implemented a new ScraperSelector dialog to make it easier to select the appropriate scrapers from the huge list. And the most requested DDF Network Scraper is now available (with some others). The whole change list can be found here: https://github.com/adultmm/AdultMediaManager/releases
UPDATE 2021.04.15:
A new version is available!
The new v0.6 version supports scraping Fanarts, multiple Posters, and Screenshots from sites. You can control this behavior under Images / AdultScraper settings. I made some processing parallel, so scraping can be faster, improved auto-matching, and much more. Also added support for more than 130 sites/subsites.
From this version I distribute the application on GitHub: https://github.com/adultmm/AdultMediaManager/releases
v0.5.12
- MOD: GammaEntertainment scrapers now report Sci-Fi and Award-Winning categories as genres
- FIX: workaround for EmberMM loses new Studio, Genre, Country for the first time. http://embermediamanager.thebuggenie.com...es/EMM-202
- FIX: app crashes when scrape found unnamed category (eg. PureTaboo)
- FIX: duplicated search results in AdultDVDEmpire scraper
- FIX: the actor name was the title of the movie in some cases in AdultDVDEmpire scraper
EMM Plugin version v0.5.12 x64 - Extract it under EmberMediaManager/Module folder, then enable AdultScraper under Edit/Settings/Movies. (Compatible with x64 versions only.)
v0.5.11
- ADD: Title matching now handles roman numbers (eg. Any Movie VI = Any Movie 6)
- ADD: TeenMegaWorld.com Network support (36 sites: 18FirstSex.com, TrickyMasseur.com, AnalBeauty.com, and more)
- ADD: Dorcel Vision Movie scraper (multilanguage)
- ADD: new Devil's Film site: OutOfTheFamily.com
- ADD: NewSensations.com Video (scenes) scraper
- MOD: speed up AdultDVDEmpire scraper a bit
- FIX: scraping error in some cases when no Actors found
- FIX: AdultScraperSettings / MovieFieldScrapers design fix
- FIX: scraping by TMDB ID didn't work in some cases
v0.5.10
- ADD: settings to add the source (site) to the Title of the video
- ADD: support Girlsway.com and subsites (16 sites: GirlsTryAnal.com, LezCuties.com, WhereTheBoysArent.com, and more)
- ADD: support EvilAngel.com and subsites (21 sites: Buttman.com, AnalAcrobats.com, MilkEnema.com, and more)
- ADD: PureTaboo.com site support
- ADD: Wicked.com site support
- ADD: GirlfriendsFilms.com site support
- ADD: BurningAngel.com site support
- ADD: 21Naturals.com site, and 2 subsites support (21FootArt.com, 21EroticAnal.com)
- ADD: GenderX.com site support
- ADD: DevilsFilm.com site support
- ADD: RoccoSiffredi.com site support
- ADD: 21Sextreme.com site and 6 subsites support (GrandpasFuckTeens.com, DominatedGirls.com, TeachMeFisting.com, and more)
- MOD: VideoScraper now removes invalid (eg. empty) genres
- MOD: skip media info tag option now removes 'HD Porn' as well
- FIX: replace <br> in the plot to new line
- FIX: remove whitespace from both ends of the plot
v0.5.9
- ADD: Vixen Group scraper supports two more sites: TushyRaw, BlackedRaw
- ADD: BackroomCastingCouch.com support
- ADD: Vixen Group scraper now supports Genres
- ADD 21Sextury.com sites (22 sites: AnalTeenAngels.com, LezCuties.com, AssHoleFever.com, DPFanatics.com, and more)
- FIX: "Object reference not set to an instance of an object". error, when the Video has no genre and the "Convert genre case" is not "No change".
- FIX: Brazzers rating cannot be read (DOM changed)
v0.5.8
- ADD: CzechAV Network sites - (CzechAmateurs.com, CzechStreets.com, CzechMegaSwingers.com and more, 34 sites)
- ADD: Vixen Media sites: Blacked, Deeper, Tushy, Vixen
v0.5.7
- FIX: "empty scraper list" error (reported by Strubdog - thanks)
- FIX: Brazzers.com video URL matcher changed
- FIX: RealityKing scraper didn't find the video in some cases
v0.5.6
- fix AdultDVDEmpire search scoring
- fixed RealityKings changed video URL format
