[Release] Simple downloader
#61
KingsCross Wrote:The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything.

Confirmed - exactly the same problem here. It seems Simple Downloader doesn't use the Commoncache (the SQLite-database in /storage/.xbmc/userdata/addon_data/script.common.plugin.cache) anymore. Nevertheless it does cache data, especially a download queue:

If I delete every temporary, unfinished download in /storage/.xbmc/userdata/addon_data/script.module.simple.downloader and try to download a video from the YouTube-addon, an empty file with a name from a download that seems to have failed before (so not the actual file I'm trying to download now) appears in that directory. So it seems something crashed during the last download(s), something is corrupted and one or more files are stuck in the download queue.

I assume this queue was stored in the Commoncache before, but is now stored elsewhere. I looked through all the Simple Downloader-directories and files and tried to grep the name of the temporary file from all files in the file system, but I can't find anything.

So deleting/resetting Simple Downloader's download queue might help temporarily - but where is it?
Reply
#62
I think you'll find the commoncache_db in userdata/cache.
Reply
#63
OK, found it - Common Plugin Cache's commoncache.db seems to be no longer in /storage/.xbmc/userdata/addon_data/script.common.plugin.cache, but in /storage/.xbmc/temp

I copied it via SCP, opened it in SQLite Database Browser, ran

Code:
UPDATE Downloader SET data = '' WHERE name = 'SimpleDownloaderQueue';

and copied it back. That resulted in the downloader not responding at all, not even showing "Preparing Download", probably because Simple Downloader expects to be at least something with a defined syntax there; an empty String won't work.

So I deleted /storage/.xbmc/temp/commoncache.db and rebooted - downloads are now working again.

Here's an excerpt from my XMBC log - the YouTube-URLs contain a lot of GET-variables that I don't know what they're for exactly, so I substituted anything looking like sensitive information (e.g. potentially password hashes etc.) with [x], because I don't want to post that on an open forum.

Code:
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:06 T:2706754368  NOTICE: [YouTube-4.4.6] _startDownload : 'Downloader not active, initializing downloader.'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderQueueLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'get' : 'SimpleDownloaderQueue'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [YouTube-4.4.6] _addItemToQueue : 'Moved Ashens and the Quest for the GameChild - Full Movie HD  Geek Week-[VS2Cnx_eL6s].mp4 to front of queue. - 8'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'set' : 'SimpleDownloaderQueue'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'SimpleDownloaderQueueLock'
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:06 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'Done: true'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _addItemToQueue : 'Done'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _processQueue : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'SimpleDownloaderQueueLock'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Downloader'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'lock' : 'Done : true'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'get' : 'SimpleDownloaderQueue'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _getNextItemFromQueue : 'queue loaded : u'[(u"The Stable of Solitude ft. Chris O\'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4", {\'Title\': u"The Stable of Solitude ft. Chris O\'Dowd | KnightHood and Decoy | YouTube Geek Week", \'url\': u\'http://r17---sn-4g57kn7e.c.youtube.com/videoplayback?ip=[x.x.x.x]&ipbits=8&fexp=[x]&mt=[x]&sver=3&expire=1375893945&upn=[x]&cp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&id=[x]&key=yt1&source=youtube&ratebypass=yes&mv=m&itag=37&ms=au&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'UWmDye4QxAA\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Ashens and the Quest for the GameChild - Full Movie HD  Geek Week-[VS2Cnx_eL6s].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r5---sn-4g57lnel.c.youtube.com/videoplayback?sver=3&cp=[x]&source=youtube&upn[x]&itag=37&mv=m&ms=au&ratebypass=yes&ipbits=8&fexp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&expire=1376154160&mt=[x]&key=yt1&ip=[x.x.x.x]&signature=[x]\', \'Title\': u\'Ashens and the Quest for the GameChild - Full Movie HD | Geek Week\', \'videoid\': \'VS2Cnx_eL6s\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Nostalgia Critic Sailor Moon-[6625851].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': \'http://j69.video2.blip.tv/14080011970964/NostalgiaCritic-NostalgiaCriticSailorMoon932.m4v?ir=181450&sr=4024\', \'videoid\': \'6625851\', \'useragent\': \'curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\', \'action\': \'download\', \'path\': \'/root/my_favorites\'}), (u\'Top 40 Shitty Shark Movies PROMO-[VKQKlzudcyo].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r4---sn-4g57lnez.c.youtube.com/videoplayback?key=yt1&ip=[x.x.x.x]&upn=[x]&fexp=[x]&itag=37&mt=1375872492&mv=m&ratebypass=yes&id=[x]&ms=au&source=youtube&expire=1375893496&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&sver=3&ipbits=8&signature=[x]\', \'Title\': u\'Top 40 Shitty Shark Movies PROMO\', \'videoid\': \'VKQKlzudcyo\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'How to watch Ashens and the Quest for the GameChild-[l-a_LxkonsE].mp4\', {\'Title\': u\'How to watch Ashens and the Quest for the GameChild\', \'url\': u\'http://r4---sn-4g57lnel.c.youtube.com/videoplayback?source=youtube&fexp=[x]&ipbits=8&ratebypass=yes&ip=[x.x.x.x]&key=yt1&expire=1375894272&sver=3&itag=22&ms=au&cp=[x]&upn=[x]&id=[x]&mv=m&mt=1375872492&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'l-a_LxkonsE\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Pogo Pins in Action-[XfxvxwSIjrU].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': u\'http://r2---sn-4g57knz7.c.youtube.com/videoplayback?mv=m&id=[x]&ms=au&cp=[x]&source=youtube&upn=[x]&expire=1375893852&mt=[x]&key=yt1&ip=[x.x.x.x]&ipbits=8&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&itag=22&fexp=[x]&sver=3&ratebypass=yes&signature=[x]\', \'Title\': u\'Pogo Pins in Action\', \'videoid\': \'XfxvxwSIjrU\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'The Matrix Sweded ft. Bart Baker and Edward Vilderman  Green Swede-[wkJx_SObhAY].mp4\', {\'Title\': u\'The Matrix Sweded ft. Bart Baker and Edward Vilderman | Green Swede\', \'url\': u\'http://r6---sn-4g57knes.c.youtube.com/videoplayback?expire=1375895915&mv=m&source=youtube&ratebypass=yes&key=yt1&cp=[x]&mt=[x]&id=[x]&sver=3&fexp=[x]&sparams=cp,id,ip,ipbits,itag,ratebypass,source,upn,expire&upn=EIda2fDmopI&itag=22&ipbits=8&ip=[x.x.x.x]&ms=au&signature=[x]\', \'download_path\': \'/storage/downloads/\', \'videoid\': \'wkJx_SObhAY\', \'action\': \'download\', \'path\': \'/root/subscriptions/new\'}), (u\'Asylum Seeker Almighty Thor-[6626036].mp4\', {\'download_path\': \'/storage/downloads/\', \'url\': \'http://j69.video2.blip.tv/14190011971385/MikeyJevs-AsylumSeekerAlmightyThor779.m4v?ir=12721&sr=1897\', \'videoid\': \'6626036\', \'useragent\': \'curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\', \'action\': \'download\', \'path\': \'/root/my_favorites\'})]''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _getNextItemFromQueue : 'returning : The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'SimpleDownloaderQueueLock'
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : ''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] '_sock_init' : 'Done: u'/storage/.xbmc/temp/commoncache.socket''
12:25:07 T:2706754368  NOTICE: [StorageClient-2.5.2] 'unlock' : 'Done: true'
12:25:07 T:3018868480   ERROR: ClearBlock: Trying to clear non existent block /storage/.xbmc/addons/script.module.simple.downloader
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _setPaths : 'Removing incomplete u"/storage/.xbmc/userdata/addon_data/script.module.simple.downloader/The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4"'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] _downloadURL : 'The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4'
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] openFile : 'u"/storage/.xbmc/userdata/addon_data/script.module.simple.downloader/The Stable of Solitude ft. Chris O'Dowd  KnightHood and Decoy  YouTube Geek Week-[UWmDye4QxAA].mp4" - 'wb''
12:25:07 T:2706754368  NOTICE: [YouTube-4.4.6] openFile : 'Trying normal: wb'
12:25:07 T:2706754368   ERROR: Exception in thread Thread-1:
                                            Traceback (most recent call last):
                                              File "./Lib/threading.py", line 551, in __bootstrap_inner
                                              File "./Lib/threading.py", line 504, in run
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 127, in _startDownload
                                                self._processQueue()
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 170, in _processQueue
                                                status = self._downloadURL(filename, item)
                                              File "/storage/.xbmc/addons/script.module.simple.downloader/lib/SimpleDownloader.py", line 548, in _downloadURL
                                                con = urllib2.urlopen(url)
                                              File "./Lib/urllib2.py", line 126, in urlopen
                                              File "./Lib/urllib2.py", line 406, in open
                                              File "./Lib/urllib2.py", line 519, in http_response
                                              File "./Lib/urllib2.py", line 444, in error
                                              File "./Lib/urllib2.py", line 378, in _call_chain
                                              File "./Lib/urllib2.py", line 527, in http_error_default
                                            HTTPError: HTTP Error 403: Forbidden

To me it looks like Simple Downloader stumbles over an 403 by YouTube and gets stuck.

BTW: Before Simple Downloader stops working (while it is still working/downloading), it shows some script-errors, but I can't find them anywhere in the log.

I noticed there's a GET-Variable "ip" in the YouTube-URL (in the log above substituted with [x.x.x.x]). Here in Germany it's quite common that you get a new IP-address for your DSL-connection every 24 hours - in fact none of the IP-addresses in the YouTube-URLs are identical with the one I have now. So maybe the downloads break if the router reconnects and gets a new IP while downloading?


BTW 2: I'm using OpenELEC 3.1.5 testing.

/EDIT:

divingmule Wrote:I think you'll find the commoncache_db in userdata/cache.
There is no such directory - maybe OpenELEC handles this differently.
Reply
#64
(2013-07-29, 01:04)AddonScriptorDE Wrote:
(2013-07-16, 20:35)KingsCross Wrote: I don't know whether this is the right place to ask but I'm also having problems with downloading videos. It gets stuck on 'preparing download' and freezes. Clearing commoncache used to work but it freezes so often on 'preparing download' that it isn't really an ideal solution. The other issue is that a commoncache file no longer appears to be being generated, so I can't clear it anyway. I've tried deleting and reinstalling the cache plugging and downloader plugging but it hasn't changed anything. I also did a fresh install of openelec and that didn't fix the issue of getting stuck on 'preparing download' either.

Any help would be greatly appreciated.
Same here. Tested on 2 systems - after a few downloads it always freezes on "prepraring download".
Would be great if this bug will be fixed.

Great addon btw! (when its working)
I would like to add to this, as I just had this problem as well. Deleting the commoncache.db file did get things working again(after trying other pointless fixesTongue). I've downloaded plenty of videos with no problem, and didn't know this was even an issue.
Reply
#65
I got stuck in queue hell too. User inflicted.

I have had ZERO problems with this addon until I caused them.

It took me longer than I want to admit to get to this thread to figure out what to do to escape the queue hell!

Once here and finding out to delete the commoncache.db did the trick in a mere 4 seconds.
Reply
#66
I also got stuck in the 403 queue. A quick workaround is to catch HTTPError from urllib2.urlopen.

SimpleDownloader.py:
Change
Code:
file = self.common.openFile(item["path_incomplete"], "wb")
con = urllib2.urlopen(url)

to
Code:
try:
    con = urllib2.urlopen(url)  
except urllib2.HTTPError, e:
    self.common.log("Probable 403 Error. Download Failed")
    self._showMessage(self.language(204), "ERROR")
    return 500

file = self.common.openFile(item["path_incomplete"], "wb")

Not the neatest solution, but it stops 403 errors from jamming the queue.

Is there an active git repo for this addon that I can pull from?

Ta
Reply
#67
I have tested downloading from youtube plugin on win7. Appears a problem with cpu load increase to 90%, and slow-downing xbmc. Have tested both on frodo and gotham. (i can provide a log on which debug mode is needed)
Reply
#68
Question 
Hello,

I successfully got simple downloader to work in my addon Big Grin

However I was not able to download files from an auth digest protected website. A popup appears with the text "Preparing download". Then nothing happens next.

I used the following url syntax without success:
Code:
https://foo:[email protected]/file.txt
and
Code:
https://foo:[email protected]/file.txt|auth=anysafe

Thanks in advance for any suggestions.

Casper
Reply
#69
(2014-03-21, 08:06)kit500 Wrote: I have tested downloading from youtube plugin on win7. Appears a problem with cpu load increase to 90%, and slow-downing xbmc. Have tested both on frodo and gotham. (i can provide a log on which debug mode is needed)

same result here - 100% CPU on both windows machines , one is stronger the other started buzzing cause of the heat.

does someone maintain this plugin - looks like no replies here and i need to know if to look for something else.

thanks

Shlomi
Reply
#70
(2012-12-31, 14:46)ulion Wrote: I modified the _downloadURL method to let it support xbmc formated url (with header options in the url after '|').
now addons can set cookies, referer, necessary headers to the download request, same with xbmc core does.
also, urlopen could raise exceptions, so I move these lines also into the try...except inner block.

Code:
def _downloadURL(self, filename, item):
        self.common.log(filename)
        url = item['url']
        url = url.split('|', 2)
        if len(url) == 2:
            url, opts = url
            opts = opts.split('&')
            opts = dict(opt.split('=', 2) for opt in opts)
            opts = dict([unquote_plus(key), unquote_plus(val)] for key, val in opts.iteritems())
        else:      
            url = url[0]
            opts = {}
        req = urllib2.Request(url)
        for key, val in opts.iteritems():
            req.add_header(key, val)
        req.add_header('User-Agent', item.get('useragent', opts.get('User-Agent', self.common.USERAGENT)))
                    
        try:
            file = self.common.openFile(item["path_incomplete"], "wb")
            con = urllib2.urlopen(req)
            
            item["total_size"] = 0
            chunk_size = 1024 * 8
            
            if con.info().getheader("Content-Length").strip():
                item["total_size"] = int(con.info().getheader("Content-Length").strip())
            
            params = {"bytes_so_far": 0, "mark": 0.0, "queue_mark": 0.0, "obytes_so_far": 0}
            item["percent"] = 0.1
            item["old_percent"] = -1

            while "quit" not in params:

COuld you please post an example how to add referer data? Thanks
Reply
#71
I don't think this code is necessary , if you set the global opener on the urllib - the simple downloader will use it , with its configured headers etc.

it works for me without using this code :-)

like this
Code:
cookie_handler = urllib2.HTTPCookieProcessor(cookiejar)
opener = urllib2.build_opener(cookie_handler)
urllib2.install_opener(opener)

Shlomi
Reply
#72
Thank you for this advice. That solved my problem Big Grin
Reply
#73
What url info needs to be passed to download a youtube video?
Thanks
Image Lunatixz - Kodi / Beta repository
Image PseudoTV - Forum | Website | Youtube | Help?
Reply
#74
(2013-07-29, 01:04)AddonScriptorDE Wrote: Same here. Tested on 2 systems - after a few downloads it always freezes on "prepraring download".
Would be great if this bug will be fixed.

Great addon btw! (when its working)

The issue of it freezing still appears to be ongoing judging by other video app threads. I guess people think its the specific video app that is the issue which is understandable.

Does anyone know if this is still being updated at all?
Reply
#75
Is there any solution to the "preparing download" problem? Its really very painful to delete the file from temp every time. Is it possible that there could be a script that runs at startup and deletes the commoncache.db file every time? So all that is needed to resume downloading is a reboot?
Reply

Logout Mark Read Team Forum Stats Members Help
[Release] Simple downloader0