[Release] Common plugin cache

  Thread Rating:
  • 1 Votes - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
flyingrat Offline
Senior Member
Posts: 258
Joined: Jul 2008
Reputation: 5
Location: Sweden
Post: #76
During some profiling work i noticed that thread XBPython StorageServer.py ("Common plugin Cache") keeps hammering my system with a socket select in StorageServer.py every 0,5 sec. This consumes a lot of cpu power on iPad/iPhone/Androids/RPi's. Is this a bug or feature? "time.sleep(0.5)" is the guilty line down below in section 2. Btw, what's the purpose of that poll loop? /Thanks, Lars.

Call path:

1. File: "addons/script.common.plugin.cache/default.py"
Code:
def run():
    sys.path = [settings.getAddonInfo('path') + "/lib"] + sys.path
    import StorageServer
    s = StorageServer.StorageServer(False)
    print " StorageServer Module loaded RUN"
    print s.plugin + " Starting server"
    s.run()
    return True


2. File: "addons/script.common.plugin.cache/lib/StorageServer.py"
Code:
while not self._aborting():
            if waiting == 0:
                self._log("accepting", 3)
                waiting = 1
            try:
                (self.clientsocket, address) = sock.accept()
                if waiting == 2:
                    self._log("Waking up, slept for %s seconds." % int(time.time() - idle_since))
                waiting = 0
            except socket.error, e:
                if e.errno == 11 or e.errno == 10035 or e.errno == 35:
                    # There has to be a better way to accomplish this.
                    if idle_since + self.idle < time.time():
                        if self.instance:
                            self.die = True
                        if waiting == 1:
                            self._log("Idle for %s seconds. Going to sleep. zzzzzzzz " % self.idle)
                        time.sleep(0.5)
                        waiting = 2
                    continue
                self._log("EXCEPTION : " + repr(e))
            except:
                pass

1. XBMC: http://github.com/FlyingRat/xbmc (ffmpeg-head-inc-xbmc-patches)
2. FFmpeg: http://github.com/FlyingRat/FFmpeg (ffmpeg-head-with-xbmc-custom-patches)
3. XBMC-updated-FFmpeg-binaries (just dev snapshots, no regular distros)
(This post was last modified: 2013-05-03 17:49 by flyingrat.)
find quote
flyingrat Offline
Senior Member
Posts: 258
Joined: Jul 2008
Reputation: 5
Location: Sweden
Post: #77
Anyone who knows if there is any official maintainer of "Common plugin Cache" nowadays?

1. XBMC: http://github.com/FlyingRat/xbmc (ffmpeg-head-inc-xbmc-patches)
2. FFmpeg: http://github.com/FlyingRat/FFmpeg (ffmpeg-head-with-xbmc-custom-patches)
3. XBMC-updated-FFmpeg-binaries (just dev snapshots, no regular distros)
find quote
TobiasTheCommie Offline
Skilled Python Coder
Posts: 618
Joined: Apr 2008
Reputation: 6
Post: #78
The purpose of the loop is to see if a client has send a request on the socket.

The time.sleep(0.5) makes xbmc NOT hit 100% on normal x86/amd64 cpu's.

I'm currently watching an episode on my pi. xbmc is idling at 27% cpu. With the commoncache running.

(2013-03-24 20:17)mswiege Wrote:  @Domin (and others who get the "NameError: global name 'sqlite' is not defined" error): The "problem" is, that we use sqlite3 and not sqlite anymore and the devlopers missed to rewrite the "except" to cover that. So the first thing that needs to be done is change line 116 of the StorageServer.py from
Code:
except sqlite.Error, e:
to
Code:
except Exception, e:
This fixes the crash shown in our logs that the "sqlite" name is not defined.

After changing the code restart XBMC and you should now see a new message in your logs (the cause why it catched an Exception in the first place).
In my case the new exception was this:
Code:
[StorageServer-2.5.0] '_startDB' : 'sql3 - C:\Users\My Username\AppData\Roaming\XBMC\userdata\addon_data\script.common.plugin.cache\com​moncache.db'
[StorageServer-2.5.0] '_startDB' : 'Exception: OperationalError('unable to open database file',)'

The problem here is that the database file could not be created, but I wasn't able to find the error in the code. Still was able to get rid of this error by simply creating the folder mentioned above manually (C:\Users\My Username\AppData\Roaming\XBMC\userdata\addon_data\script.common.plugin.cache). After that everything worked as expected.

tl;dr: In line 116 of StorageServer.py replace "sqlite.Error" with "Exception" and create a folder called "script.common.plugin.cache" in your "addon_data" folder.

Patched. That was stupid of me.
(This post was last modified: 2013-05-07 20:16 by TobiasTheCommie.)
find quote
TobiasTheCommie Offline
Skilled Python Coder
Posts: 618
Joined: Apr 2008
Reputation: 6
Post: #79
(2013-03-19 22:53)awschmidt Wrote:  I have an issue with this script, when xbmc is started while UMS or PMS is already running, it tries to use the same socketadress:

Patched
(This post was last modified: 2013-05-07 20:46 by TobiasTheCommie.)
find quote
flyingrat Offline
Senior Member
Posts: 258
Joined: Jul 2008
Reputation: 5
Location: Sweden
Post: #80
(2013-05-07 20:08)TobiasTheCommie Wrote:  The purpose of the loop is to see if a client has send a request on the socket.

The time.sleep(0.5) makes xbmc NOT hit 100% on normal x86/amd64 cpu's.

I'm currently watching an episode on my pi. xbmc is idling at 27% cpu. With the commoncache running.

Well, 27% cpu is not really my definition of idling. Wink
But anyhow, I was actually wondering about the propose of the "poll loop" where accept() polls an empty queue of pending connections (and returns error) instead of just call listen(bigval) with error control and then make accept() return only with an pending connection, i.e why:

Code:
sock.listen(1)
sock.setblocking(0)

Btw, here are some suggestions how to avoid polling:
- http://stackoverflow.com/questions/10090...ccept-call
- http://docs.python.org/2/howto/sockets.h...ng-sockets
- http://pymotw.com/2/select

1. XBMC: http://github.com/FlyingRat/xbmc (ffmpeg-head-inc-xbmc-patches)
2. FFmpeg: http://github.com/FlyingRat/FFmpeg (ffmpeg-head-with-xbmc-custom-patches)
3. XBMC-updated-FFmpeg-binaries (just dev snapshots, no regular distros)
(This post was last modified: 2013-05-07 22:36 by flyingrat.)
find quote
TobiasTheCommie Offline
Skilled Python Coder
Posts: 618
Joined: Apr 2008
Reputation: 6
Post: #81
(2013-05-07 21:02)flyingrat Wrote:  
(2013-05-07 20:08)TobiasTheCommie Wrote:  The purpose of the loop is to see if a client has send a request on the socket.

The time.sleep(0.5) makes xbmc NOT hit 100% on normal x86/amd64 cpu's.

I'm currently watching an episode on my pi. xbmc is idling at 27% cpu. With the commoncache running.

Well, 27% cpu is not really my definition of idling. Wink
But anyhow, I was actually wondering about the propose of the "poll loop" where accept() polls an empty queue of pending connections (and returns error) instead of just call listen(bigval) with error control and then make accept() return only with an pending connection, i.e why:

Code:
sock.listen(1)
sock.setblocking(0)

IIRC it caused problems on some systems if i didn't do it like that.

When i initially tuned the timeout, at 0.5s i didn't see any change in cpu usage from xbmc, compared to running xbmc with the cache disabled.

So 0.5s had no performance impact on my test systems. Granted i haven't tested on the rpi.

If you really think it is so much of an issue, you can just turn off the "auto start". Then the cache will be initiated when needed. And quit after it has been idle for a little while.
find quote
TobiasTheCommie Offline
Skilled Python Coder
Posts: 618
Joined: Apr 2008
Reputation: 6
Post: #82
(2013-05-07 21:02)flyingrat Wrote:  Well, 27% cpu is not really my definition of idling. Wink
But anyhow, I was actually wondering about the propose of the "poll loop" where accept() polls an empty queue of pending connections (and returns error) instead of just call listen(bigval) with error control and then make accept() return only with an pending connection, i.e why:

Btw, here are some suggestions how to avoid polling:
- http://stackoverflow.com/questions/10090...ccept-call
- http://docs.python.org/2/howto/sockets.h...ng-sockets
- http://pymotw.com/2/select

Does xbmc actually use less cpu on your pi if you disable the commoncache autostart?
find quote
lzoubek Offline
Member
Posts: 81
Joined: Aug 2010
Reputation: 0
Post: #83
Hi guys, thanks for great work on this plugin.

I wanted to use cacheFunction for class methods. I was able to simply override __str__ class function and it worked like a charm.
Just Sharing a piece of code https://gist.github.com/lzoubek/5572438
find quote
flyingrat Offline
Senior Member
Posts: 258
Joined: Jul 2008
Reputation: 5
Location: Sweden
Post: #84
(2013-05-08 16:30)TobiasTheCommie Wrote:  
(2013-05-07 21:02)flyingrat Wrote:  Well, 27% cpu is not really my definition of idling. Wink
But anyhow, I was actually wondering about the propose of the "poll loop" where accept() polls an empty queue of pending connections (and returns error) instead of just call listen(bigval) with error control and then make accept() return only with an pending connection, i.e why:

Btw, here are some suggestions how to avoid polling:
- http://stackoverflow.com/questions/10090...ccept-call
- http://docs.python.org/2/howto/sockets.h...ng-sockets
- http://pymotw.com/2/select

Does xbmc actually use less cpu on your pi if you disable the commoncache autostart?

Only when it's activated. The cpu is distributed 50% between commoncache and rendering. I'm testing a scenario where we replace the poll loop with the event onAbortRequested that is triggered during service shutdown that will reduce the idle cpu to 0%. Unfortunately I'm having trouble with onAbortRequested one at the moment ("xbmc interface callback to python onAbortRequested fails") but I'll let you know when i have a working solution ready for test. /Cheers, Lars.

1. XBMC: http://github.com/FlyingRat/xbmc (ffmpeg-head-inc-xbmc-patches)
2. FFmpeg: http://github.com/FlyingRat/FFmpeg (ffmpeg-head-with-xbmc-custom-patches)
3. XBMC-updated-FFmpeg-binaries (just dev snapshots, no regular distros)
find quote
TobiasTheCommie Offline
Skilled Python Coder
Posts: 618
Joined: Apr 2008
Reputation: 6
Post: #85
(2013-05-14 22:29)flyingrat Wrote:  
(2013-05-08 16:30)TobiasTheCommie Wrote:  
(2013-05-07 21:02)flyingrat Wrote:  Well, 27% cpu is not really my definition of idling. Wink
But anyhow, I was actually wondering about the propose of the "poll loop" where accept() polls an empty queue of pending connections (and returns error) instead of just call listen(bigval) with error control and then make accept() return only with an pending connection, i.e why:

Btw, here are some suggestions how to avoid polling:
- http://stackoverflow.com/questions/10090...ccept-call
- http://docs.python.org/2/howto/sockets.h...ng-sockets
- http://pymotw.com/2/select

Does xbmc actually use less cpu on your pi if you disable the commoncache autostart?

Only when it's activated. The cpu is distributed 50% between commoncache and rendering. I'm testing a scenario where we replace the poll loop with the event onAbortRequested that is triggered during service shutdown that will reduce the idle cpu to 0%. Unfortunately I'm having trouble with onAbortRequested one at the moment ("xbmc interface callback to python onAbortRequested fails") but I'll let you know when i have a working solution ready for test. /Cheers, Lars.

If it can work in mac, linux, windows, and rPI, without polling, then that is clearly to preferred. I'd be more than willing to accept a patch.
find quote
Eldorado Offline
Fan
Posts: 534
Joined: May 2009
Reputation: 14
Post: #86
I noticed the most recent update switched the path of the DB back to the Database folder.. but... everyone has now lost their settings that I was storing in common cache which is now orphaned in the old location

Just posting as a heads up, maybe if it ever changes again you can include a quick function to move it if currently exists? Smile
find quote
azybmz Offline
Donor
Posts: 10
Joined: Aug 2013
Reputation: 0
Post: #87
Hi, I am running Android 4.2.2 on a rooted MK908 and script.common.plugin.cache errors on startup. I am running the latest version 2.5.2. Here are the logs - http://xbmclogs.com/show.php?id=47045; it errors out with a sql3 error. I saw old threads where a similar problem was fixed in version 2.5.1.

--
Thanks.
find quote
azybmz Offline
Donor
Posts: 10
Joined: Aug 2013
Reputation: 0
Post: #88
I changed the location of commoncache.db to 'special://database', now I don't see the sql error anymore, but I still get a notification that commoncache failed to start. The new logs files are at - http://xbmclogs.com/show.php?id=47394. Please help.

--
Thanks.
find quote
phinc Offline
Junior Member
Posts: 23
Joined: Aug 2013
Reputation: 0
Post: #89
I have the same problem like azybmz. Tried with Android 4.2.2 and 4.1.1 on my MK808B. I already posted this in the Android section but got no reply. Here is my post:
(2013-09-01 15:38)phinc Wrote:  I installed the gotham alpha 7 for android today. When i installed the youtube plugin, plugin.commom.cache was installed automatically and now it crashes on every boot saying: Failed to start commoncache. Check log."
Here is my log file: http://xbmclogs.com/show.php?id=53312
And here is a log with debugging enabled in commoncache plugin settings: http://xbmclogs.com/show.php?id=53317
I also had this problem with previous alphas. The last one i can remember that worked was alpha1.

Please help me to fix this problem!

It is very strange because i found no errors in the log that may relate to the error message.
find quote
seicalata Offline
Junior Member
Posts: 1
Joined: Sep 2013
Reputation: 0
Post: #90
I also I have the same error common cache plugins right in the beginning the program xbmc. I have a stick CX-919 android 4.2.2. I was with frodo 12.2 smoothly and then I tried last nightly build when I started having problems with this plugin which is installed automatically with or pelisalacarta sportdevil. I do not know if this error affects the functioning of the plugins that I have tested and go well if needed will include the log with running plugins.

Here is my log: http://xbmclogs.com/show.php?id=54434

I used the nighty build aceleraccion to use the hardware of my stick activating "stagefright" in advancedsettings.xml but not if frodo 12.2 this can also activate but from what I've read I think not.
(This post was last modified: 2013-09-03 20:27 by seicalata.)
find quote
Post Reply