Addons and execution concurrency
#1
I have received a bug report affecting to Advanced Emulator Launcher. The problem is related to the concurrent model used by Kodi tu run addons. Every time the user clicks on a menu, opens a context menu or launches a playable item, a new instance of the Python interpreter is created to run the addon. This is absolutely fine and I have trying to avoid concurrent-related problems in AEL by making AEL as blocking as possible (by using lot of progress dialogs, etc.). However, even a careful programmed addon has it limits and there are situations that this concurrency causes trouble. One example is the bug reported.

Another issue is the database corruption that plagued old good Advanced Launcher: roughly speaking, the problem was caused by two instances of AL writing at the same time to the file launchers.xml which completely destroyed the XML database. I have reduced this issue in AEL a lot with better programming practises but in theory has not completely disappeared. Yes, I know... I can use file locking mechanisms in order to avoid this problem but those are highly operating system dependant, non easily portable even in Python.

I would like to kindly ask the core developers to create a new Python API call like this

Code:
xbmcplugin.numRunningInstances()

OR

xbmcaddon.numRunningInstances()

that will return 1 if the current instance is the only one running or >= 1 if there are more than one instance running concurrently at the time of instantiating the addon. With a call like this I can make AEL to ensure never to write an XML/JSON database and show a dialog like "An operation is pending or you have not close the previously launched application. Please wait and/or close the launched app before continuing."

Alternatively, is there any other solution to solve this problem with the current Python API? Thanks a lot for your time Big Grin
Reply
#2
You can use xbmcgui.Window to store some temporary flag to signal to other processes that your database is locked. Something like that:
PHP Code:
from contextlib import contextmanager
from xbmc import Monitor
from xbmcgui import Window

monitor 
Monitor()
win Window(10000)


@
contextmanager
def lock
():
    while 
win.getProperty('locked') == 'true' and not monitor.waitForAbort(1):
        
pass
    
if monitor.abortRequested():
        
raise SystemExit
    win
.setProperty('locked''true')
    try:
        yield
    finally:
        
win.setProperty('locked''')


with lock():
    
# Do something 

Window properties are visible to all running Python instances and as I understand setProperty/getProperty use locks to ensure thread safety: https://github.com/xbmc/xbmc/blob/master...w.cpp#L625

Just want to warn you: I've invented this code out of my head just now and never tested it on practice.

PS. Fixed my example to correctly process Kodi exit.
Reply
#3
@Roman_V_M Thanks a lot, your code looks very interesting. Some questions:

1) I have no idea about the usage of the Monitor class. Is it intended for addons running on the background? Can you please tell me where is documented (better than in the Kodi Pydocs... maybe an example in Github). Is the comment in the code correct?

PHP Code:
# If property 'locked' is 'true' then monitor.waitForAbort(1) is called. After 1 second
# monitor.waitForAbort(1) will return False and while loop repeats until 'locked' property is 'false'
while win.getProperty('locked') == 'true' and not monitor.waitForAbort(1):
    
pass

# I don't understand this line. Who requests the abort? In other words, what are the conditions that make
# monitor.abortRequested() to return True?
if monitor.abortRequested():
    
raise SystemExit 

2) If AEL crashes whithin the with lock(): then the lock will not be released and we will enter a deadlock. AEL cannot be executed anymore and Kodi must be rebooted. The solution I have proposed avoids deadlocks and race conditions. I think it should be easy to implement in the core, but maybe I'm wrong...
Reply
#4
(2017-03-29, 09:54)Wintermute0110 Wrote: @Roman_V_M Thanks a lot, your code looks very interesting. Some questions:

1) I have no idea about the usage of the Monitor class. Is it intended for addons running on the background? Can you please tell me where is documented (better than in the Kodi Pydocs... maybe an example in Github).

I don't know, for me Kodi Python docs are clear. xbmc.Monitor class is used (as the name implies) to monitor various internal Kodi conditions, and it is OK to use it in different addons as many times as you need. One of those conditions is abortRequested flag which is set when Kodi is preparing to exit. Addons with long running processes must check this flag one way or another to ensure that their long running processes are properly terminated.

Quote:# If property 'locked' is 'true' then monitor.waitForAbort(1) is called. After 1 second
# monitor.waitForAbort(1) will return False and while loop repeats until 'locked' property is 'false'

Not quite. First, strings for locked and and unlocked states can be any, provided they are different. In my example an empty string '' is used for unlocked condition. Second, waitForAbort returns either after the specified interval, or immediately as soon as abortRequested flag is set. That is, if your interval is 1s and after 0.5s Kodi signals that it is going to exit, waitForAbort will return True right then.

Quote:# I don't understand this line. Who requests the abort? In other words, what are the conditions that make
# monitor.abortRequested() to return True?

See above. If your loop is exited because of abortRequeste flag, SystemExit is raised to terminate your addon and allow Kodi to exit cleanly.

Quote:2) If AEL crashes whithin the with lock(): then the lock will not be released and we will enter a deadlock. AEL cannot be executed anymore and Kodi must be rebooted.

I'm afraid you don't understand how Python context managers work. I'd recommend to read Python docs or some books, but in short a Python context manager guarantees that some clean-up action will be executed always, in every case. There are some edge cases when a context manager won't work, but if your addon is pure Python with no C code, you are safe.

PS. Just to be clear. If your Python program is terminated because of an unhandled exception, it is not a "crash". It still a normal exit (sort of) with all necessary clean-up actions like executing finally blocks, __exit__ and __del__ methods and releasing memory occupied by Python objects ("garbage collecting").
Reply
#5
(2017-03-29, 11:49)Roman_V_M Wrote: I don't know, for me Kodi Python docs are clear.

Well... excuse me but if you think that currently Kodi Pydocs are crystal clear enough so anyone that has not coded an addon before can do it so just with that documentation we have a serious disagreement here Wink

(2017-03-29, 11:49)Roman_V_M Wrote: xbmc.Monitor class is used (as the name implies) to monitor various internal Kodi conditions, and it is OK to use it in different addons as many times as you need. One of those conditions is abortRequested flag which is set when Kodi is preparing to exit. Addons with long running processes must check this flag one way or another to ensure that their long running processes are properly terminated.

So the abortRequested is triggered when the user tell Kodi to exit or some other addon tells exit to terminate, right?

(2017-03-29, 11:49)Roman_V_M Wrote: I'm afraid you don't understand how Python context managers work. I'd recommend to read Python docs or some books, but in short a Python context manager guarantees that some clean-up action will be executed always, in every case. There are some edge cases when a context manager won't work, but if your addon is pure Python with no C code, you are safe.

PS. Just to be clear. If your Python program is terminated because of an unhandled exception, it is not a "crash". It still a normal exit (sort of) with all necessary clean-up actions like executing finally blocks, __exit__ and __del__ methods and releasing memory occupied by Python objects ("garbage collecting").

Yes, although AEL is quite a complex plugin with many lines of code there are aspects of Python that I do not master yet.

OK, since AEL main code is totally encapsulated in a class like this:

PHP Code:
plugin resources.main.Main()
plugin.run_plugin() 

then there are ways in Python to trap exceptions so the lock will always be released even when the plugins exits because of an unhandled exception. I will check this out and think on a solution based on your code. Thanks again for your help.
Reply
#6
(2017-03-29, 13:01)Wintermute0110 Wrote: So the abortRequested is triggered when the user tell Kodi to exit or some other addon tells exit to terminate, right?

abortRequested is set when Kodi begins exit procedure. No matter how it was initiated: by a user selecting "Exit" option in the menu, or by receiving SIGINT from OS (not sure about this, though), or by some other way, like ShutDown built-in function.

Quote:then there are ways in Python to trap exceptions so the lock will always be released even when the plugins exits because of an unhandled exception. I will check this out and think on a solution based on your code. Thanks again for your help.

Context manager's clean-up action (__exit__ method for class-based context-managers or finally block for function-based context managers like in my example) is executed unconditionally as soon as Python interpreter leaves with block of that context manager, exception or not. This guarantees that any resources held by the context manager will be released no matter what.

PS. Just to be clear about xbmcgui.Window class. setProperty/getProperty methods allow to store and retrieve arbitrary stings. There are no pre-defined properties, AFAIK. getProperty returns an empty string if no property with such name exists.

PPS. Updated my code example. Window and Monitor is better to use globally.
Reply
#7
@Roman_V_M Thanks again. I believe I understand now how to code a solucion based on your idea to fit AEL's needs.

A little bit off-topic but it seems you are the right person to answer this. I already use setProperty() to inform skins of whether AEL is rendering Categories/Launchers or ROMs (see references [1] and [2]). Skins can then dynamically change the view which is a very nice feature. Also, I use setProperty() on window textviewer with ID 10147 to request a monospaced font when AEL is displaying text (a monospaced font is much nicer than a proportional font for AEL's purposes) (see ref [3]).

My question is: Paradix (author of Cirrus Extended) told me to use always window ID 10000 (Home window) to set global properties. Is there a reason for using ID 10000 or any other window ID can be used (for example 10001 which is MyPrograms window)?

Another question: Paradix also told me that the property set by setProperty() must be set by AEL and then set to an empty string when the addon finishes. Can the skin use the property in that way? For example, when the addon is called to render ROMs in a launcher, I set the property with setProperty(10000, 'AEL_Content', 'roms'), then fill in the lisitem contents, then xbmcplugin.endOfDirectory() and finally call setProperty(10000, 'AEL_Content', ''). Will the skin be able to use the AEL_Content property in this way?

References

[1] https://github.com/Wintermute0110/plugin...in.py#L115

[2] https://github.com/Wintermute0110/plugin...in.py#L486

[3] https://github.com/Wintermute0110/plugin...n.py#L5434
Reply
#8
I have zero knowledge of Kodi skinning so I have no idea how a skin interacts with window properties. Yes, I know that initially window properties were intended for passing some parameters to skins but that's all. In my pet projects I use them as a temporary in-memory storage.
Reply

Logout Mark Read Team Forum Stats Members Help
Addons and execution concurrency0