[pvagner]

Hello,
I have good and also bad news.
The good news is that I no longer have to poll for the current control and I can avoid hammering json interface over http. I have implemented the approach outlined in my previous post (edits 7 and 8). It's currently divided into two add-ons in order to be able to make it also running on linux. Currently I have only tested it on Windows. Each of these add-ons is only a few lines of code.
The first add-on only provides interface to a TTS engine. To make it really simple I am just sending text string to NVDA screen reader on Windows.
The second addon parses its arguments, if an argument named action is found it uses Action built-in function to execute that action also waiting for that action to finish. Finally that addon calls that JSON API query I have posted in my previous message parsing the output and sending that for text to speech to the first add-on.
Then to make it work I have to install modified keymap to my user profile which replaces actions that should trigger the speaking.
For example I have just bound left, right, up and down arrow keys so after moving inside a menu or list I will get spoken feetback.

The bad news is that there is most likelly nothing more I can do. in order to inprove the access. There is no way on how to retrieve states of the controls for example I can't retrieve if a check control is selected. There is no way on how to speak content of notifications. There is no way on howto access and / or get feetback about the text while typing into an edit field. There is no way on how to retrieve the text for reading for example while reading add-on changelogs and other info. There is no way to retrieve info labels as they are displayed. The only way around that would be to write seperate info labels presentation bound to this add-on specific keyboard controls.

As a prototype this is hopefully a good attempt but still it is verry limited. Imagine it provides similar level of access we had back in the days to devices running Android 1.6.

I can see in the XBMC 13 version there is going to be support for JSON notifications using python. Ideally we would need a lot of notifications added to make all this usefull. For example focus change when arrowing inside a menu or inside a list. Window activate while opening new wiews or new folders. Text added and text inserted notifications while typing into edit fields. And also it would be helpfull to be able to access other control properties with xbmcgui not just the base class properties so we might be able to make the flat representation of the current window and provide access to the control states such as selected, disabled and maybe others. Maybe this can be rethough so it won't be needed if the proposed notifications can provide enough usefull information. I think this would be quite a lot of new notifications what might however confuse other developers I am afraid.
I haven't yet looked into the XBMC source code I have only read python API docs and JSON API docs. I am not verry experienced programmer maybe it's why.
I would be happy to receive hints and / or corrections and ideas on how to proceed if someone else is interested.

I might be able to post what I have so far with a bit of description stating how to set it up the way it works over here if someone wishes to play with this.

Thanks and greetings

Peter

[byron27]

I'd like to play with what you've put together. I am not sure how to install it or how we'd get me set up with it but maybe we can figure that out over the next couple of days. I'm really excited about any possibility of access to to XBMC. Thanks for putting some work into it and figuring out some hurdles we have to deal with.

[pvagner]

@byron27 tonight I might be able to provide you with what I have so far. Are you on linux or on windows? Because I have done most of the testing on Windows and have briefly also tested linux.

Edit: oops, I'm sorry I am a bit late. I am trying to make it working initially on both the platforms. Basic functionality is there as I have just tested it on linux however I need to write at least basic instructions on how to set it up. Hopefully tommorow or soon enough I will manage to get into it.

[byron27]

Peter,
I'm on Windows so I'm right along with you on your platform. Thanks for everything!

[pvagner]

och. I have both windows and linux working. Unfortunatelly I will only be able to upload it tommorow as the latest working stuff is on my laptop and I've forgotten its charger in the office today. So no real real updates from today.

Edit: Source code of my attempts at xbmc-addons is now on github. Still writing some basic readme so you can set it up.

Edit2: fixed the link

Edit3:
Okay here are things you need to do in order to try out what I am working on.

0) If you are on Windows make sure you have NVDA working. If you wish you can choose to run portable version.
If you are on linux you will have to have speech-dispatcher installed, configure it to use the voice of your choice. It has to be running prior to running XBMC. So for now it is probably a good idea to run XBMC from within the desktop (if you are using orca with gnome or xfce it means you have working enviromment). Speech-dispatcher is packaged for most of the distros including debian, ubuntu, open suse, fedora and arch linux. I don't know the details about the others.
If you are using no screen reader and / or you have no sound from your speech synthesizer then you should get to install and configure one before proceeding further.
1) You will have to have XBMC installed if you are on either of these platforms windows or linux. On windows just download the installer on linux it is verry likelly your distro has XBMC packaged so install it using your prefered package manager. I am on arch linux so I know you can do sudo pacman -S xbmc in the terminal.
2) Launch XBMC eventhough you will have no access to its UI if you are blind. Then try to execute the following steps in the order as I am describing. If you find something different or unexpected please describe it so we can try figure out what might be different and suggest how to make it work for you.
- Press some arrow keys e.g. left, right, up or down. As you are pressing these you should hear clicking sound as you navigate over the controls on the XBMC home screen.
- Try to change to other window e.g. by pressing alt+tab and make sure you can access it using your screen reader. If you are on linux and you do only have single display configured you may have issues switching to other windows while XBMC is in full screen. You can cycle full screen and windowed modes by pressing \ [back slash] key. So if you are unable to switch to other window using alt+tab press the back slash key on your keyboard and retry pressing alt+tab again.
- Now try to change back to the XBMC window and verify you can still navigate XBMC home screen. Again you are looking for the click sound caused as you navigate over the controls.
- Finally try to exit XBMC by pressing ctrl+end on your keyboard.
3) Now you have to install my er... add-ons. If you can see good enough or you have a friend you can ask for help just launch your install of XBMC and navigate to Settings -> Add-ons and in there choose the entry saying something like Instal an add-on from a zip file. Then browse for a zip file downloaded from this post.
You have to install two add-ons. If you are on linux install both these in this order. script.module.speechdispatcher.zip and then script.voicemenus.zip. If you are on windows install the add-ons in this order script.module.nvda.zip and then script.voicemenus.zip.
If you were able to complete this step number 3 then do not execute the next step just read it so you are able to find out xbmc directory and continue with the step 5.
4) Here I assume you are unable to install XBMC Add-ons using the XBMC UI, so I will describe how you can extract them into a proper directory manually so XBMC will be able to load them.
Now you do need to find the XBMC addon directory. On windows press windows+r to display the run dialog and enter "%appdata%\xbmc" without the quotes. On linux navigate to the folder ~/.xbmc .
In the xbmc folder you will see some files and additional subfolders. Enter the addons folder. This is the folder you should extract the addon zip files into.
If you are on linux extract both files script.module.speechdispatcher.zip and script.voicemenus.zip into the addons folder.
If you are on windows extract both files script.module.nvda.zip and script.voicemenus.zip into the addons folder.
5) Now go to the xbmc folder as described at the beginning of the previous step. Inside the xbmc folder find the subfolder named userdata. In the userdata folder either find the keymaps subfolder or if the keymaps folder does not exist at this location then please create it.
Into the keymaps folder copy the file keyboard.xml. If you already have your user specific keyboard.xml in this location then you have to manually merge the two files. If you have ever edited keyboard.xml your-self you might be able to do this without further explanation.
6) Make sure your screen reader / speech-dispatcher is still working as it should and start / restart XBMC. When XBMC starts you should get voiced entries as you navigate over the controls in the home screen inside the various libraries, menus and settings.
There are still many controls where the voice menus don't work. These include dialog windows where you can only navigate over the buttons, media info entries where you can only hear names of media properties, settings where you can only hear control names and not their states and types.

Please ask for more details if you are unsure while performing some of these steps or if something does not work the way you expect it to.
Still if someone can give us more power to make this better I would be glad to receive hints, suggestions etc.

Downloads
script.module.speechdispatcher.zip
script.module.nvda.zip
script.voicemenus.zip
keyboard.xml

[byron27]

Oh wow, this is great! I didn't realize I had to do the part about the keymap but once I got it all installed I was able to play a video from one of my add-ons. I haven't explored XBMC in great detail with NVDA but I am planning on documenting all the areas that are not accessible. One thing I noticed is that if a video starts but is not fully buffered right away it'll stay in paused mode until I hit space once a significant amount of buffer is saved. This is normal operation with XBMC but there is no indication that my video is playing, it just does it. I wonder if that can be fixed by your script or if it's something XBMC Devs need to acknowledge. Also, add-ons that require filling out a captcha will be completely inaccessable so I wonder if there's a way to add an option for an audio captcha? Thanks for all of your hard work Peter, this is a great first step. I wonder if its allowed to make a version of the XBMC installer with this script and keymapping already included? If so, could somebody do that and make it available on the official XBMC site somewhere? Thanks!

[pvagner]

Hello,
Thank you for the test. I am verry new to XBMC and I haven't yet experienced the situation with buffering. I was only using youtube and some addons playing from some local slovak services.
Maybe I will be able to do some little improvements. However unfortunatelly currently I have no ideas on how to make more stuff in XBMC accessible.
Anyway if some one more knowledgeable enough is reading this I would be happy to receive hints on where to look for into improving all this. I don't think this is mature enough to be packaged into the installer or otherwise suitable for general use. Also what I am afraid of is that this is still verry limited.
Also I have found a bug which I am unable to figure out a fix for.
While overriding navigation controls e.g. left, right, up, down what this attempt at voice menus is doing keyboard navigation freezes in the add content dialog when arrowing in the list of scrapers.

[byron27]

I've told a few people on Twitter about XBMC and they've tried it and I've asked them to join the forum to report issues. I can send you a recording I made of XBMC in action if you can private message me your e-mail address. I discuss a few issues I ran into in this recording.

[pvagner]

Hello,
The most anoying issue I have identified from your podcast is the fact color codes and other formatting related characters are being spoken. I haven't came accross these while using it my-self. I have added simple regular expression matching to strip out the HTML tags from the text before speaking. Hopefully it will solve this issue.
Other things I have to work on:
*Try to convert TTS libraries into services so we can get speech interuption working under linux. On Windows with NVDA we are getting that for free.
*Explore how we can avoid sending commands and substitute these with keypresses simulation. Hopefully that might fix freezes in the add / change content dialog. Unfortunatelly that will require yet another update / tweaking of user keymap.
And even for the later time I am considering looking into making at least some of the addon dialogs somewhat accessible. If there's a need or a desire to do so we might be able to fork some addons to provide self voicing capabilities.

Also please note I have mapped my plugin to the F12 key without doing no action other than reporting the current status. So if you have pressed an enter key to start a video you can press the F12 key to hear whether it's still buffering or whether it has turned into a fullscreen view meaning the playback has just started. I think the video should not start paused at least I haven't yet seen that happening. While the video is buffering a progress bar is displayed. Pressing the enter key at that time cancels the buffering.

[ruuk]

Noticed this thread and did some testing on an idea. Here is a video of my results. The audio is bad for the screen capture, but sounds fine in real life:

http://www.youtube.com/watch?v=_yVv9dx88x0

This is done with a service addon running in the backround speaking the text with festival on ubuntu.

I think ideally with this approach, using a simplified XBMC skin would be best. (plus the skin could be tailored for those with partial blindness as well)

I am making the assumption that navigating with a remote and having the focused controls and window names spoken is sufficient.
Please feel free to correct me on my assumptions or to suggest or ask something.

[byron27]

Window names are fine but if there's an error the window name might not be enough to determine what's going on. I would think of it this way, any information you have to read with your eyeballs should somehow be accessible. That includes dialog boxes, keyboards including what letter you currently have selected and rather or not you typed it, edit fields, lower-right hand corner status updates, descriptions of media or add-on details, etc should have a way to read them. Of course I understand that we are a long way from all of that but a developer shouldn't take the attitude of "good enough" when considering accessibility. Also, I am personally against the idea of a custom skin unless it concerns the access of particular elements that otherwise would be inaccessible. The last thing you want to do is make XBMC users who are blind have to do all kinds of tinkering and tweaking just to have equal access to what sighted users already have. Most of the time blind users use the exact same UI as their sighted counterparts and I think that is the correct line of thinking. I'm only one voice but I can be pretty loud at times :-)

[nickr]

Window names are fine but if there's an error the window name might not be enough to determine what's going on. I would think of it this way, any information you have to read with your eyeballs should somehow be accessible. That includes dialog boxes, keyboards including what letter you currently have selected and rather or not you typed it, edit fields, lower-right hand corner status updates, descriptions of media or add-on details, etc should have a way to read them. Of course I understand that we are a long way from all of that but a developer shouldn't take the attitude of "good enough" when considering accessibility. Also, I am personally against the idea of a custom skin unless it concerns the access of particular elements that otherwise would be inaccessible. The last thing you want to do is make XBMC users who are blind have to do all kinds of tinkering and tweaking just to have equal access to what sighted users already have. Most of the time blind users use the exact same UI as their sighted counterparts and I think that is the correct line of thinking. I'm only one voice but I can be pretty loud at times :-)

Those are admirable goals but like most open source development you need to find a developer who wants to scratch this particular itch, or learn to be a developer

[ruuk]

Window names are fine but if there's an error the window name might not be enough to determine what's going on. I would think of it this way, any information you have to read with your eyeballs should somehow be accessible. That includes dialog boxes, keyboards including what letter you currently have selected and rather or not you typed it, edit fields, lower-right hand corner status updates, descriptions of media or add-on details, etc should have a way to read them. Of course I understand that we are a long way from all of that but a developer shouldn't take the attitude of "good enough" when considering accessibility. Also, I am personally against the idea of a custom skin unless it concerns the access of particular elements that otherwise would be inaccessible. The last thing you want to do is make XBMC users who are blind have to do all kinds of tinkering and tweaking just to have equal access to what sighted users already have. Most of the time blind users use the exact same UI as their sighted counterparts and I think that is the correct line of thinking. I'm only one voice but I can be pretty loud at times :-)

This was just a quick demo of the possibilities with an addon. I spent about 20 mins on it. Certainly anything on the screen can be spoken.

One of the strengths of XBMC is it's skinning capabilities. There are dozens of skins to satisfy different ways people use XBMC.

Correct me if I'm wrong, but I have to assume that the way a blind person interfaces with XBMC is different than the way a sighted person does. It makes sense to build a skin that makes usage easier.
There is no reason to believe that that means it has to have less features.

I'm not sure what you mean by

The last thing you want to do is make XBMC users who are blind have to do all kinds of tinkering and tweaking just to have equal access to what sighted users already have.

Why would making a more accessible skin make someone have to do more tinkering and tweaking. It would have the opposite effect.

I really have no experience with how a blind person would use XBMC. When I ask is something is good enough I am asking if it would be "good enough to build an interface out of", not if it's the bare minimum someone could tolerate.

My thinking was that with a skin addon and a service addon, XBMC could possibly transformed to fit the needs of blind people the same way skins and addons do it for sighted people who just want it to work differently for other reasons.

[byron27]

Blind and partially sighted people would utilize the keyboard to access XBMC the same way a sighted person would if they were using the arrow keys to navigate instead of clicking. We just need what ever the visual cursor has focus on to read aloud. There are exceptions to this such as the area where the description of an add-on would normally just get a glance by a sighted person. This is where a custom skin may become useful because you could tab to a read-only field and arrow up and down through the lines of text. Also, parts of the screen that are not navigable by the arrow keys may need a special keystroke or a call to grab focus from the read cursor temporarily to read things like the clock or the buffer or script loading status. Forgive me for jumping the gun regarding your "good enough" comment, I just want to be clear on what exactly constitutes proper accessibility. I would give Peter's mod a try with NVDA "a free screenreader" installed. If you can see it but you can't hear it and you think it's something you'd benefit from as a sighted user that probably means we'd gain use from it as well. Regarding Nick's comment about needing a developer who cares about scratching the itch of accessibility... I feel that this thread has drummed up a considerable amount of interest. While I cannot code, I am an XBMC and accessibility enthusiast. We are so much further along with what Peter and Nuuk have done than we were just a few short weeks ago. I apologize if I came off agitated earlier and it's not my intent to upset Nuuk or anything. I think Peter would appreciate as much help and guidance as possible in regard to this monumental task. Teamwork is what XBMC is all about and I am all for any leap forward in accessibility. I just worry about things being built or developed that don't "really" benefit to the end goal of making XBMC work for those without the ability to see their screens. People suggest things like voice recognition or simplified menus with limited features which actually limit and take away from our ability to equally access everything possible. Please understand my point of view and forgive me for not taking the time to understand yours. I'll try not to immediately freak out in the future :-).

[nickr]

Is there a reason a blind person would prefer a keyboard over a remote? It should work the same way with either input device.