Project idea : Sync audio with remote devices (Play audio on remote controls)

  Thread Rating:
  • 3 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
[email protected] Offline
Junior Member
Posts: 29
Joined: May 2013
Reputation: 0
Location: France
Post: #1
Since XBMC was not one of the mentoring organizations last year, I couldn't work on my proposal, so here it is (again Tongue)
(I selected the most popular from last year)

Quote:PLAY AUDIO ON REMOTE DEVICES

Name: Mayeul Cantan
forum/e-mail: [email protected] / mayeul.cantan [at] gmail.com
Summary: Not everyone has wireless headphones. Bot almost everyone has a wifi-enabled device, as an android/iOS smartphone, and it would be great (in my opinion) to turn this one into a remote sink for audio.
How will I achieve this: There are multiple ways to achieve this : either send the whole soundtrack, and play it later, or stream it (with or without caching). Each of these methods needs investigation to determine whether it is achievable over a somewhat limited Wi-Fi network. Downsampling, downmix, and other format conversions needs to be taken care of too. First, I will look into the API used to synchronize playback position with the remote, and ensure it is precise enough. If it is not, implement my own version of it, based on NTP. Then, conduct tests to determine the best way to send audio to the remote player (probably using the official remote app). Third, implement sound playback in the remote, synchronized with the movie playing.
What will the project focus on: The most important aspects of the project are usability, from an end-user perspective, and from a developer perspective. The remote play option should be easy to use, work out of the box with popular remotes, and to achieve the latter, API must be stable, clean, easy to understand and implement.
Benefits: Users will be able to use any smartphone as a headphone, either to watch a movie from the couch without disturbing someone else, or to watch a movie simultaneously in different languages/with different soundtracks (think about descriptive soundtracks for the visually impaired, etc...). This would also allow the use of bluetooth (or other technology) sinks if the main player does not support those. There is a lot of possibilities.
Goals: At least implement audio mirroring, maybe soundtrack selection, and transcoding from the remote control.
What does it touch in Kodi: Network code/API, audio decoder/transcoder.
Requirements: C++, JSON-RPC ? (I would need to learn that), Java for the remote control (not particularly familiar with it either)
Possible mentors: Place to add possible mentors (Team-Kodi will add this).

What do you think ?
I had another idea, but I lost it writing this. If I find it again, I may post it back here.
EDIT : maybe a headless XBMC server option (like plex ?)

I need to improve my English skills, feel free to correct me ;-)
My GSOC 2014 post CANCELLED 2015 one
find quote
Solo0815 Offline
Posting Freak
Posts: 927
Joined: Sep 2004
Reputation: 9
Location: Germany
Post: #2
Quote:maybe a headless XBMC server option (like plex ?)
You could help improving this one, so it runs on headless-servers or NAS:
http://forum.kodi.tv/showthread.php?tid=212061

I for myself have OpenMediaVault running with the (old) Kodi-Library-Scanner.so

PC: Kubuntu 16.10 and Win7/10 --- Wetek Play: LibreELEC 7.95.x --- RPi3: 7.95.x --- NAS: OMV 2.x
find quote
Dark_Slayer Offline
Fan
Posts: 309
Joined: Nov 2012
Reputation: 9
Post: #3
I think that's a fun idea. My first thought was Yatse already does this, but then I realized the point was to split audio off to a wifi enabled device (Yatse streams a/v, not just audio). Anything that can get sound from the device can playback your kodi audio, wired headphones, bt headphones/set, etc

There are certainly a lot of Roku fans who would love this. I myself have been a little envious of the sound-through-your-remote-with-headphones option the roku has, especially since my wife and I work different schedules
find quote
[email protected] Offline
Junior Member
Posts: 29
Joined: May 2013
Reputation: 0
Location: France
Post: #4
(2015-03-02 09:11)Solo0815 Wrote:  
Quote:maybe a headless XBMC server option (like plex ?)
You could help improving this one, so it runs on headless-servers or NAS:
http://forum.kodi.tv/showthread.php?tid=212061

I for myself have OpenMediaVault running with the (old) Kodi-Library-Scanner.so

Thank you for pointing this out. I may work on It on my spare time, I really want this feature, but I think the first one outlined in my proposal would be useful to more users.

(2015-03-03 05:27)Dark_Slayer Wrote:  I think that's a fun idea. My first thought was Yatse already does this, but then I realized the point was to split audio off to a wifi enabled device (Yatse streams a/v, not just audio). Anything that can get sound from the device can playback your kodi audio, wired headphones, bt headphones/set, etc

There are certainly a lot of Roku fans who would love this. I myself have been a little envious of the sound-through-your-remote-with-headphones option the roku has, especially since my wife and I work different schedules
Yep, that's it, basically.
You may gather some more information on my GSOC 2014 post, even though the main idea is not the same.

I need to improve my English skills, feel free to correct me ;-)
My GSOC 2014 post CANCELLED 2015 one
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,232
Joined: Apr 2008
Reputation: 128
Location: Kansas
Post: #5
Hi Mayeul, as I mentioned last year, I quite like this idea. Is there any chance you could point to some of your C++ experience, so potential mentors could see what work you've done in the past to get a grasp on your current skill level?
find quote
RockerC Offline
Posting Freak
Posts: 1,485
Joined: May 2011
Reputation: 28
Post: #6
IMHO if they should be done right then it should be implemented as a feature inside the exciting (or next-generaion) official Kodi Remote apps for Android and iOS, and not as a new stand-alone app

http://forum.kodi.tv/forumdisplay.php?fid=129
http://forum.kodi.tv/forumdisplay.php?fid=193
find quote
Tolriq Offline
Donor
Posts: 3,043
Joined: Jun 2009
Reputation: 101
Location: France
Post: #7
As last year + 100000000 this is the most asked feature from Yatse users.

And surely all other remotes.

From previous Topfs answer in one thread, the main problem will be to have a generic way to have sync whatever user connection.
find quote
[email protected] Offline
Junior Member
Posts: 29
Joined: May 2013
Reputation: 0
Location: France
Post: #8
(2015-03-04 05:12)natethomas Wrote:  Hi Mayeul, as I mentioned last year, I quite like this idea. Is there any chance you could point to some of your C++ experience, so potential mentors could see what work you've done in the past to get a grasp on your current skill level?
I must warn you that my C++ skills are quite low compared to my C skills (I tend to program in C++ like I would in C). But that's for not being taught how to do it correctly, and I am quite confident I won't have any problem adopting Kodi's guidelines.
For reference, I did a C++ project about 2 years ago that you can find here : https://github.com/MayeulC/Chimie This project was made upon a professor request, and I didn't have much computer science at school at this time. Therefore, the project code isn't as clean as it should be. I can provide you with more C examples, if you would like.
I'll try to take a better look at the codebase, and maybe produce some patches, but I am really busy with my studies, I can't promise I'll be able to do it before next month.

(2015-03-04 18:34)RockerC Wrote:  IMHO if they should be done right then it should be implemented as a feature inside the exciting (or next-generaion) official Kodi Remote apps for Android and iOS, and not as a new stand-alone app

http://forum.kodi.tv/forumdisplay.php?fid=129
http://forum.kodi.tv/forumdisplay.php?fid=193
Of course, that's the plan. To provide a reference client implementation in the official app that others could follow.

(2015-03-04 20:32)Tolriq Wrote:  As last year + 100000000 this is the most asked feature from Yatse users.

And surely all other remotes.

From previous Topfs answer in one thread, the main problem will be to have a generic way to have sync whatever user connection.
Well, I'll make my best to make it happen this year :-)
About sync, my first move will be to have a look at the actual implementation of position sync in the remote API, and if it is not reliable enough, to extend it based on the NTP protocol. If not feasible (which I highly doubt), I'll code my own implementation of the NTP protocol to give the remote enough precision for audio sync. Assuming there is not too much divergence between the two system's clocks (which I think is safe), there should be no problem to sync audio, whatever connection is used.

I need to improve my English skills, feel free to correct me ;-)
My GSOC 2014 post CANCELLED 2015 one
find quote
topfs2 Offline
Team-Kodi Developer
Posts: 4,548
Joined: Dec 2007
Reputation: 17
Post: #9
(2015-03-04 20:32)Tolriq Wrote:  As last year + 100000000 this is the most asked feature from Yatse users.

And surely all other remotes.

From previous Topfs answer in one thread, the main problem will be to have a generic way to have sync whatever user connection.

And I added a very generic trial code for it using JSON RPC. It was based on zeromq which I tbh think we don't need for it, so one could make a PR based on mine but using TCP jsonrpc instead. Which would be awesome.

If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

[Image: badge.gif]

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
find quote
zag Offline
Retired Team-Kodi Member
Posts: 4,006
Joined: Oct 2007
Reputation: 75
Location: UK
Post: #10
Very nice feature, I look forward to it if you are selected.
find quote
da-anda Offline
Team-Kodi Member
Posts: 5,777
Joined: Jun 2009
Reputation: 77
Location: germany
Post: #11
I heared that many of the consoles (incl the upcoming nVidia Shield) are using bluetooth to broadcast the audio to their remotes and controllers - but I think this is not possible in our case because of all the different platforms and BT drivers
find quote
RockerC Offline
Posting Freak
Posts: 1,485
Joined: May 2011
Reputation: 28
Post: #12
(2015-03-04 23:31)topfs2 Wrote:  
(2015-03-04 20:32)Tolriq Wrote:  As last year + 100000000 this is the most asked feature from Yatse users.

And surely all other remotes.

From previous Topfs answer in one thread, the main problem will be to have a generic way to have sync whatever user connection.
And I added a very generic trial code for it using JSON RPC. It was based on zeromq which I tbh think we don't need for it, so one could make a PR based on mine but using TCP jsonrpc instead. Which would be awesome.
is this the your branch for that work? https://github.com/topfs2/xbmc/tree/zeromq

Everything contained within that single branch or?

(2015-03-05 11:54)da-anda Wrote:  I heared that many of the consoles (incl the upcoming nVidia Shield) are using bluetooth to broadcast the audio to their remotes and controllers - but I think this is not possible in our case because of all the different platforms and BT drivers
Don't all modern Bluetooth remotes just make use the same AVRCP and A2DP profiles standard?

https://developer.bluetooth.org/Technolo...AVRCP.aspx
https://www.bluetooth.org/docman/handler..._id=260861

The AVRCP profile can be used in concert with A2DP, GAVDP, and/or VDP profile standards

http://en.wikipedia.org/wiki/List_of_Blu...28AVRCP.29
http://en.wikipedia.org/wiki/List_of_Blu....28A2DP.29
http://en.wikipedia.org/wiki/List_of_Blu...28GAVDP.29
http://en.wikipedia.org/wiki/List_of_Blu..._.28VDP.29

[Image: AVRCP.jpg]
(This post was last modified: 2015-03-05 12:53 by RockerC.)
find quote
natethomas Offline
XBMC Chief and Kodi Project Manager
Posts: 6,232
Joined: Apr 2008
Reputation: 128
Location: Kansas
Post: #13
The shield controller connects via wifi direct rather than Bluetooth, so I'm presuming that audio is being transported over that wifi connection.

Edit: I'm reviewing the conversation and I may be missing why Bluetooth is being brought up. The proposal appears to be about sending over wifi as well.
(This post was last modified: 2015-03-05 17:09 by natethomas.)
find quote
topfs2 Offline
Team-Kodi Developer
Posts: 4,548
Joined: Dec 2007
Reputation: 17
Post: #14
(2015-03-05 12:50)RockerC Wrote:  
(2015-03-04 23:31)topfs2 Wrote:  
(2015-03-04 20:32)Tolriq Wrote:  As last year + 100000000 this is the most asked feature from Yatse users.

And surely all other remotes.

From previous Topfs answer in one thread, the main problem will be to have a generic way to have sync whatever user connection.
And I added a very generic trial code for it using JSON RPC. It was based on zeromq which I tbh think we don't need for it, so one could make a PR based on mine but using TCP jsonrpc instead. Which would be awesome.
is this the your branch for that work? https://github.com/topfs2/xbmc/tree/zeromq

Everything contained within that single branch or?

Oooops, I just remember I never pushed it since I wanted to do more on it (check viability etc) before I made any noise Smile
However, I can put it up as soon as I am near my home computer. I mostly started on the abstraction, highlighting were it could be hooked in. And I got it working sending audio from kodi to another process through it. The other way around didnt work (possibly due to me doing something wrong with AE).

So in general I think it would be a good starting point for this project, and there is still lots and lots to do (I didn't have time to investigate the actual latency calculations, which is needed for sending audio over eth or wifi). But it should be very doable I think. And a feature I would personally love to have in kodi Smile

If you have problems please read this before posting

Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.

[Image: badge.gif]

"Well Im gonna download the code and look at it a bit but I'm certainly not a really good C/C++ programer but I'd help as much as I can, I mostly write in C#."
find quote
RockerC Offline
Posting Freak
Posts: 1,485
Joined: May 2011
Reputation: 28
Post: #15
(2015-03-05 17:07)natethomas Wrote:  The shield controller connects via wifi direct rather than Bluetooth, so I'm presuming that audio is being transported over that wifi connection.
We are talking in regards to the Shield Remote, not the Shield Controller. Yes the Shield Controller uses WiFi, but the Shield Remote looks use Bluetooth

http://shield.nvidia.com/store/remote

Plug in headphones to play games or watch movies without disturbing anyone and crank it up with volume control.

[Image: shield-store-920x540_0009s_0003_remote-gallery_4_0.jpg]


Take control with Bluetooth from anywhere in the room, even without line of site to your SHIELD device.

[Image: shield-store-920x540_0009s_0001_remote-gallery_2_0.jpg]


(2015-03-05 17:07)natethomas Wrote:  Edit: I'm reviewing the conversation and I may be missing why Bluetooth is being brought up. The proposal appears to be about sending over wifi as well.
It was only mentioned so that it is kept in mind just in case, so that if the implementation is properly abstracted then the code could perhaps be reused for these types of Bluetooth remotes too.
(This post was last modified: 2015-03-05 19:08 by RockerC.)
find quote
Post Reply