DB - SQLLite/MySQL - Performance issues - Printable Version +- Kodi Community Forum (https://forum.kodi.tv) +-- Forum: Development (https://forum.kodi.tv/forumdisplay.php?fid=32) +--- Forum: Kodi Application (https://forum.kodi.tv/forumdisplay.php?fid=93) +--- Thread: DB - SQLLite/MySQL - Performance issues (/showthread.php?tid=199805) |
RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 (2014-07-15, 14:47)Tolriq Wrote: Export your nfs share only to your rpi IP. And you'll know if the data goes through the rpi or not I can see from the level of network traffic that the Pi isn't streaming anything - 323 bytes/second on TX/RX for a 1080p/DTS movie? That's some heavy-duty compression! Edit: Also, I can shut down the Pi and the movie/music continues playing on x86 (until the next track, that is, when playback stops with an error on x86). RE: DB - SQLLite/MySQL - Performance issues - Montellese - 2014-07-15 (2014-07-15, 14:31)Milhouse Wrote: I have configured the Pi as a UPnP "server" with the following options enabled: That shouldn't be the case. If so it is a bug. RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 (2014-07-15, 15:10)Montellese Wrote: That shouldn't be the case. If so it is a bug.Yeah sorry I updated the post - it may have been a glitch first time around. Now whenever I disable that second option on the Pi, the UPnP sources remain visible to x86. RE: DB - SQLLite/MySQL - Performance issues - Tolriq - 2014-07-15 (2014-07-15, 14:52)Milhouse Wrote:(2014-07-15, 14:47)Tolriq Wrote: Export your nfs share only to your rpi IP. And you'll know if the data goes through the rpi or not So you do realize that this is not normal. As explained earlier, if the NFS is only exported to rpi or using smb with password you will not be able to access the media. And this is not really wanted. There's no way to have some magic sources sharing like that. RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 (2014-07-15, 15:14)Tolriq Wrote: So you do realize that this is not normal. Hence, my confusion. (2014-07-15, 15:14)Tolriq Wrote: As explained earlier, if the NFS is only exported to rpi or using smb with password you will not be able to access the media. Yet, I can. Granted, the NFS share is not *only* exported to the Pi (it's visible to the entire subnet) but even when the x86 client knowns nothing about the NAS (in terms of sources.xml) it is still able to access the nfs:// media directly, because it's certainly not pulling this media via the Pi. (2014-07-15, 15:14)Tolriq Wrote: And this is not really wanted. Really, why not? It seems like a perfectly reasonable thing to me, as I certainly don't want data to be streamed from my NAS to a separate UPnP server (which might not have much network bandwidth, CPU grunt, or both) only for the data to then have to stream somewhere else in order to reach the UPnP client for playback. Now *that* is definitely not wanted. RE: DB - SQLLite/MySQL - Performance issues - m.savazzi - 2014-07-15 Again let me try to see if I can separate the issue: 1) we need a DB abstraction layer. This will allow to use local file, NAS MySQL, Cloud services as user likes. da-anda explained how to do it in his great post #26 (http://forum.xbmc.org/showthread.php?tid=199805&pid=1753147#pid1753147). This will also make the XBMC code better as there will be a unified way of asking for data from libraries. ODBC abstracts in the correct way all major data formats, they take care of the conversion between TEXT and VARCHAR of the different DBs. For users this will not be more difficult, at basic level, than server/username/password. For advanced users they could do what they like. 2) is there is an issue with paths the solution is simply to store the UNC (\\servername\pathname\path\filename.ext ) instead of the filepath (v:\movies\movie1.ext) , especially on network shares. Otherwise there should be a path manipulation procedure that "alligns" the path stored on the db to the local machine... but I prefer the UNC 3) for the thumbnails cache it can be local (if is a real cache) build upon the central DB. If we use OOBE in the DB we can store the thumbnails on the central DB and copy them locally for performace considerations 4) network traffic will be IN ANY CASE lower than uPNP as only relevant data will go back and forth the content will go directly form the source to destination As I wrorte before what I've read of the headless XBMC looks like a custom RDBMS server (this part) so I would not advise it as becomes a mess to manage in time. Usually I try to tackle those big changes a step at the time. The first one could be to replace the CDatabase, Video and Music libraries to use ODBC exactly like they are doing today. Then pictures and other db can be added. @da-anda: when you write "QOM I'm working with day by day:" means you're a super expert on this? It has been quite some years since I worked on DB design and code in c++ and I can be a little rustied RE: DB - SQLLite/MySQL - Performance issues - Tolriq - 2014-07-15 Well an nfs:// path is generic and can use loaded from any XBMC. But if the path is only exported to one XBMC then the second one accessing the UPnP libray will not be able to access the media : So useless UPnP sharing. Now take the example of smb, if you add a password protected share to your main XBMC, then 2 possible things : - Other XBMC host does not have access to the login password and as such cannot connect to the nas and no access to media. - Main XBMC does send the login and password in clear form and it's a major security flaw. Both case should not happens. And there's lot's of more example. So I'm sorry this is really not wanted and is not an normal usage of a library sharing feature. RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 (2014-07-15, 15:27)Tolriq Wrote: Well an nfs:// path is generic and can use loaded from any XBMC. OK, I can understand that when a client doesn't have access to the data then it can't stream it directly and must stream via the other device, but what I'm after is some clarity on this double-network bandwidth issue - the answer seems to be that it's not actually an issue (doesn't happen) at all whenever the client CAN access the data directly. Perhaps if I restricted the export to only the Pi, the x86 data would start streaming via the Pi - if so, that's actually understandable. Assuming it works that way of course - does the client fall back to UPnP streaming when it experiences an NFS (or SMB, or whatever) error? RE: DB - SQLLite/MySQL - Performance issues - Tolriq - 2014-07-15 (2014-07-15, 15:36)Milhouse Wrote: OK, I can understand that when a client doesn't have access to the data then it can't stream it directly and must stream via the other device, but what I'm after is some clarity on this double-network bandwidth issue - the answer seems to be that it's not actually an issue (doesn't happen) at all whenever the client CAN access the data directly. Well you should test to confirm but I doubt there's fallback. Anyway 80% + of users does use SMB with login password as NFS is more complicated for basic users, so will go through the main source or fail depending on current implementation. And the rest as da-anda says earlier will mount locally the share so same result (but well there's no easy solution for those) RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 I restricted the NFS export to only the Pi, and when trying to play media on the x86 I now get an NFS error: Code: 14:47:10 T:140542976878464 DEBUG: OnPlayMedia upnp://39c25aae-4226-1d78-fed0-2284f5a11214/videodb%3a%2f%2frecentlyaddedmovies%2f796/ Let me test SMB... RE: DB - SQLLite/MySQL - Performance issues - Milhouse - 2014-07-15 And SMB behaves the same as with NFS - the x86 client (which knows nothing about the SMB source, or the SMB username/password required to access the source) tries to stream direct from the SMB source, *not* via the UPnP server (the Pi). Code: 15:02:14 T:140173212690304 DEBUG: OnPlayMedia upnp://39c25aae-4226-1d78-fed0-2284f5a11214/videodb%3a%2f%2frecentlyaddedmovies%2f801/ Maybe it's a bug... RE: DB - SQLLite/MySQL - Performance issues - m.savazzi - 2014-07-15 hem... this is derailing a little on uPNP and similar. uPNP is not a solution for media library sharing and synchronization. Is a solution to stream data among devices. @Tolriq: if in the DB you store the correct UNC path In a network, the Universal Naming Convention (UNC) is a way to identify a shared file in a computer without having to specify (or know) the storage device it is on. but they are not related with uPNP M RE: DB - SQLLite/MySQL - Performance issues - Tolriq - 2014-07-15 UNC is only tied to SMB, you forgot all the other protocols XBMC support. (And you completely miss the authentication problem that was already discussed in this thread). I'm going to be a little but do not take it wrong (I add lot's of smiley to help). But while the initial post was a good idea, your actual knowledge about XBMC internals, how database and abstraction layer, ORM, QOM, DAL and anything debated here are really too minimal to make your follow ups credible. Your posts just looks like you do some Google on some terms, then post them here like you have found the Graal. RE: DB - SQLLite/MySQL - Performance issues - m.savazzi - 2014-07-15 sorry I'm focusing only on: how to make the DN better. Then there was an issue on the shared files... UNC is a solution for that and is easy to convert to NFS formats if SMB is not available In the libraries there are not "other protocols", but maybe I'm wrong. in any case we need to choose a file format name... and today we save in the database in the path table idPath strPath strContent strScraper strHash scanRecursive useFolderNames strSettings noUpdate exclude dateAdded 10 smb://NAS-HOMELAN/vDisk1/Film/movie 1/VIDEO_TS/ For the odbc I've found this super interesting: http://www.m2osw.com/odbcpp The license of the odbcpp library is the GPL v3.0. This means you are free to use this library in your open source projects as much as you like. For SQLite: http://www.ch-werner.de/sqliteodbc/ For MySQL: http://www.mysql.com/downloads/ RE: DB - SQLLite/MySQL - Performance issues - Tolriq - 2014-07-15 Well time to remove notification from this thread Library support way more than smb and UNC or whatever will not handle login and password unless plain text and it's not possible for obvious security reasons .... Furthermore Xbmc is gpl v2 and cannot use gpl v3 library without upgrading to v3 and from what I read this is not desired... So as I said earlier document yourself a little more before pasting Google As all desired actions needed to go further with the DB was already answered by the team. |