How to rebuild central MySQL database
#1
I have been running Kodi using a centralized database on a MariaDB docker on Unraid with 3 PC "clients" and 4 Fire TV Cubes running Kodi.  This setup has been running flawlessly for over 5 years. Late last year, I let MariaDB on the Unraid server update to 10.5.13.  Immediately, the Fire TV Kodi installs would crash with database errors.  I have tried various remedies suggested on this forum and some fixed it for a while, but each time, Kodi would run once and then after any change in the database (such as marking a movie "watched") they would start crashing again.

So, I have given up trying to fix my current DBMS and database and I just want to start over.
However, I have some questions about the official method (https://kodi.wiki/view/MySQL/Setting_up_Kodi) and my particular setup.

Here are the steps as listed in the wiki.

1. Make files accessible over the network
This is already done.  My media files are on SMB shares on the Unraid server.

2. Exporting
Since this assumes that you are moving from a computer-hosted library to a MySQL server, if I export the library now, I assume that the "Library" that it is "exporting" is actually in the MySQL database.  Is this correct? If not, is there some way to import the database back into a local Library?

This method works by creating .nfo files in the media directories.  This could be a problem.  I store movies as full DVD/Bluray rips, in folders named Movie_Name(date).  But, I have also manually created many .nfo files for special purposes (e.g. creating a "set" of a 2-DVD set, making fake "TV episodes" for my Youtube dowloads, etc.)  Will the "Export Library" command write over these .nfo files?  If so, is there any way to save the ones that I already have?

I would then delete (or actually save and move) the current MariaDB database files so that I can allow Kodi to create a new database.

 3. MySQL and advancedsettings.xml
I would set this up as if I were doing it for the first time.  Actually, I guess it is already set up properly.

4. Importing
If I understand how this works, it would read all the .nfo files created during the library export and then create the MySQL tables needed to form the database from them.
I assume that it would read all my manually-created .nfo files, as well?

5. Adding new Kodi devices to the MySQL setup
I think I don't have to do anything here since the PCs and Fire TVs are already set up?
One question though, it says

"Copy from the same userdata root folder the files, such as "favourites.xml", "sources.xml", "mediasources.xml", "passwords.xml" (if present)."

I actually never did this before.  What use are the "sources.xml" and "mediasource.xml" files to the clients?  Isn't all this information in the database itself?
Anyway, I have never copied these files to the clients and they still worked perfectly.

So that's my plan.  Any help would be greatly appreciated.
Thanks.
Current Kodi: 20.2, Mariadb 10.2 running on Unraid server.
Reply
#2
(2022-03-15, 19:46)CaptainTivo Wrote: So, I have given up trying to fix my current DBMS and database and I just want to start over.

With which database server? I'm a lil bit surprised that someone using setups like Docker are not also using or even knowing of MySQL database sql dumps. It's a form of backup that could save you lots of time.

BTW, rescraping your media collection even with straightforward exported nfo files (same Kodi version) is not without problems as I found out myself today. Using MariaDB 10.5.12 on RPiOS Bullseye. Some 10% of my TVshows were not scanned properly, not a clue yet as to why. And things went pretty sluggish too (I blame the scraper).
Reply
#3
(2022-03-15, 20:37)Klojum Wrote:
(2022-03-15, 19:46)CaptainTivo Wrote: So, I have given up trying to fix my current DBMS and database and I just want to start over.

With which database server? I'm a lil bit surprised that someone using setups like Docker are not also using or even knowing of MySQL database sql dumps. It's a form of backup that could save you lots of time.

BTW, rescraping your media collection even with straightforward exported nfo files (same Kodi version) is not without problems as I found out myself today. Using MariaDB 10.5.12 on RPiOS Bullseye. Some 10% of my TVshows were not scanned properly, not a clue yet as to why. And things went pretty sluggish too (I blame the scraper).


Well, I have to admit to ignorance of using databases.  I simply followed the instructions in the wiki years ago and it worked perfectly.  I have learned a bit more since this problem started and I made both bit-for-bit (by simply copying all of the files that MariaDB uses) and MySQL dumps.  The problem is, that I think the database has been corrupted.  The MariaDB update in December started this and one of the attempts to fix it was to learn that I should have executed a mysql_upgrade command.  I thought that had fixed it because the Android boxes worked again.  But the next time I added a movie (and thus, modified the database) they started crashing again.

Here is a sample of the errors in the Kodi log file from one of the Fire TV boxes:
 
Code:
ERROR: GetDirectory - Error getting videodb://tvshows/studios/
   ERROR: SQL: [MyVideos116] An unknown error occurred
             Query: SELECT type,url FROM art WHERE media_id=43 AND media_type='tvshow'
   ERROR: GetArtForItem(43) failed
   ERROR: SQL: Missing result set!
   ERROR: GetArtForItem(142) failed
   ERROR: SQL: [MyVideos116] An unknown error occurred
             Query: SELECT type,url FROM art WHERE media_id=216 AND media_type='tvshow'
   ERROR: GetArtForItem(216) failed

The weird thing is, the Kodi apps on my PCs work fine.  This would imply that the database client on the Fire TV's has a compatibility problem.
The thing is, I have been chasing this for more than a month and all the various solutions suggested have not worked so I am at a dead end.

Also, I really don't want to re-scrape my movies and tv shows.  What I want is to build a database from the already-scraped info.  I thought that is what I was doing by exporting the library and then letting the database server suck that data into a database.  If not, then, yes, I don't know what I'm doing.
Hence the call for help :-)
Current Kodi: 20.2, Mariadb 10.2 running on Unraid server.
Reply
#4
(2022-03-15, 21:49)CaptainTivo Wrote: Well, I have to admit to ignorance of using databases. 

I also don't have all the answers, databases are not part of my daily activities. But I can see over time that various default MariaDB setups have been causing different strange problems. The so-called claimed MySQL-compatibility seems it's not always 100%. And it's also feasible that Docker itself (I still have to dive into that world) is having some part in the troubles you're having.

As far as mysqldump goes for backups, its workings are available in a Wiki page via scripts I wrote. The Linux variant is working 100% AFAICT, but the Windows script has its own challenges. That one is still under construction.

My "production server" at home was first Ubuntu Server 12.04 thru 18.04 now (command line only) running MySQL 5.x, it seems to have had no real hiccups over the years. MariaDB looks to be a new bag of tricks with each of their stable releases.

(2022-03-15, 21:49)CaptainTivo Wrote: ERROR: SQL: [MyVideos116] An unknown error occurred
             Query: SELECT type,url FROM art WHERE media_id=43 AND media_type='tvshow'
   ERROR: GetArtForItem(43) failed
   ERROR: SQL: Missing result set!
This is one of the easiest SQL queries for a database table. If that is not working, then something fundamental is wrong. Perhaps Kodi's log cannot find the reason, but MariaDB's own log file should be able to tell you more on what is going wrong.

(2022-03-15, 21:49)CaptainTivo Wrote: Also, I really don't want to re-scrape my movies and tv shows.  What I want is to build a database from the already-scraped info.  I thought that is what I was doing by exporting the library and then letting the database server suck that data into a database.

That has always been the go-to action for rescraping a library. Exporting to nfo files, and have Kodi read them into a new database. But as said, my experience yesterday shook that up a bit (again, MariaDB could be the culprit but more investigation is required) as some 10% of TVshows got scraped only half. All the episode info got in fine, but main tv show details were sometimes completely blank (NULL fields). Never had that with a MySQL database. Luckily, the manual refresh of each of those TVshows went okay.

Could be that Kodi 19.4 suddenly has been introduced with an export/import bug... Anyway, more bug hunting is necessary.
Reply
#5
These errors with MariaDB/MySQL have cropped up before and been solved, for example see https://forum.kodi.tv/showthread.php?tid=357217
It that doesn't help serach for "An unknown error occurred" and check the other threads.
Reply
#6
(2022-03-16, 15:13)DaveBlake Wrote: These errors with MariaDB/MySQL have cropped up before and been solved, for example see https://forum.kodi.tv/showthread.php?tid=357217
It that doesn't help serach for "An unknown error occurred" and check the other threads.

Yes, I found that thread, too.  I edited custom.cnf to raise the sort_buffer_size.  After that, the FireTV Kodi installs worked -- ONCE.  That is, they launched and all appeared normal.  I exited and launched again and Kodi hung and/or crashed.  I upped the buffer size to 30M and the same behavior happened again.  I also ran the mysql_update command and it completed with OK messages.  Again, same behavior.  Kodi would run and then next time crash.  Finally, I realized that each time I ran Kodi and , for example, watched a movie or TV espisode, it was updating the database, at the very least with the watched information.  My hypothesis is that it is this database update this is causing the problem.  I don't know if the database is corrupted in some way that causes this or if the MySQL client on the Android boxes is causing it.

What I do know is that after searching for a couple of months, I have found very few complaints of this nature.  So, either there are very few people using Android for their Kodi clients and using a central database (I think that's very unlikely) or, I have a problem with my particular system.

The central fact that confounds me is this: my PC Kodi clients have NEVER had a problem through all this, only the Android installs have a problem.  This would imply that the problem is with the Fire TV boxes.  Someone suggested that the MySQL client code for Android was different from that on the PC.  I don't know how this is implemented in Kodi.  Does Kodi use the same database code on all platforms or do they use local libraries, or what?  If they use Android libraries, then maybe they are incompatible with MariaDB 10.3.15? 
Can someone suggest who/where I should ask these questions?
Thanks.
Current Kodi: 20.2, Mariadb 10.2 running on Unraid server.
Reply
#7
Yes you have a problem with your particular system, and that problem is not in Kodi code.

Kodi uses the MariaDB Connector/C  client library to connect MariaDB and MySQL databases. This external lower level library is probably implemented differently for different platforms. You could try asking for MariaDB support, but I suspect that they will bounce you straight back to Kodi because all you can show them is a Kodi error log. My hope was that running mysql_update would fix it (since it has for others), sorry I don't have any other help to offer.

To rule out basic incompatibility you could create a new MariaDB database just for testing and only point your FireTV at it. If you can get that working then the issue is more likely a corrupted db, and at least recreating all your libraries (using SQL export/import might be quickest) would be a way forwards.
Reply

Logout Mark Read Team Forum Stats Members Help
How to rebuild central MySQL database0