Guest - Testers are needed for the reworked CDateTime core component. See... https://forum.kodi.tv/showthread.php?tid=378981 (September 29) x
Synology NAS upgrade and Maria DB backup
#1
I've been running a Synology NAS with 5x8TB drives with RAID 6 for about 5 years now.
I'm out of space and need to upgrade.
I'm adding 2x14TB to the existing 5x8 and want to change to SHR1 RAID.

I can't just change RAIDs like that so I have to destroy my existing 5x8 volume and start over.
I installed the 2x14 as a non RAID and copied all the data to it as a temp backup... the NAS isn't my primary storage... all my media is on my PC as well as externals... this is just for faster data transfer later.

So the plan is to delete the 5x8 volume and reinstall it as SHR1 then I will copy all the media back from my 2x14 volume ... then delete the 2x14 volume and add it to the 5x8... with the redundancy I should have roughly 50TB of usable storage.

What I don't know how to do is backup any NAS settings and more importantly my MariaDB database for Kodi.
Can someone point me in the right direction here... I read that Hyperbackup on my NAS should be able to back it up but I see nothing in this aside from simply backing up regular data.
Reply
#2
50 TB ... How do you start to choose what to watch? Laugh

Saving the settings of your NAS is a Synology problem, not a Kodi problem.

Backups of your MariaDB database should be fairly easy using the built-in PhpMyadmin web application, or an external MySQL client such as MySQL workbench, or perhaps a MariaDB-based client. Heck, you can also make backups via an Telnet/SSH connection and start the backup from the command line if that has your preference.

Make sure that when doing the database dump, both the structure and data is included, and that sql commands for tables being (re)created are also added. Basically, you'll see DROP commands of tables before the CREATE commands in the SQLdump, in case you want to restore a backup onto an existing/running database.
Reply
#3
(2021-02-26, 05:29)Klojum Wrote: 50 TB ... How do you start to choose what to watch? Laugh

Saving the settings of your NAS is a Synology problem, not a Kodi problem.

Backups of your MariaDB database should be fairly easy using the built-in PhpMyadmin web application, or an external MySQL client such as MySQL workbench, or perhaps a MariaDB-based client. Heck, you can also make backups via an Telnet/SSH connection and start the backup from the command line if that has your preference.

Make sure that when doing the database dump, both the structure and data is included, and that sql commands for tables being (re)created are also added. Basically, you'll see DROP commands of tables before the CREATE commands in the SQLdump, in case you want to restore a backup onto an existing/running database.
Haha, sometimes the more you have the harder it is to choose! But seriously though things keep getting bigger... example, if I rip my new 4K Lord of the Rings/Hobbit trilogy discs without compression that would take nearly a TB alone! But I would compress them with Handbrake for example.

And yeah, I know this is more of a Synology question but I thought I would try here...

As for the rest of what you said... I have no clue what you're talking about. When I set this up 5 years ago it was the first and only time I've ever set up a NAS or database like that and I don't remember much.

So when I login to phpMyAdmin do I just go to the export tab... lot of options there. I do see the 'structure and data' checkbox you mentioned but also many others that I'm not sure what to check.
Also I'm not sure about there being MariaDB 5 and a 10... is it using both?... do I need to make 2 different backups?
Reply
#4
(2021-02-26, 14:41)huttdes Wrote: As for the rest of what you said... I have no clue what you're talking about.

It's not always easy to know/see what forum users are capable of when it comes to 'computer techniques'.

Structure and data = both the schema/layout of the database tables (what to store and how) as well as the data itself. All other default settings should be okay. The export file is normally one .SQL file.

MariaDB 5 & 10 = simple the old and the new versions of MariaDB. Currently there is v10.5.x I think. You should be able to get that info from your running NAS MariaDB server app. One backup from the running MariaDB server will do.
Reply
#5
There's a free MySQL / MariaDB app out there that I use to do exactly what your attempting. It's called HeidiSQL:

https://www.heidisql.com/

It's reasonably easy to use, and is much more user-intuitive than SQL Workbench (if memory serves). My Dbs are running on a QNAP NAS, so I'm in much the same sort of scenario as you.

Connect to the IP and port that the NAS SQL server is running on using your ADMIN credentials (You may need to allow remote access to the SQL server, in the NAS GUI in order for this to work). You will want to save the credentials in HeidiSQL for future use.


1. Backing Up The Database(s):

With the required database selected (in the left pane), right click on the database you want to export with the right mouse button or context menu key on the keyboard, and select the option "Export Database as SQL"...

Image


You should then see a new dialog window pop-up with a number of export options...

Image

Once your happy with your selections (and target file name/path) just click on the "Export" button to the bottom-right of the Window, and the database and it's contents will be exported to disk. You can export multiple (selected) databases to a single SQL file, or you can do as I do, and simply repeat the process for each of the two databases Kodi uses.

Note: You may wish to play about with the "Max INSERT Size" setting to avoid fragmented queries when restoring the database, but I think this will largely be dictated by the SQL server settings for Max Inserts, and the amount of (free) RAM your NAS is able to use. Don't quote me on that though, as this is supposition on my part.

EDIT: I'm changing my advice / stance on this setting, due to an issue with truncated tables / data I myself experienced recently (i.e: last weekend) trying to restore a database backup. If your doing this procedure for the first time, and are backing up a fully-working set of databases, from a Kodi install with no (major) issues, and want a 'foundation' on which to build / test, knowing you have a rock-solid backup to go back to and not have to re-scrape your entire library again (as I had to do), then keep the 'Max INSERT Size' setting in HeidiSQL at '0' (Single INSERTS) for your first backup of both databases, and select the output file from the pull-down menu below it, to 'ZIP compressed .sql file' instead.

Be aware this will take MUCH longer to export / import, but will avoid HeidiSQL having to guess where to split queries that are too large to send all in one go. Because in my experience it tends to get the split-points all wrong with regards Kodi's databases, and the imported data is either malformed, truncated or at worst unusable. Especially for TV shows where the number of seasons are large (8-10+). Also, the LAN is the weakest link, and sending large payloads of data to a low-power device like a NAS is at best, risky. As always, YMMV.


2. Restoring the Database(s):

To restore an exported database, you have two options from the "File" menu. "Load SQL File (CTRL-O)" and "Run SQL File". It's highly likely that your NAS will baulk on the SQL file if run directly, so you'll probably want to use the "Load SQL File" option, then run the resulting SQL queries that were loaded by going to the "Query" tab and pressing the blue "Play" icon in the toolbar, and thereby apply the changes to the database server...

Image

You should see the progress of the import on the fly, same as occurs with the initial export process. Once the loaded SQL has been run, check for any errors in the log/status window at the bottom, and you should be good to go.


3. Database Optimization (Experimental!):

By default, Kodi uses the MyISAM database engine for it's databases. Which on a fully-fledged system, is probably OK. But since most consumer NAS devices use low-power sillicon (Atoms, Celerons, etc.), once the databases start to grow in size, scraping and library updates can become painfully slow, depending on the size of your media libraries. One tweak that can be performed, which I can confirm has a noticable speed improvement, is converting the database table engine from MyISAM to InnoDB.

For Pros and Cons, just Google MyISAM vs. InnoDB. There may be a specific reason why Kodi defaults to MyISAM, but I have tried this tweak and not noticed any issues during day-to-day running, with one caveat. A converted database will not be processed correctly by Kodi during a version upgrade. I tried it and it failed, at least on my QNAP(s). So you will probably need to revert the change prior to migrating between Kodi versions, which may come with added issues. But that's why you backed up your databases to begin with, right?

Edit: The above caveat only applies, generally, to major-release version updates where the database version has been updated and thus both databases need to be "migrated". Point releases, where there are no database structure / version changes will not require reversion back to MyISAM tables in order to apply the point release (e.g: 19.0 to 19.1 if/when it is released).

Edit2: I've determined that Kodi doesn't 'choose' a database engine type when creating the databases fresh. It simply uses the 'Server Default' setting (as defined in the my.cnf / my.ini or mariadb equivalent). Therfore you may want to change the default to 'InnoDB', but be aware this may break operation of any newly installed addons that may rely on the database engine type being 'MyISAM'. It should circumvent the need to switch back database engines prior to installing major new versions of Kodi however. I'm now convinced the failure I had, was due to it trying to write from a set of converted InnoDB databases to a set of newly-created MyISAM databases and fell on it's face for some reason. As usual, proceed with caution.

Wink

To perform the switch, with the target database selected, open the "Tools" menu from the main window, and select the option "Maintenance". In the dialog window that opens, select the "Bulk Table Editor" tab, and select the checkbox for the "Change Table Engine" parameter. Open the drop-down list to the right and select "InnoDB" as the table engine. It is possible to expand the database tree and select each table in the database prior (as this procedure won't work on "views" or "triggers", only "tables"), but as HeidiSQL simply ignores "views" and "triggers" anyway, it's not an essential step as far as I can tell...

[b]Image[/b]

Press the "Update" button to the bottom-right of the window and HeidiSQL will then go through all the tables in the selected database(s) and convert them from MyISAM to InnoDB. Do not perform this switch on any of the default databases the NAS ships with, as there are some databases on the SQL server that must remain as MyISAM for the server to function correctly.

As alluded to above, I'm not aware of any pitfalls from making this alteration, and if the devs know of a reason why this shouldn't be done, i'll happily edit the post and remove this section. But doing the change literally HALVED my library update / scraping time on my TS453-PRO with both a raw (i.e: empty) database and updates to a heavily-laden (i.e: fully loaded) database. I'm no SQL whiz/expert, so use this tip with due caution.

Smile

Dan / Gib.
Image
Reply
#6
Thanks for the replies guys... and I'll look into that software, Dan.

Got a quick question... should I have so many different MyMusic and MyVideos listings?
I'm thinking some things are still there from day one when I may have been testing things out... like the test one for example.
Do you think all of these are being used and if not is there a way to tell which are and should I delete the unused ones?

If I click on each database the number of tvshows/movies goes up the higher the number so did it just save some old databases for some reason? How would I delete these if they are not necessary. MyVideos 119 reflects my current number of movies and TVshows

Also it appears I'm using MariaDB 5... should I be using 10? What is the difference?

Image
Reply
#7
(2021-02-26, 22:23)huttdes Wrote: Thanks for the replies guys... and I'll look into that software, Dan.

Got a quick question... should I have so many different MyMusic and MyVideos listings?
I'm thinking some things are still there from day one when I may have been testing things out... like the test one for example.
Do you think all of these are being used and if not is there a way to tell which are and should I delete the unused ones?

If I click on each database the number of tvshows/movies goes up the higher the number so did it just save some old databases for some reason? How would I delete these if they are not necessary. MyVideos 119 reflects my current number of movies and TVshows

Also it appears I'm using MariaDB 5... should I be using 10? What is the difference?

Image

MariaDB 10 is a more recent version, with a better feature set than MariaDB 5. But like me, you'll be likely reliant on if (rather than when) Synology decide to upgrade the version your NAS comes installed with. If Synology are anything like QNAP, I wouldn't hold your breath. You could install a 3rd-party SQL server via a community add-on, perhaps, to facilitate the upgrade and use that over the one supplied by default, but that comes with it's own set of (potential) issues. While MariaDB 5 continues to be supported in Kodi, i'm planning on leaving well alone in my case.

As the old adage goes: "If it's not broke, don't fix it."

Kodi does not delete the old database on a Kodi upgrade, it migrates the old one to a new version, leaving the old one intact. Upon startup, Kodi checks for the existence of the most recent database version supported for each type of media (videos and music) and if missing, works backwards in version number to the earliest it can migrate. If no earlier version databases are found, it attempts to create new (blank) databases, using the credentials specified in your advancedsettings.xml file. Kodi will only use the latest version of either database at any one time. If the existing databases are from an earlier install, Kodi will auto-migrate them on first-run (hence the delay at startup on a new version of Kodi), but retains the originals, which you will need to manually remove once your sure the new setup is OK, and you don't need to roll-back to an earlier version of Kodi.

The "119" tacked onto the end of the video database name relates to the version number of said database. The same is true for the music database (82). As you can see from my screenshots, I modified my database names (as specified in the advancedsettings.xml file), to include "_v" at the end of the database name, so I would remember that it refers to the version number. I also did it so that I would be able to easily identify the SQL export version should the scenario arise where I have multiple backups of the same database but in different versions.

In the advancedsettings.xml file:
Code:
<videodatabase>
    <name>kodi_video_v</name>
</videodatabase>

The text between the two "name" tags can take any form you wish, but the version number will always be appended to the end, so Kodi can identify the version of the database at startup and on upgrades.
Image
Reply
#8
One more thing. If you decide to test out the optimisation conversion, you might want to check out the following thread for tweaks to make to the "my.cnf" config file for the MySQL / MariaDB server to further optimize performance. This will require SSH access to the NAS (which can be facilitated with the free WinSCP / PuTTy combo of apps) but is not recommended for those not confident messing with the internal software / OS of their NAS.

https://forum.kodi.tv/showthread.php?tid=329046

On QNAP NAS devices, the file is located at /etc/.config/my.cnf (which is symlinked to /mnt/HDA_ROOT/.config). It may be in the same (or similar) location on a Synology NAS, since they both run Linux. YMMV.

Dan / Gib.
Image
Reply
#9
(2021-02-26, 23:49)gibxxi Wrote: One more thing. If you decide to test out the optimisation conversion, you might want to check out the following thread for tweaks to make to the "my.cnf" config file for the MySQL / MariaDB server to further optimize performance. This will require SSH access to the NAS (which can be facilitated with the free WinSCP / PuTTy combo of apps) but is not recommended for those not confident messing with the internal software / OS of their NAS.

https://forum.kodi.tv/showthread.php?tid=329046

On QNAP NAS devices, the file is located at /etc/config/my.cnf (which is symlinked to /mnt/HDA_ROOT/.config). It may be in the same (or similar) location on a Synology NAS, since they both run Linux. YMMV.

Dan / Gib.

Thanks for all the advice... Just getting home now so I will dive more into this over the weekend!

PuTTy!! That was the software I was trying to remember... Yes, I used that to access the NAS on my initial setup...

Thanks again, and like I said I'll get more into this in the next couple days.

I know I'll get it all figured out... I got it running 6 years ago with minimal issues but I'm spoiled now and don't want my NAS down for a week if I run into problems :-(
Reply
#10
(2021-02-27, 01:55)huttdes Wrote:
(2021-02-26, 23:49)gibxxi Wrote: One more thing. If you decide to test out the optimisation conversion, you might want to check out the following thread for tweaks to make to the "my.cnf" config file for the MySQL / MariaDB server to further optimize performance. This will require SSH access to the NAS (which can be facilitated with the free WinSCP / PuTTy combo of apps) but is not recommended for those not confident messing with the internal software / OS of their NAS.

https://forum.kodi.tv/showthread.php?tid=329046

On QNAP NAS devices, the file is located at /etc/config/my.cnf (which is symlinked to /mnt/HDA_ROOT/.config). It may be in the same (or similar) location on a Synology NAS, since they both run Linux. YMMV.

Dan / Gib.

Thanks for all the advice... Just getting home now so I will dive more into this over the weekend!

PuTTy!! That was the software I was trying to remember... Yes, I used that to access the NAS on my initial setup...

Thanks again, and like I said I'll get more into this in the next couple days.

I know I'll get it all figured out... I got it running 6 years ago with minimal issues but I'm spoiled now and don't want my NAS down for a week if I run into problems :-(

Hi huttdes,

Here's a tip that will work in HeidiSQL when exporting your SQL files, relating to file naming. Once you have your export path set (I export mine back to a share on the NASes) you can tack the following onto the end of the path to represent the exported filename:

Code:
%db - %d-%m-%y.sql

This will name the exported SQL file as: "Database Name" - "day" - "month" - "year".sql

This allows you to know the date you made the backup, and the name of the database being exported. It also becomes useful if, like me, you export the music and video databases separately, that you won't need to keep editing the path / filename field in HeidiSQL before doing each export, because the file naming is dynamic, preventing accidental overwrites.

Smile

Dan / Gib.
Image
Reply
#11
[Double Post]
Image
Reply
#12
(2021-02-27, 17:50)gibxxi Wrote: You may need to allow remote access to the SQL server, in the NAS GUI in order for this to work).

Well, I'm stuck early... I can't find this setting on my Synology.

When I try to start a new session with Heidi I get the error PLink exited unexpected. Command line was: plink.exe -ssh root@http://192.168.1.121 -pw "******" -P 22 -N -L 3306:http://192.168.1.121:3306uery

I'm assuming because that setting isn't done?

For network type I have MariaDB or MySQL (SSH tunnel)
For library I have libmariadb.dll


Also, tried to just backup with phpMyAdmin... It seems to export the databases fine but I have no idea where the files are. It doesn't tell me where it's saving them nor does it give me the option to choose.
Reply
#13
(2021-02-28, 16:14)huttdes Wrote:
(2021-02-27, 17:50)gibxxi Wrote: You may need to allow remote access to the SQL server, in the NAS GUI in order for this to work).

Well, I'm stuck early... I can't find this setting on my Synology.

When I try to start a new session with Heidi I get the error PLink exited unexpected. Command line was: plink.exe -ssh root@http://192.168.1.121 -pw "******" -P 22 -N -L 3306:http://192.168.1.121:3306uery

I'm assuming because that setting isn't done?

For network type I have MariaDB or MySQL (SSH tunnel)
For library I have libmariadb.dll


Also, tried to just backup with phpMyAdmin... It seems to export the databases fine but I have no idea where the files are. It doesn't tell me where it's saving them nor does it give me the option to choose.

Quite possibly. With WinSCP, assuming you can log in with your regular NAS ADMIN username and password, find the "my.cnf" file. I don't know where this is stored on Synology devices, as I've never owned one, but on QNAPs it's at "/etc/config/my.cnf". Once you've found it, right click on the file and click "edit - with internal editor" (Don't open linux files with Windows Notepad as it will mess up the line endings in the file when you re-save them), and look for the following line:
 
Code:
skip-networking

and comment the line out like so:

Code:
#skip-networking

EDIT: Also, depending on the MySQL / MariaDB version the NAS is running, check the file for the following term:

Code:
bind-address = 127.0.0.1
(Only listen on TCP for clients running on the same machine / host - like phpMyAdmin for example).

...and change it to:

Code:
bind-address = 0.0.0.0
(Listen for clients running on *ANY* host on the local network).

This does have security connotations, in that if you have a root user on the SQL server, with an unsecured / blank password that is allowed access from anything other than 127.0.0.1 (localhost) this is a major security concern. But on the flip-side, assuming you are behind a NAT-router, and are not running any web-apps like WordPress or exposing the TCP port for MySQL / MariaDB (3306) to the web, it's not absolutely critical for inbound attacks. I would however look into making sure that phpMyAdmin does not allow login with the root user with a blank password, and check the SQL users list and ensure that the only root user present in the users list is: root@127.0.0.1 and/or root@localhost. Don't delete or modify those entries, should they exist, as it's likely NAS-based services and addons will be making use of those accounts, and disabling them may have negative effects.

The changes to the "my.cnf" file should allow the server to allow incoming connections from remote hosts, which will have been disabled by default as a security feature. You will need to restart the SQL server for the change to take effect. This might be doable via the NAS web GUI, but you can also check the init.d folder with WinSCP for the relevant startup script and enter the following command in PuTTy after logging in. On QNAPs this is as follows:

Code:
/etc/init.d/mysqld.sh restart

Mine also has a "mariadb.sh" file, but running it / restarting it doesn't seem to do anything on my NAS (no on-screen output). If all else fails, just restart the NAS via the GUI and try again.

---

One thing I forgot to mention with regards HeidiSQL, if/when re-importing a saved SQL file, make sure in the "load SQL" dialog window, when browsing for and selecting the file to be loaded, change the "character encoding" field at the bottom of the dialog Window from "AUTO" to "UTF8", otherwise special foriegn characters won't be processed properly.

Dan / Gib.
Image
Reply
#14
Obviously, with the above, you need to ensure you have an admin-level user present in the MySQL / MariaDB users table that is able to connect from remote IP addresses (i.e: addresses on your local LAN). If you only ever envisage logging in from one machine you can create a user or users that will only be able to connect from that IP address. This will require however that your router is set to always assign said machine the same IP address via DHCP.

In order to create that user (or users), this will need to be done via phpMyAdmin. Add a user via phpMyAdmin like "[email protected]" (or whatever IP address your local PC runs on), and secure it with a strong password. This user will need global privileges equal to that of the root user, for use with HeidiSQL and/or Kodi. If you run Kodi on multiple machines, you can add the user as "[email protected].%" (The "%" is used as a wildcard character. You can also use "192.168.%" and omit the last octet), and secure with a password. Any IP address on your local network will then be able to connect via those credentials. It might be safer to restrict priviledges for the user you wish to use with Kodi, versus the one you use for HeidiSQL, but i'm not sure exactly what minimum-level priviledges the Kodi user requires, so I just assign global priviledges to my Kodi user, but use a strong password.

Due to limitations with the way MySQL / MariaDB works, you can only assign one IP address / address range per user account, hence the reason why you often see multiple users with the same username / credentials appear multiple times in the server's user list, but with differing IP addresses / ranges, in so far as I understand it.

Dan / Gib.
Image
Reply
#15
I have a question... all this back up stuff aside... I did find where it saved the backup with phpMyAdmin by the way...

However since all of my packages are installed on Volume 1 it doesn't want to let me delete it even though I have a second volume installed.
This is so frustrating... I generally have zero trouble figuring things out on a PC but when it comes to the NAS I'm freaking lost!!

I'm getting ready to just yank all 5 hard drives out and start over!!


EDIT>>>>> Volume 1 in the process of being removed...
EDIT2>>>> Volume 1 cleared but still RAID6 and it wont let me change it... all data wiped.
If it's been wiped I don't know why it wont let me change RAID type and start over. Shoot me please
EDIT3>>>> I think I'm making progress ... getting my 5x8 set to SHR1
Reply

Logout Mark Read Team Forum Stats Members Help
Synology NAS upgrade and Maria DB backup0