FRODO Thumbnail sync storage
#1
seeing as you are not supposed to use path substitutions where does frodo store the thumbnails so i can sync across to a second machine
Reply
#2
Stores it locally in the userdata (wiki) folder in the thumbnails directory. In Frodo you can delete the textures db file in the databases directory to force XBMC to re-download all thumbs, which should be the same on each system (thus in sync, but downloaded individually on each system).
Reply
#3
Deleting the textures db will, as far as I know:

1. Leave all existing artwork present on disk.
2. Download new artwork for every item, resulting in a size doubling of the cache. Plus, this is no guarantee that every client gets the same artwork unless you've made an external export of your database, which places all the fanart next to your media. This only applies to the videodatabase.

imo Frodo is a step backwards in artwork management. In Eden, you have a seperate folder for video and music fanart and all thumbnails are stored in the 0 to f folders. Now everything is stored in 0 to f, making it impossible to keep things tidy. This weekend, I wanted to recreate my music db, but cleaning up old artwork was impossible.
CoreElec on a tn95 (s905x). Onkyo NR-656. Canton Movie CD-1000. LG 55B6V.

If it ain't broke: break it, fix it, repeat
Reply
#4
It won't size double the cache at all. It will overwrite the existing files.

Further, assuming that each client uses the same database, the artwork will be identical, as that's where it's stored.

You also have a database showing the actual mapping from original URL to hashed URL. Thus, it's trivial to clean up: Anything not in Textures*.db can be deleted off disk. Further, you have tracking of how used each texture is in the database, so you could delete every texture not used in the last K years with a simple query.

Apologies welcome.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#5
(2012-10-02, 01:23)jmarshall Wrote: It won't size double the cache at all. It will overwrite the existing files.

Further, assuming that each client uses the same database, the artwork will be identical, as that's where it's stored.

You also have a database showing the actual mapping from original URL to hashed URL. Thus, it's trivial to clean up: Anything not in Textures*.db can be deleted off disk. Further, you have tracking of how used each texture is in the database, so you could delete every texture not used in the last K years with a simple query.

Apologies welcome.

"As far as I know" implies I could be wrong, so I'm not apologizing for anything. If I'm wrong correct me, but please don't be so arrogant. I'm active on a forum to share and to learn, not to be dissed.

Deleting the textures.db file, like Ned Scott said, will delete the reference between the artwork and the media right? I've tried this and as far as I can tell, the next time you start xbmc, the artwork will reload. Since the filename of the cached artwork gets a unique identifier, the redownloaded artwork gets a new name, thus doubling the cache size. That's what I meant by my first point.

As for point two, if you don't have artwork stored locally, you get artwork from the internet. If you delete the textures.db, chances are you get different artwork every time you do this.

As far as cleaning up the artwork folder, I could use the database and delete files which haven't been accessed in a certain timeframe manually, but I'd rather see that artwork get's deleted when the corresponding item is no longer in the video or music database. You may think it's trivial, but not everyone has xbmc installed on a large disk.

CoreElec on a tn95 (s905x). Onkyo NR-656. Canton Movie CD-1000. LG 55B6V.

If it ain't broke: break it, fix it, repeat
Reply
#6
(2012-10-02, 08:23)Raytestrak Wrote: Deleting the textures.db file, like Ned Scott said, will delete the reference between the artwork and the media right? I've tried this and as far as I can tell, the next time you start xbmc, the artwork will reload. Since the filename of the cached artwork gets a unique identifier, the redownloaded artwork gets a new name, thus doubling the cache size. That's what I meant by my first point.
Wrong.
If the original url stays the same so will the local cache name

Quote:As for point two, if you don't have artwork stored locally, you get artwork from the internet. If you delete the textures.db, chances are you get different artwork every time you do this.
Wrong.The url is stored in database and will not change unless you rescrape or refresh.



Quote:"As far as I know" implies I could be wrong, so I'm not apologizing for anything. If I'm wrong correct me, but please don't be so arrogant. I'm active on a forum to share and to learn, not to be dissed.
I see no arrogance apart from a bit crude response from your initial response that gave us the idea we did some mayor fuck up
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
#7
(2012-10-02, 08:28)Martijn Wrote:
(2012-10-02, 08:23)Raytestrak Wrote: Deleting the textures.db file, like Ned Scott said, will delete the reference between the artwork and the media right? I've tried this and as far as I can tell, the next time you start xbmc, the artwork will reload. Since the filename of the cached artwork gets a unique identifier, the redownloaded artwork gets a new name, thus doubling the cache size. That's what I meant by my first point.
Wrong.
If the original url stays the same so will the local cache name

Quote:As for point two, if you don't have artwork stored locally, you get artwork from the internet. If you delete the textures.db, chances are you get different artwork every time you do this.
Wrong.The url is stored in database and will not change unless you rescrape or refresh.



Quote:"As far as I know" implies I could be wrong, so I'm not apologizing for anything. If I'm wrong correct me, but please don't be so arrogant. I'm active on a forum to share and to learn, not to be dissed.
I see no arrogance apart from a bit crude response from your initial response that gave us the idea we did some mayor fuck up

Thanks for explaining Martijn. I was indeed wrong in my assumptions.
CoreElec on a tn95 (s905x). Onkyo NR-656. Canton Movie CD-1000. LG 55B6V.

If it ain't broke: break it, fix it, repeat
Reply
#8
Np Smile It can be a complicated
Read/follow the forum rules.
For troubleshooting and bug reporting, read this first
Interested in seeing some YouTube videos about Kodi? Go here and subscribe
Reply
#9
1. Understanding this thread has me lost. Artwork stored locally on headless machine with limited storage makes no sense. I have a 12 TB NAS for storage, and very, VERY Small storage on the local clients. I live in the country, the proverbial BFE. I don't want or can depend on URL artwork when the internet decides to take a dump for the night.

2. Can't possibly understand why local NAS stored artwork could possibly load "Slower" then URL over the internet or why it has to be stored on each machine on the local network.

All the data, Movies and Music and their artwork should be stored on the NAS to be shared with all instances of the front end. Am I missing something here?

Thanks,
Bill


Reply
#10
(2012-10-05, 02:41)wcrowder Wrote: 1. Understanding this thread has me lost. Artwork stored locally on headless machine with limited storage makes no sense. I have a 12 TB NAS for storage, and very, VERY Small storage on the local clients. I live in the country, the proverbial BFE. I don't want or can depend on URL artwork when the internet decides to take a dump for the night.

2. Can't possibly understand why local NAS stored artwork could possibly load "Slower" then URL over the internet or why it has to be stored on each machine on the local network.

All the data, Movies and Music and their artwork should be stored on the NAS to be shared with all instances of the front end. Am I missing something here?

Thanks,
Bill

If you already have local artwork on your NAS, then XBMC will prefer that over external artwork when scraped. So in that case, while it will cache it local for each device, it will never have to go to some URL over the internet if you decide to rebuild your thumbnails.
Reply
#11
Something I thought of the other day with the new Frodo thumbnail setup that could be advantageous (as long as I understand how things work with it). One could have 2 systems with synced thumbnails and could adjust the thumbnail size using the thumbsize tag in advancedsettings.xml independent of each machine. Maybe one machine is lowend so someone would want to decrease thumbsize for better performance and another machine is attached to a large display and the thumbsize would be increased a bit for better quality.

The new thumbnail caching system will allow for that while at the same time keeping thumbnails in sync, correct?
Reply
#12
Quote:The new thumbnail caching system will allow for that while at the same time keeping thumbnails in sync, correct?

Sure.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#13
Quote:1. Understanding this thread has me lost. Artwork stored locally on headless machine with limited storage makes no sense. I have a 12 TB NAS for storage, and very, VERY Small storage on the local clients. I live in the country, the proverbial BFE. I don't want or can depend on URL artwork when the internet decides to take a dump for the night.

Each client has a local cache. They don't hit the internet if it's in the cache.

Quote:2. Can't possibly understand why local NAS stored artwork could possibly load "Slower" then URL over the internet or why it has to be stored on each machine on the local network.
The texture cache doesn't care where the artwork originally came from: It'll cache it anyway. If it's from your NAS, then caching will be faster than from the internet, but once cached, it won't be touched again.

Quote:All the data, Movies and Music and their artwork should be stored on the NAS to be shared with all instances of the front end. Am I missing something here?
Yes, you're missing speed. Loading images off a local disk versus loading them off a network disk is quite a large difference. You don't need huge amounts of space on the clients - a few GB is all you need for most purposes.
Always read the XBMC online-manual, FAQ and search the forum before posting.
Do not e-mail XBMC-Team members directly asking for support. Read/follow the forum rules.
For troubleshooting and bug reporting please make sure you read this first.


Image
Reply
#14
So, if I understand correctly, Frodo stores artwork URLs in the (possibly centralized) database, so in theory each client should use the same artwork?

There are still reasons to prefer a shared cache for all clients.

(2012-10-05, 04:47)jmarshall Wrote: Each client has a local cache. They don't hit the internet if it's in the cache.
Yes, but after initial scraping by one client, all other clients have to hit the internet at least once per artwork.

For example, I may add a big bunch of episodes or movies, where scraping and artwork download already costs a considerable amount of time on my slow internet connection. Also, some of my clients can stay unused for days or even weeks, so new and uncached artwork URLs in the database would pile up. When such a client is started again, having to wait for a big pile of artwork to be downloaded could be a nuisance. And what if some of those URLs are broken or have changed in the meantime, or the server hosting them is unresponsive at the moment. Happens often enough for me.

(2012-10-05, 04:47)jmarshall Wrote: Yes, you're missing speed.
Not necessarily. My linux clients don't use local disks at all, they boot with pxe and nfs over a gigabit link. And my different Windows user profiles including the XBMC thumbnail cache are on network shares to enable roaming.

Currently all my thumbnails are stored in a single location on the server, and all XBMC instances are sharing them, with help of a combination of nfs mounts and symlinks. What would happen to that scenario with Frodo? I would need to centralize the TextureDB, which contains URL->hash mappings too, I assume? Is that even possible?
Reply
#15
They are more scenarios where a local cache may be worse than a cache in the local network. Consider a cheap system like Rasperry Pi, which boots from a small SD flash card, attached over a gigabit link to a powerful server with a fast and big RAID.

A flexible solution addressing those different needs: provide an optional second level cache.

The local cache inside the userdata folder would be the first level. It would be the only one enabled be default.

Additionally, a second level cache on a network share (nfs, smb...) can be configured via advancedsettings, and optionally the local cache can be disabled if only network caching is wanted.

Reply

Logout Mark Read Team Forum Stats Members Help
FRODO Thumbnail sync storage0