!END OF LIFE! centralized and deduplicated .actors directory v0.5
#1
PLEASE USE THE NEWER VERSION WITH HARD LINKS -> https://forum.kodi.tv/showthread.php?tid...pid2925727

Unfortunately this version has a big disadvantage. If some directory names (especially the collection path) are changed the links will break. But maybe the source code is useful for someone.

Restore your data with v0.5 e.g.  ./actorsCentralizer.sh --restore ASAP!
To check if you have already broken links, run this script with
--check
The last option is to run this script with  --delete to delete all broken links and then download the images with tmm again. Sorry for the inconveniences.


Centralization and deduplication of ".actors" directories? yes, please!

Discussed many times in various forums, but never really implemented...

Well, with the script below, you finally can do that....
Admittedly a very quick & dirty assembled solution to simply accept the challenge....

V0.5 is no longer compatible with V0.1, but don't worry, you can still restore your data, see below.
KODI: Do not forget to enable Settings -> Media -> Videos -> Library -> Download actor thumbnails option, or you will not see any actor thumbnail.
Local files will be used first, that's the trick.

Windows user:
The script was created and tested under Linux BASH (Ubuntu). It may or may not work on Windows 10 if you enable Linux support there, but feel free to try it if you are brave enough.

Source code:
Unfortunately you can't upload files here, for the source code see post 2
Not every distribution supports every command or their parameters. So it is possible that it will not work on every system.
But I strongly recommend that you read on first.

Requirements:
The file system where your movie collection is stored must support symbolic links.

Worst case scenario:
The ".actors" directories will be shot up.
In this case simply delete them and reload the images via tmm. That's it.
But there is also a local possibility to restore them, see below.

Nevertheless a word of warning:
I have tested the script myself, but there is no WARRANTY...
If you are not familiar with Linux, better leave it alone.
ALWAYS test it yourself in a test environment first. To keep the script small, it generates hardly usable error messages.

The savings potential:
Since we are talking about a relation of one too many it depends on how many movies/tv shows an actor appears in. Since everyone usually has favorite actors, the hit rate will definitely be high. So if you have a large collection you can easily get several 100MB of the same data. Without big bla bla, about 50% of space saving.

The how:
Per collection a central directory called ".centralized"  will be created to store the actor images.
The script will first copy and then link all files in any ".actors" directory.

Code:
/absolutepath of collection/%movie%/.actors/%actor%.jpg --> /absolutepath of collection/.centralized/token/%actor%.jpg

A simplified example with Christian Bale from the dark knight collection. Already with one actor we have saved 100KB (3 x 37KB on disc), or 75% in this case.

Code:

tmm_test
├──.centralised
│         └── c
│           └── ch
│             └── Christian_Bale.jpg
├── Movies
    ├── Movie 1
    │        └── Christian_Bale.jpg -> /home/USER/Videos/tmm_test/.centralised/c/ch/Christian_Bale.jpg
    ├── Movie 2
    │        └── Christian_Bale.jpg -> /home/USER/Videos/tmm_test/.centralised/c/ch/Christian_Bale.jpg
    ├── Movie 3
    │        └── Christian_Bale.jpg -> /home/USER/Videos/tmm_test/.centralised/c/ch/Christian_Bale.jpg
    └── Movie 4
              └── Christian_Bale.jpg -> /home/USER/Videos/tmm_test/.centralised/c/ch/Christian_Bale.jpg

Unfavorable cases:
You have distributed different collections to different external systems, e.g. Nas, then the potential savings may be smaller. Whether it works depends on the settings of the respective system, which I cannot go into here.
The files might not be linked anymore and will still appear twice. You could try running this script locally on these devices, but remember to configure the script with absolute paths.

Restore your data:
Starting with version 0.5 you can also undo all changes. To do so, change to the collection directory and call the script via its abspolute path. Everything else is done automatically. For more details see the example below.

Configuration:
There are only 2 settings, found at the top of the script. Be sure to set them correctly, otherwise the world may end...
yourCollections:  Folders in which your "COLLECTIONS" are stored. Use only absolute paths.
Code:

└── tmm_test <----- This is what you need
    ├── movies
    │   └── .....
    └── shows
        └── .....
checkForTmm: Can be 1|0 BE CAREFUL! 0 = The script is also executed when tmm is running. This has unforeseeable consequences.
However, checking for tmm only works if you use the script on the same system.

Code:

#################### USER CONFIGURATION ######################
yourCollections=("Folder 1" "Folder 2" "and so on")           
checkForTmm=1                                                                  

Usage:
  1. Never use the standard applications of windows, as this will change the formatting and the script will no longer work.
  2. Copy the source code in a file e.g. "actorCentralizer.sh
  3. mark it as executable chmod +x actorCentralizer.sh
  4. you can start it with ./ actorCentralizer.sh
If you scrape new content just execute the file. The script will automatically detect if there is something to do. Sounds like a lot of work, but on a normal PC this happens very fast, mostly in a few seconds.
For some fun you can even see how fast certain commands are.

New command line parameters:

--help            Shows this help

Call following options inside the desired collection folder.
--restore       All symlinks are replaced with their original files
--migrate01  Undo all changes from v0.1
--check         Shows a list of broken symlinks
--delete         Deletes all broken symlinks

Virtualization:
The script has already processed tmm_test, so no new (unlinked) files are found.
Code:

---------------------------------------------------------
 .actors centralization & deduplication has started...
                     Version 0.5
---------------------------------------------------------
Collection: [/media/NAS/Multimedia/TMM_TEST]
Symlinks: not supported...                                   <--- Symlinks over Samba are not supported on this NAS
---------------------------------------------------------
Collection: [/home/user/Videos/tmm_test]
Symlinks: supported...                                        <--- Symlinks are supported, everything is fine
Searching for .actors folders...Done in 0.00855936 second(s)
Found 0 MB in 0 files.                                        <--- No physical files found (already symlinked collection)
no files found...
---------------------------------------------------------
Collection: [/home/user/Videos/tmm_test2]
Symlinks: supported...
Searching for .actors folders...Done in 1.1085 second(s)
Found 26.2695 MB in 823 files.
Centralized folder: [/home/user/Videos/tmm_test2/.centralized]
Copy OK: 823 Errors: 0 Symlink OK: 823 Errors: 0
------------------------ summary ------------------------
Centralized folder before 0 MB, now 11.1367 MB       <--- 11MB vs 26MB!!!!
---------------------------------------------------------

Restoring content, undo virtualization:
We are already in the collection directory tmm_test2 and call the script via its absolute path and parameter --restore. The script will now restore all symlinked files.
All images are now restored in their original place. Note that the same images are still present in the .centralized folder. You can now delete this directory if you no longer need it.
Code:

user@pc:~/Videos/tmm_test2$ /home/user/Videos/actorsCentralizer/actorsCentralizer_0.5.sh --restore
Restoring .actors folders in /home/user/Videos/tmm_test2
Searching for .actor folders...
Found 823 files.
Copy OK: 823 Errors: 0
Done in 1.5599 second(s)

Migration of v0.1:
We are already in the collection directory tmm_test and call the script via its absolute path and parameter --migrate01.
This collection was virtualized with v0.1.
Code:

user@pc:~/Videos/tmm_test$ /home/user/Videos/actorsCentralizer/actorsCentralizer_0.5.sh --migrate01
Migrating v0.1 content in /home/user/Videos/tmm_test
Searching for .actors folders...
Found 11 folders.
Copy OK: 11 Errors: 0
Done in 0.0639293 second(s)
---------------------------------------------------------
Restoring .actors folders in /home/user/Videos/tmm_test
Searching for .actors folders...
Found 823 files.
Copy OK: 823 Errors: 0
Done in 1.68524 second(s)
---------------------------------------------------------
You should now have all files on their old places. Run this script again to virtualize them.
---------------------------------------------------------
#2
V0.5
  • New concept
  • support for ARGs
  • no need for CRC anymore
  • findBrokenLinks(),killBrokenlinks
  • restoring content
  • data migration from v0.1

 V0.4
  • bugfix: more accurate search
  • code compacting
  • renaming variables for better readabilty
 V0.3
  • bc, changed to more compatible awk
  • crc32, changed to more compatible cksum
  • tidy up
 V0.2
  • typos

 V0.1
  • Initial version




Code:

#!/bin/bash
# .actors centralization & deduplication for tinyMediaManager
# Copyright 2020 MicroDevil
# #
# You can use this "Software" free of charge and exclusively for personal and non-profit use.
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
# IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

#################### USER CONFIGURATION ######################
# Folders where your files are stored. Ensure it's an absolute path. No distinction between films and series is necessary
#yourCollections=("Folder 1" "Folder 2" "and so on")
yourCollections=("Folder 2" "Folder 2" "and so on")

# Possible setting 1/0. BE CAREFUL! 0 = The script is also executed when tmm is running. This has unforeseeable consequences.
checkForTmm=1
####################### DO NOT TOUCH ANYTHING BELOW THIS LINE ######################
Version=0.5
# This is an advanced setting, just leave it alone AND DO NOT USE IT
# forceCentralizedFolder=""
functionRuntime=0
OLDIFS=$IFS #save IFS
CurrentDir=${PWD}
ScriptPath="$(readlink -f "${BASH_SOURCE[0]}")"

############################################## Colors
Cyellow_b="\e[93m"
Cred="\e[31m"
Cgreen_b="\e[92m"
Creset="\e[0m"
CLine="---------------------------------------------------------"
########################################### Functions
help() {
  echo ".actors centralization & deduplication for KODI, V $Version"
  echo
  echo "option:"
  echo -e "\t--help\t\tShow this help"
  echo
  echo -e "\t\t\tCall following options inside the desired collection folder."
  echo
  echo -e "\t--restore\tAll symlinks are replaced with their original files "
  echo -e "\t--migrate01\tUndo all changes from v0.1"
  echo -e "\t--check\t\tShows a list of broken symlinks"
  echo -e "\t--delete\tDeletes all broken symlinks"
  echo
}
mEcho() {
  functionRuntime=$(date +%s%N) && echo -ne "${1}"
}
jobDone() {
  echo -ne "$Cgreen_b"" \bDone in $(calc "($(date +%s%N) - ${functionRuntime}) * 0.000000001") second(s)\n""$Creset"
}
calc() {
  awk "BEGIN { print $*}"
}
checkForSymLinkSupport() {
  checkFile="${1}/checklink"
  ln -sf "/tmp" "${checkFile}" 2>/dev/null
  [ -L "${checkFile}" ] && [ -e "${checkFile}" ] && rm "${checkFile}" 2>/dev/null && return 0 || return 1
}
restore() {
  echo "Restoring .actors folders in $CurrentDir" && echo "Searching for .actors folders..."
  centralRestore="${PWD}/.centralized" && [ ! -d "${centralRestore}" ] && echo -ne "$Cyellow_b"".centralized folder not found$Creset\n$CLine\n" && exit 1
  toBeRestoreAllActorFiles=()
  while IFS= read -r line; do toBeRestoreAllActorFiles+=("${line}"); done < <(find "$CurrentDir" -type l | grep "/.actors/")
  IFS=$OLDIFS # restore IFS
  echo -e "Found $Cgreen_b${#toBeRestoreAllActorFiles[@]}$Creset files."
  mEcho "" && Csuccess=0 && Cerror=0
  for linkedFile in "${toBeRestoreAllActorFiles[@]}"; do
    symlink="$(readlink -e "${linkedFile}")" && rm -f "${linkedFile}" && cp -f "${symlink}" "${linkedFile}"
    [ -f "${linkedFile}" ] && [ ! -L "${linkedFile}" ] && ((Csuccess += 1)) || ((Cerror += 1))
    echo -ne "\rCopy OK: $Cgreen_b$Csuccess$Creset Errors: $Cred${Cerror}$Creset"
  done && echo && jobDone # && exit 0
  }
migrate() {
  echo "Migrating v0.1 content in $CurrentDir" && echo "Searching for .actors folders..."
  centralRestore="${PWD}/.centralized" && [ ! -d "${centralRestore}" ] && echo -ne "$Cyellow_b"".centralized folder not found$Creset\n$CLine\n" && exit 1
  toBeRestoreAllActorFiles=()
  while IFS= read -r line; do toBeRestoreAllActorFiles+=("${line}"); done < <(find -L "$CurrentDir" -type d | grep "/.actors")
  IFS=$OLDIFS # restore IFS
  echo -e "Found $Cgreen_b${#toBeRestoreAllActorFiles[@]}$Creset folders."
  mEcho "" && Csuccess=0 && Cerror=0
  for virtFolder in "${toBeRestoreAllActorFiles[@]}"; do
    theBasename="$(basename "${virtFolder}")"
    dirName="$(dirname "${virtFolder}")"
    [ ! "${theBasename}" == ".actors" ] && continue
    mv -f "${dirName}/.actors" "${dirName}/.actors_conv"
    mkdir -p  "${dirName}/.actors"  && cp -fr "${dirName}/.actors_conv/"* "${dirName}/.actors" && rm -rf "${dirName}/.actors_conv"
    [ -d "${dirName}/.actors" ] && ((Csuccess += 1)) || ((Cerror += 1))
    echo -ne "\rCopy OK: $Cgreen_b$Csuccess$Creset Errors: $Cred${Cerror}$Creset"
  done
  echo && jobDone && echo -ne "$CLine\n"
  restore
  rm -rf "${centralRestore}/virtual" 2>/dev/null && rm -rf "${centralRestore}/physical" 2>/dev/null && rm -f "${centralRestore}/relations.csv" 2>/dev/null
  echo -e "$Cgreen_b$CLine\nYou should now have all files on their old places. Run this script again to virtualize them."
  echo -e "$CLine$Creset"
  exit 0
}
######################################### Some checks
[ $checkForTmm != 0 ] && [ $(ps aux | grep -co tmm.jar) -gt 1 ] && echo -e "$Cred""tmm seems to be running, exiting...""$Creset" && exit 1
######################################### ARGs parse
for option in "$@"; do
  case "${option,,}" in
  --help | -h) help && exit 0 ;;
  --restore) restore && exit 0 ;;
  --migrate01) migrate && exit 0 ;;
  --check) echo "Checking for broken symlinks" && find -xtype l && exit 0 ;;
  --delete) echo "Deleting broken symlinks" && find -xtype l -delete && exit 0 ;;
  *) help && exit 1 ;;
  esac
done
############################################### START
clear
echo -e "$Cgreen_b$CLine\n .actors centralization & deduplication has started..."
echo "                     Version ${Version}"
echo -e "$CLine$Creset"

for singleCollection in "${yourCollections[@]}"; do
  [ ${#singleCollection} -lt 1 ] && [ "${singleCollection}" == "\n" ] && continue
  collectionPathFilesCount=0 # Number of unvirtualized folders
  collectionPathSizeInKB=0   # Size
  #####################################################   Per collection path checks
  echo -e "Collection:$Cyellow_b [${singleCollection}]"
  [ ! -d "${singleCollection}" ] && echo -ne "$Cyellow_b""this path is inaccessible$Creset\n$CLine\n" && continue
  ! checkForSymLinkSupport "${singleCollection}" && echo -e "Symlinks:$Cred not supported...\n$Creset$CLine" && continue
  echo -e "Symlinks:$Cgreen_b supported...$Creset"
  collectionPathCentralizedFolder="${forceCentralizedFolder}" # Not used yet
  [ ${#forceCentralizedFolder} -lt 1 ] && collectionPathCentralizedFolder="${singleCollection}/.centralized"
  [ ${#forceCentralizedFolder} -ge 1 ] && echo -e "$Cyellow_b""forced centralized folder:\b$Creset ${forceCentralizedFolder}" && mkdir -p "${forceCentralizedFolder}" 2>/dev/null
  mEcho "Searching for .actors folders..."
  collectionAllActorFiles=()
  while IFS= read -r line; do
    ((collectionPathFilesCount += 1)) && collectionAllActorFiles+=("${line}")
    thisSize="$(du -s "${line}" | cut -f 1)" && ((collectionPathSizeInKB += thisSize))
  done < <(find "${singleCollection}" -type f | grep "/.actors/")
  IFS=$OLDIFS # restore IFS
  jobDone && echo -e "Found $Cgreen_b""$(calc ${collectionPathSizeInKB}/1024) MB$Creset in $Cgreen_b${collectionPathFilesCount}$Creset files."
  [ ${#collectionAllActorFiles[@]} -lt 1 ] && echo -e "$Cred""no files found...""$Creset\n$CLine" && continue
  echo -e "Centralized folder:$Cyellow_b [${collectionPathCentralizedFolder}]$Creset"
  mkdir -p "${collectionPathCentralizedFolder}" 2>/dev/null
  sizeAllbefore=$(du -s "${collectionPathCentralizedFolder}" | cut -f 1)
  mEcho "" && Csuccess=0 && Cerror=0 && Lsuccess=0 && Lerror=0
  for realFile in "${collectionAllActorFiles[@]}"; do
    theBasename="$(basename "${realFile}")"
    theToken1="${theBasename:0:1}" && theToken2="${theBasename:0:2}"                                     #first 2 chars
    tokenPath="${theToken1}/${theToken2}" && dirPath="${collectionPathCentralizedFolder}/${tokenPath,,}" #Lowercased tokenspath
    newPath="${dirPath}/${theBasename}"                                                                  #file path
    mkdir -p "${dirPath}"                                                                                #create dir
    cp -f "${realFile}" "${newPath}" && ln -sf "${newPath}" "${realFile}"
    [ -f "${newPath}" ] && ((Csuccess += 1)) || ((Cerror += 1))
    [ -L "${realFile}" ] && ((Lsuccess += 1)) || ((Lerror += 1))
    echo -ne "\rCopy OK: $Cgreen_b$Csuccess$Creset Errors: $Cred${Cerror}$Creset Symlink OK: $Cgreen_b$Lsuccess$Creset Errors: $Cred${Lerror}$Creset"
  done
  sizeNowThis=$(du -s "${collectionPathCentralizedFolder}" | cut -f 1)
  echo -e "\n$Cyellow_b""------------------------ summary ------------------------$Creset"
  echo -e "Centralized folder before $Cgreen_b$(calc ${sizeAllbefore}/1024)$Creset MB, now $Cgreen_b$(calc ${sizeNowThis}/1024) MB$Creset"
  echo -e "$Cyellow_b""$CLine$Creset"
  echo -e "\n$CLine$Creset"
done
exit 0


#3
RESERVED

Logout Mark Read Team Forum Stats Members Help
!END OF LIFE! centralized and deduplicated .actors directory v0.50