FileIO on zip files Fails ARM v18
#1
I'm testing my addon on v18 and found a strange difference between ARM 32-bit OS and other OS's (Windows/OSX/Linux x86) with normal fileIO.

Here's my test Code:

print 'This is a file test'
import zipfile
with open(os.path.join(get_userdata_temp_dir(),'Test_File.zip'),'rb') as content_file:
    content_file.seek(0,2)
    print content_file.tell()
    print zipfile.is_zipfile(content_file)

Android v7a log:

22:48:06.681 T:18446744072283060448  NOTICE: special://profile/ is mapped to: special://masterprofile/
22:48:06.681 T:18446744072283060448  NOTICE: -----------------------------------------------------------------------
22:48:06.682 T:18446744072283060448  NOTICE: Starting Kodi (18.0-ALPHA1 Git:20180308-d366eb4). Platform: Android ARM 32-bit
22:48:06.682 T:18446744072283060448  NOTICE: Using Debug Kodi x32 build
22:48:06.682 T:18446744072283060448  NOTICE: Kodi compiled Mar  8 2018 by Clang 5.0.300080  for Android ARM 32-bit API level 21 (API level 21)
22:48:06.682 T:18446744072283060448  NOTICE: Running on amlogic MXQ pro+ with Android 5.1.1 API level 22, kernel: Linux ARM 64-bit version 3.14.29
22:48:06.682 T:18446744072283060448  NOTICE: FFmpeg version/source: 3.4.1-Kodi
22:48:06.682 T:18446744072283060448  NOTICE: Host CPU: AArch64 Processor rev 4 (aarch64), 4 cores available
22:48:06.683 T:18446744072283060448  NOTICE: Product: p201, Device: p201, Board: p201 - Manufacturer: amlogic, Brand: Android, Model: MXQ pro+, Hardware: amlogic
22:48:06.687 T:18446744072283060448  NOTICE: External storage path = /storage/emulated/0; status = ok
22:48:06.687 T:18446744072283060448  NOTICE: ARM Features: Neon enabled
22:48:06.688 T:18446744072283060448  NOTICE: special://xbmc/ is mapped to: /data/data/org.xbmc.kodi/cache/apk/assets
22:48:06.688 T:18446744072283060448  NOTICE: special://xbmcbin/ is mapped to: /data/data/org.xbmc.kodi/cache/apk/assets
22:48:06.688 T:18446744072283060448  NOTICE: special://xbmcbinaddons/ is mapped to: /data/app/org.xbmc.kodi-2/lib/arm
22:48:06.688 T:18446744072283060448  NOTICE: special://masterprofile/ is mapped to: /storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/userdata
22:48:06.688 T:18446744072283060448  NOTICE: special://envhome/ is mapped to: /storage/emulated/0/Android/data/org.xbmc.kodi/files
22:48:06.688 T:18446744072283060448  NOTICE: special://home/ is mapped to: /storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi
22:48:06.688 T:18446744072283060448  NOTICE: special://temp/ is mapped to: /storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/temp
22:48:06.688 T:18446744072283060448  NOTICE: special://logpath/ is mapped to: /storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/temp
22:48:06.689 T:18446744072283060448  NOTICE: The executable running is: /data/app/org.xbmc.kodi-2/lib/arm/libkodi.so
22:48:06.728 T:18446744072283060448  NOTICE: Local hostname: 192.168.1.14
22:48:06.729 T:18446744072283060448  NOTICE: Log File is located: /storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/temp//kodi.log
22:48:06.729 T:18446744072283060448  NOTICE: -----------------------------------------------------------------------
22:48:06.777 T:18446744072283060448  NOTICE: load settings...
...
22:48:07.206 T:18446744072283060448  NOTICE: ADDON: xbmc.python v2.25.0 installed
...
22:48:30.017 T:18446744072304996032   DEBUG: This is a file test
22:48:30.023 T:18446744072304996032   DEBUG: 2
22:48:30.023 T:18446744072304996032   DEBUG: False

OSX Log

13:59:15.991 T:140735181774848  NOTICE: -----------------------------------------------------------------------
13:59:15.991 T:140735181774848  NOTICE: Starting Kodi (18.0-ALPHA1 Git:20180308-d366eb411a). Platform: OS X x86 64-bit
13:59:15.991 T:140735181774848  NOTICE: Using Debug Kodi x64 build
13:59:15.991 T:140735181774848  NOTICE: Kodi compiled Mar  8 2018 by Clang 9.0.0 (clang-900.0.37) for OS X x86 64-bit version 10.8.0 (1080)
13:59:15.992 T:140735181774848  NOTICE: Running on Apple Inc. MacBookPro12,1 with OS X 10.11.6, kernel: Darwin x86 64-bit version 15.6.0
13:59:15.992 T:140735181774848  NOTICE: FFmpeg version/source: 3.4.1-Kodi
13:59:15.992 T:140735181774848  NOTICE: Host CPU: Intel® Core™ i5-5257U CPU @ 2.70GHz, 4 cores available
13:59:15.992 T:140735181774848  NOTICE: special://xbmc/ is mapped to: /Applications/Kodi_Retroplayer.app/Contents/Resources/Kodi
13:59:15.992 T:140735181774848  NOTICE: special://xbmcbin/ is mapped to: /Applications/Kodi_Retroplayer.app/Contents/Resources/Kodi
13:59:15.992 T:140735181774848  NOTICE: special://xbmcbinaddons/ is mapped to: /Applications/Kodi_Retroplayer.app/Contents/Resources/Kodi/addons
13:59:15.992 T:140735181774848  NOTICE: special://masterprofile/ is mapped to: /Users/XXX/Library/Application Support/Kodi/userdata
13:59:15.993 T:140735181774848  NOTICE: special://envhome/ is mapped to: /Users/XXX
13:59:15.993 T:140735181774848  NOTICE: special://home/ is mapped to: /Users/XXX/Library/Application Support/Kodi
13:59:15.993 T:140735181774848  NOTICE: special://temp/ is mapped to: /Users/XXX/.kodi/temp
13:59:15.993 T:140735181774848  NOTICE: special://logpath/ is mapped to: /Users/XXX/Library/Logs
13:59:15.993 T:140735181774848  NOTICE: special://frameworks/ is mapped to: /Applications/Kodi_Retroplayer.app/Contents/Libraries
13:59:15.993 T:140735181774848  NOTICE: The executable running is: /Applications/Kodi_Retroplayer.app/Contents/MacOS/Kodi
13:59:15.993 T:140735181774848  NOTICE: Local hostname: .local
13:59:15.993 T:140735181774848  NOTICE: Log File is located: /Users/XXX/Library/Logs//kodi.log
13:59:15.993 T:140735181774848  NOTICE: -----------------------------------------------------------------------
13:59:16.003 T:140735181774848  NOTICE: load settings...
...
13:59:16.181 T:140735181774848  NOTICE: ADDON: xbmc.python v2.25.0 installed
...
15:44:42.743 T:123145305063424   DEBUG: This is a file test
15:44:42.744 T:123145305063424   DEBUG: 53602
15:44:42.744 T:123145305063424   DEBUG: True

Any old test zip file will have the same issue (my test file was just two text files zipped up).  This did not previously occur in v17.6 that I'm aware of.  Is there a difference in v18 now that would account for this change?
Reply
#2
UP! The problem still exists in the latest version of Kodi 18 Alpha 3 on Android devices. Someone has found a solution to this problem?
Reply
#3
Yes, @peak3d identified a workaround.

Instead of:

print 'This is a file test'
import zipfile
with open(os.path.join(get_userdata_temp_dir(),'Test_File.zip'),'rb') as content_file:
  content_file.seek(0,2)
  print content_file.tell()
  print zipfile.is_zipfile(content_file)

You can use io:

print 'This is a file test'
import zipfile, io
with io.FileIO(os.path.join(get_userdata_temp_dir(),'Test_File.zip'),'rb') as content_file:
  content_file.seek(0,2)
  print content_file.tell()
  print zipfile.is_zipfile(content_file)

I'm hesitant to change all my open calls with io.FileIO though, so it would be an Android only kind of fix.  I'm not sure what the differences are or how that would effect other OS'.
My other workaround would be to use vfs.libarchive which will handle all types of archive files (but that is not yet available for all operating systems)
Reply
#4
Point is, that android seems to have focused on the lseek / lssek64 implementation (using the file descriptor (int) instead of the FILE type).

io.FileIO is using lseek(64) whereas the standard builtIn File implementation uses seek / fget- / fsetpos what is a mess.
The toolchain (provided from android) states file offsets as 64 Bit, but most arm32 devices implement the offset in 32 Bit -> This cannot work.

From my POV it seems nearly impossible to solve in kodi, because it could from theory differ between the devices.

Edit: My suggestion is simply using io.FileIO always if you use zip container or do manual seek / tell operations.
The implementation of the builtIn / fileIO handling are identical except they are using different file descriptors.
Reply
#5
Thanks @peak3d 

I believe this issue may also effect xbmcvfs.
For example, if I use

xbmcvfs.Stat(path_to_file).st_size()

In most instances it returns "-6026299017565735676" on ARM v18 and a valid number on ARM v17 (and other operating systems).

Also, trying 

with closing(xbmcvfs.File(dest,'w')) as file_to_write:
...
Results in an error of [Errno 2] No such file or directory when xbmcvfs is used to try and write a file.

I will note that it works once, the first time Kodi is opened, and then fails on every subsequent call to xbmcvfs
Reply
#6
Thank you, guys, for your answers. Now everything is clear Smile
Reply
#7
(2018-06-27, 21:41)zachmorris Wrote: Thanks @peak3d 

(...)

In most instances it returns "-6026299017565735676" on ARM v18 and a valid number on ARM v17 (and other operating systems).

interesting, but yes, most probably same issue there because we build kodi with the same options we build python library.
Reply
#8
you know how to fix the problem with the code 18 that works normally on the android cell phone where the god version is downloaded and the zip does not pull it as needed 17.6 thanks to normal thank you
Reply
 
Thread Rating:
  • 0 Vote(s) - 0 Average



Logout Mark Read Team Forum Stats Members Help
FileIO on zip files Fails ARM v1800