My issue looks a lot like the one reported in:
http://forum.xbmc.org/showthread.php?tid=165253 and for which the following git pull request is associated:
https://github.com/xbmc/xbmc/pull/2815
The referenced bug relates to an uninitialized member in the VDPAU code, and I am wondering if the same could hold true for VAAPI or if this is just a coincidence.
I performed some extensive testing on this today & yesterday– re-ripped a feature, watched it through to the end (with VAAPI acceleration), no issues. Also looked at nearly every file in my small but growing MKV library, again with VAAPI enabled (starting/stopping ~30 video files without issues until trying the 'problematic one', which consistently produces a segfault). Not always the first time, but always within 4-5 tries. Also re-ripped this disc to see if it would behave any differently, which (of course) it didn't. When disabling VAAPI (which works ok for every other video), this video works fine.
I did run XBMC through gdb, but this will have to be redone as it did not produce much useable output (not following the threads), nor produce a stacktrace which would give any indication as to what is failing. GDB syntax welcome.
Including it here anyway.
Code:
(gdb) run
Starting program: /usr/lib/xbmc/xbmc.bin "-fs"
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe7128700 (LWP 6969)]
[New Thread 0x7fffe6927700 (LWP 6970)]
[New Thread 0x7fffe58fd700 (LWP 6973)]
[Thread 0x7fffe58fd700 (LWP 6973) exited]
[New Thread 0x7fffe58fd700 (LWP 6975)]
[New Thread 0x7fffd7fff700 (LWP 6976)]
[New Thread 0x7ffff7fac700 (LWP 6979)]
[New Thread 0x7fffd4b47700 (LWP 6980)]
[New Thread 0x7fffc7fff700 (LWP 6981)]
[New Thread 0x7fffc77fe700 (LWP 6982)]
[New Thread 0x7fffc6ffd700 (LWP 6983)]
[New Thread 0x7fffc67fc700 (LWP 6984)]
[New Thread 0x7fffc5ffb700 (LWP 6985)]
[New Thread 0x7fffc57fa700 (LWP 6986)]
[New Thread 0x7fffaffff700 (LWP 6987)]
[New Thread 0x7fffaf7fe700 (LWP 6988)]
[New Thread 0x7fffaeffd700 (LWP 6989)]
[New Thread 0x7fffae7fc700 (LWP 6990)]
[New Thread 0x7fffadffb700 (LWP 6991)]
[New Thread 0x7fffd4346700 (LWP 6992)]
[Thread 0x7fffc7fff700 (LWP 6981) exited]
[Thread 0x7fffc67fc700 (LWP 6984) exited]
[Thread 0x7fffc5ffb700 (LWP 6985) exited]
[Thread 0x7fffc6ffd700 (LWP 6983) exited]
[New Thread 0x7fffc6ffd700 (LWP 6993)]
[Thread 0x7fffd4346700 (LWP 6992) exited]
[Thread 0x7fffc6ffd700 (LWP 6993) exited]
[New Thread 0x7fffc6ffd700 (LWP 6998)]
[New Thread 0x7fffc5ffb700 (LWP 6999)]
[New Thread 0x7fffc67fc700 (LWP 7000)]
[Thread 0x7fffc67fc700 (LWP 7000) exited]
[New Thread 0x7fffc67fc700 (LWP 7001)]
[Thread 0x7fffc67fc700 (LWP 7001) exited]
[New Thread 0x7fffc67fc700 (LWP 7002)]
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
[New Thread 0x7fffc7fff700 (LWP 7003)]
[New Thread 0x7fff924f8700 (LWP 7009)]
[New Thread 0x7fff91cf7700 (LWP 7010)]
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
[Thread 0x7fffaffff700 (LWP 6987) exited]
[Thread 0x7fffc6ffd700 (LWP 6998) exited]
[Thread 0x7fffaeffd700 (LWP 6989) exited]
[Thread 0x7fffc5ffb700 (LWP 6999) exited]
[Thread 0x7fff91cf7700 (LWP 7010) exited]
[Thread 0x7fff924f8700 (LWP 7009) exited]
[Thread 0x7fffc67fc700 (LWP 7002) exited]
[New Thread 0x7fffc67fc700 (LWP 7032)]
[Thread 0x7fffc67fc700 (LWP 7032) exited]
[Thread 0x7fffc7fff700 (LWP 7003) exited]
[New Thread 0x7fffc7fff700 (LWP 7034)]
[New Thread 0x7fffc67fc700 (LWP 7037)]
[New Thread 0x7fff924f8700 (LWP 7038)]
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
[New Thread 0x7fff91cf7700 (LWP 7039)]
[New Thread 0x7fffaeffd700 (LWP 7040)]
[New Thread 0x7fff931fd700 (LWP 7041)]
libva info: VA-API version 0.34.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_34
libva info: va_openDriver() returns 0
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffaeffd700 (LWP 7040)]
0x0000000000d9d875 in _aligned_free(void*) ()
(gdb) backtrace
#0 0x0000000000d9d875 in _aligned_free(void*) ()
#1 0x00000000008f4a76 in CDVDDemuxUtils::FreeDemuxPacket(DemuxPacket*) ()
#2 0x0000000000ef3520 in CDVDMsgDemuxerPacket::~CDVDMsgDemuxerPacket() ()
#3 0x0000000000ef3559 in CDVDMsgDemuxerPacket::~CDVDMsgDemuxerPacket() ()
#4 0x0000000000eda679 in IDVDResourceCounted<CDVDMsg>::Release() ()
#5 0x0000000000ee8a0f in CDVDPlayerVideo::Process() ()
#6 0x00000000011087e0 in CThread::Action() ()
#7 0x0000000001108a92 in CThread::staticThread(void*) ()
#8 0x00007ffff55cff8e in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ffff1378e1d in clone () from /lib/x86_64-linux-gnu/libc.so.6