[topfs2]

Nice calculations And you bring up a good discussion about "randomly" selecting movies/episodes. Neither service has an API for random items, so that could be difficult. Besides, is P(time|movie) not biased to begin with, given that a user is more likely to download a popular movie than an unpopular movie? Go to a torrent tracker, and you'll clearly see that popular movies are downloaded more, and therefore contribute heavier weights to the time histogram.

If we're calculating P(movie|time) and P(time|movie), I would argue in favor of using the GSOC statistics over random movie samples, precisely because their bias is directly in the domain of the evidence we're considering, and it only helps push the results closer to the truth.

That is a very valid point. And with that stated it might actually be better to use the stats then. Sadly I have none on episodes though (limitation of json rpc at the time)

Anyways, topfs2, I'd like to know, what's the next step for the little-scraper-that-could?

I want to do two things. A) solve the complex object problem https://github.com/topfs2/heimdall/issues/7 as that would solve so many problems and then B) get it in xbmc repo as a library.

I think A and B could be done parrallell so nothing really stopping any of them?

My guess is something like a recursive partition tree would work well for these data (rpart in R), though even something like naive Bayes would probably work out OK. You have some very good discriminators for episodes vs other (based on regexp matches) and discriminating musicvideos vs movies is pretty clear cut based on time I should think. Long-format music videos (concert footage etc.) will be trickier - things like audio track format might be useful here (higher quality track more indicative of a concert?) Further, things like foldernames can be grepped for typical things like "Movie", "Music", "Concert", "TV", "Show" (perhaps with other common languages included) which would help. You'd maybe use something like fstrcmp() and take the maximum value from each portion of the path?

Yeah getting the extra features out of the path was the main goal of the stats gathering really. Right now we have the paths and know what title and type it is. Then we can extract a bunch of features regarding that probably. P(Movie|MovieInFilePath) for example would be very doable to gather

topfs2: perhaps you could summarise what information (predictors) you have for each item that might be useful? Is it essentially all the DB variables? What else on top of the DB stuff?

https://github.com/topfs2/script.statist...tion.py#L6

Thats the ones available. These stats was mostly geared towards getting enough information to test/train filepath extraction really.
But now when jsonrpc is even more powerful we could perhaps do a second round?, after we have analyzed this completely