[hdmkv]

@hdmkv

I think what people wants to know is how does a device get A+ ratings what are there criteria that makes a device to get a better ratings than others if some of the features are missing.

I did propose a while back that we develop a common rating system. The A- I gave X9S is based on several A/V playback criteria that tally up to that grade. In the end analysis, grades have subjectivity. But, some kind of a common, comprehensive system we can agree on would help. Maybe a technical, purely objective grade (for things like bitrate tests, 3D MVC pass/fail, etc.) and a overall reviewer's impressions grade (for things like PQ, launcher, features).

[movie78]

@hdmkv

I think what people wants to know is how does a device get A+ ratings what are there criteria that makes a device to get a better ratings than others if some of the features are missing.

I did propose a while back that we develop a common rating system. The A- I gave X9S is based on several A/V playback criteria that tally up to that grade. In the end analysis, grades have subjectivity. But, some kind of a common, comprehensive system we can agree on would help. Maybe a technical, purely objective grade (for things like bitrate tests, 3D MVC pass/fail, etc.) and a overall reviewer's impressions grade (for things like PQ, launcher, features).

If the ratings are base on 10 points ratings then this some of my suggestion.

NO NETFLIX SUPPORT = -1 point
NO HDR SUPPORT = -1 Point
NO 3D MVC (MKV/ISO) SUPPORT = -1 Point
PLAYBACK ISSUE = - 2 point
NOT SUPPORTING ALL THE HD AUDIO FORMAT= -2
PICTURE QUALITY = -2


Just a few i can think of.

[ceaca]

hellyeah

[hdmkv]

Now you see why this stuff is hard to standardize. I for one could care less about Netflix for example. PQ is very subjective. Some don't care about HD audio ... and I could go on .

[movie78]

If anyone agrees with me we can start from here.

[couto27]

If the ratings are base on 10 points ratings then this some of my suggestion.

NO NETFLIX SUPPORT = -1 point
NO HDR SUPPORT = -1 Point
NO 3D MVC (MKV/ISO) SUPPORT = -1 Point
PLAYBACK ISSUE = - 2 point
NOT SUPPORTING ALL THE HD AUDIO FORMAT= -2
PICTURE QUALITY = -2


Just a few i can think of.

That way its too complex, Why not divide by categories:

Streaming services ( netflix, amazon hulu etc etc)
Built quality & connections ( design, box, remote, internals, )
PQ quality ( video, HDR, 10bit, TV etc )
Sound Quality (flac, DSD, hd audio, atmos etc )
Features ( 3D, set up, menus options, hdmi, bitstreaming etc )
Playback Performance ( bluray, DVD, TV, DVR, speed, usability, networking, streaming, startup )
Extras e value for money ( gaming, portability etc)
overall

[wrxtasy]

Now you see why this stuff is hard to standardise. I for one could care less about Netflix for example. PQ is very subjective. Some don't care about HD audio ... and I could go on .

Yes. exactly. I could care less about HDR as I've not got the Hardware to support it. I would not care at all for 3D, same as most mainstream Kodi users. Then there is 24p and HD Audio, deinterlacing, the list goes on and on.
Far too hard to standardise.

Its much easier to list Box advertised features that don't work properly.
What are companies selling - does it work properly ?

This is the important stuff that trips up everyday A/V playback and annoys end users the most in the long term, particularly if they have picked a very specific box based on unique advertised feature(s).

[couto27]

Far too hard to standardise.

Not really..the magazine and specialised reviewers are doing it for years..most of the reviewers have THX & ISF certification.

Some gear that every reviewer should have:

Sound Level Meter
Calibrated Microphone UMIK -1+ REW is always handy
Hdmi analyser
Colimeter
test patterns or pattern generator
Reference disks
Reference player
Calibrated display.
CAT5/5e/6 LAN Performance Verifier
and possible some good oscilloscope..

Personally i follow archimago reviews for a couple years for some audio gear.
Secrets of home theather Hifi for everything related with Hifi
audioholics for more Hifi
digital trends for more stuff
Hdtvtest.co.uk for TV/Projector reviews
PJHC.fr for projectors and projector screens
anantech for PC components technical reviews.
and a few other forums..

Most of this editors/reviewers have many years of experience, and sometimes i found them doing comments in avsforum.com about product A or B...

Heres a few examples of standardise methods:
AreaDVD for lastest video & hifi news
how cnet review bluray players.
how digital trends test receivers
Audio tests

[wrxtasy]

Yeah good luck getting a bunch of people with far differing views and requirements to agree on that for any Kodi box.

The scope is too wide for a Kodi Box vs the narrow focus of specialist Magazines or Bluray players, receivers etc. and I doubt anyone is going to be bothered buying specialised Hardware to test numerous Kodi boxes either, especially when they are not paid to do it.

Are you volunteering lots of free time to do this couto27 ?

[Ned Scott]

Its much easier to list Box advertised features that don't work properly.

This + 1000

[BORIStheBLADE]

Now you see why this stuff is hard to standardise. I for one could care less about Netflix for example. PQ is very subjective. Some don't care about HD audio ... and I could go on .

Yes. exactly. I could care less about HDR as I've not got the Hardware to support it. I would not care at all for 3D, same as most mainstream Kodi users. Then there is 24p and HD Audio, deinterlacing, the list goes on and on.
Far too hard to standardise.

Its much easier to list Box [b]advertised features that don't work properly.
What are companies selling - does it work properly ?[/b]

This is the important stuff that trips up everyday A/V playback and annoys end users the most in the long term, particularly if they have picked a very specific box based on unique advertised feature(s).

I agree 100% This should be the baseline for every box.

Maybe make the test/review two parts. One is the baseline, covering everything on the box. Then another that goes over all the other stuff that kodi users ask about. I'm sure people could chime in and we can make a quick list of all the other stuff Kodi users would like reviewed that arent on the box and use that for future boxes. Doing it that way you can knock out the baseline early and post it up first. I notice in the past you have posted up reviews over the course of multiple days..

Just a thought..

[couto27]

Are you volunteering lots of free time to do this couto27 ?

Not really, i wish to have more free time for this hobby.
But if i received any free products, i would need at least 30 days of use before any review.

Aprox 15 days and i would report all the Bugs to the Manufacturer.
Wait 15 days for a fix from Manufacturer
Release the review with score XYZ.... and that score would be final.

What im seeing now is some users doing "extra leg work" (being beta testers instead of being Reviewers) testing every firmware that comes out.... and that requires lots of free time.

[jd17]

divide by categories:

Streaming services ( netflix, amazon hulu etc etc)
Built quality & connections ( design, box, remote, internals, )
PQ quality ( video, HDR, 10bit, TV etc )
Sound Quality (flac, DSD, hd audio, atmos etc )
Features ( 3D, set up, menus options, hdmi, bitstreaming etc )
Playback Performance ( bluray, DVD, TV, DVR, speed, usability, networking, streaming, startup )
Extras e value for money ( gaming, portability etc)
overall

I think this is a good idea and I could imagine a working, point-based rating system if we use categories.

For instance:

Basic Kodi capabilities:
- Flawless H.263/H.264 playback for all framerates, resolutions and bitrates up to High/4.1 profile
- Dolby Digital and DTS decoding
- Proper limited RGB BT.709 output
.
.
.
(A checklist could be used and each checked item is worth 1 point.)

Only if a device meets the basic criteria (for instance 10/10 points in this category), it qualifies for a "regular" recommendation.

Then you add advanced categories:

Advanced video decoding capabilities:
- Deinterlacing...
- HEVC 1080p
- HEVC 2160p
- VP9
- VC-1
- BT.2020
- 10 Bit
- 12 Bit
- HDR10
- Dolby Vision
- Hybrid Log Gamma
.
.
.

Advanced audio decoding capabilities:
- Dolby TrueHD
- Dolby Atmos
- DTS HD Master Audio
- flac
.
.
.

3D capabilities:
.
.
.

Streaming services:
.
.
.


You see where this is going.
If we can agree on a standardized checklist with easily accessible test files (most are available already) - this system would be immensely helpful.

You could have a sum up to easily filter and then go into details if you want to know more.

For instance (just random numbers):

Raspberry Pi 3:
Basic: 10/10
Advanced video: 1/15
Advanced audio: 7/12
3D: 3/5
Streaming: 1/10

NVIDIA Shield TV:
Basic: 10/10
Advanced video: 14/15
Advanced audio: 12/12
3D: 0/5
Streaming: 10/10

So you could immediately exclude one or the other depending on your needs and go into the details if you are uncertain.

What do you guys think?


Sure, wrxtasy has done a lot of this already in his great thread, but a simple point-based system would make things easier for him as well.

[hdmkv]

Its much easier to list Box advertised features that don't work properly.

This + 1000

Agree on this as well. Some kind of point system to subtract for features or functions that don't work as advertised publicly by the manufacturer (-2) or partially work (-1). Box starts w/100 points. We'd probably need to put weight to features/functions as well... as I think we can agree A/V playback/performance is paramount. Less important are things like OpenWRT, rooting, clock/LED, certain peripherals not working, etc.

A secondary rating for reviewer's overall impressions for things more subjective (value for money, 3D depth quality, PQ, and things not advertised but important for the type of box it is). That last bit for Zidoo's X9S would be things like 3D flipeyes, MKV chapters support, 3D subtitles, etc.). Finally, quality of support, with manufacturer responsiveness, is huge.

[movie78]

If the ratings are base on 10 points ratings then this some of my suggestion.

NO NETFLIX SUPPORT = -1 point
NO HDR SUPPORT = -1 Point
NO 3D MVC (MKV/ISO) SUPPORT = -1 Point
PLAYBACK ISSUE = - 2 point
NOT SUPPORTING ALL THE HD AUDIO FORMAT= -2
PICTURE QUALITY = -2


Just a few i can think of.

That way its too complex, Why not divide by categories:

Streaming services ( netflix, amazon hulu etc etc)
Built quality & connections ( design, box, remote, internals, )
PQ quality ( video, HDR, 10bit, TV etc )
Sound Quality (flac, DSD, hd audio, atmos etc )
Features ( 3D, set up, menus options, hdmi, bitstreaming etc )
Playback Performance ( bluray, DVD, TV, DVR, speed, usability, networking, streaming, startup )
Extras e value for money ( gaming, portability etc)
overall

I also like this method.

Now you see why this stuff is hard to standardise. I for one could care less about Netflix for example. PQ is very subjective. Some don't care about HD audio ... and I could go on .

Yes. exactly. I could care less about HDR as I've not got the Hardware to support it. I would not care at all for 3D, same as most mainstream Kodi users. Then there is 24p and HD Audio, deinterlacing, the list goes on and on.
Far too hard to standardise.

Its much easier to list Box advertised features that don't work properly.
What are companies selling - does it work properly ?

This is the important stuff that trips up everyday A/V playback and annoys end users the most in the long term, particularly if they have picked a very specific box based on unique advertised feature(s).

Then you don't care about improvement of technology.