Statistical quality of experience analysis: on planning the sample size and statistical significance testing

Kjell Brunnström; Marcus Barkowsky

doi:10.1117/1.JEI.27.5.053013

25 September 2018 Statistical quality of experience analysis: on planning the sample size and statistical significance testing

Kjell Brunnström, Marcus Barkowsky

Author Affiliations +

Journal of Electronic Imaging, Vol. 27, Issue 5, 053013 (September 2018). https://doi.org/10.1117/1.JEI.27.5.053013

Abstract

This paper analyzes how an experimenter can balance errors in subjective video quality tests between the statistical power of finding an effect if it is there and not claiming that an effect is there if the effect is not there, i.e., balancing Type I and Type II errors. The risk of committing Type I errors increases with the number of comparisons that are performed in statistical tests. We will show that when controlling for this and at the same time keeping the power of the experiment at a reasonably high level, it is unlikely that the number of test subjects that are normally used and recommended by the International Telecommunication Union (ITU), i.e., 15 is sufficient but the number used by the Video Quality Experts Group (VQEG), i.e., 24 is more likely to be sufficient. Examples will also be given for the influence of Type I error on the statistical significance of comparing objective metrics by correlation. We also present a comparison between parametric and nonparametric statistics. The comparison targets the question whether we would reach different conclusions on the statistical difference between the video quality ratings of different video clips in a subjective test, based on the comparison between the student T-test and the Mann–Whitney U-test. We found that there was hardly a difference when few comparisons are compensated for, i.e., then almost the same conclusions are reached. When the number of comparisons is increased, then larger and larger differences between the two methods are revealed. In these cases, the parametric T-test gives clearly more significant cases, than the nonparametric test, which makes it more important to investigate whether the assumptions are met for performing a certain test.

Citation Download Citation

Kjell Brunnström and Marcus Barkowsky "Statistical quality of experience analysis: on planning the sample size and statistical significance testing," Journal of Electronic Imaging 27(5), 053013 (25 September 2018). https://doi.org/10.1117/1.JEI.27.5.053013

Received: 14 March 2018; Accepted: 17 August 2018; Published: 25 September 2018

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available