What statistical test to be used to get the accuracy between two samples with the same number of trials?

Question

I have a prototype of a device which intends to improve the original one. I need to get the accuracy of my prototype based on the the original. The output for every trial is actually whether passed or failed and not an actual value. How can I get the accuracy and what statistical method should I use?

Answer 1

So I suppose you have your 2 devices, the prototype as well as the original. However, what do you mean by "to get the accuracy of my prototype based on the original"? Do you mean that you want to compare the accuracy of the prototype versus the accuracy of the original and see if there is a significant difference between the 2? If this assumption is correct, then there are many methods one can employ, but I'm just going to state the most common one.

First, since your random variables have only 2 states, ie "Pass" or "Fail", this signals that your random variable is a Bernoulli Random variable, ie they have the bernoulli distribution. P(x=0) = 1-p and P(x=1) = p, ie p is the probability of getting a passed. Now, want you want to do is to compare the p of the prototype (p-prototype) and the p of the original (p-original) and see if there is a significant difference between them. Usually, this sort of test can be done using what is known as the student's t-test (http://en.wikipedia.org/wiki/T-test). However, to perform the student t-test, one have to ensure that the Random Variables that you are testing are normally or gaussianlly distributed. Therefore, you can get an estimate for p-original in the following way, take the original device, peform N (a large number, best to be >30) number of test and assign 0 for failed and 1 for passed. Then get the estimated p-original by taking the sum of the values (0+0+1+1+1+0+.....) divided by N. Now repeat the same thing to get the estimate for p-prototype. So now, you have 1 sample of estimated-p-prototype and estimated-p-original. These estimated values are also random variables, but they can be shown through the law of large number to be normally distributed. You can repeat again to get more samples of the estimated values (roughly >30 as well) and with them you can perform the t-test.