When you start doing more serious research you will need to conduct a statistical test. It helps you to determine weather you hypothesis is significant or can be rejected. If you start browsing for statistical tests online, you will find a large number of possible test that can be chosen depending on you task. In this post lets focus on simple example of statistical test that would let us understand what is statistical significance and what is confidence interval.
Building a test scenario
In order to understand the matter lets use some random data which is distributed normally. For this we can use 100 hypothetical observations of price tag for new Nokia 3310 from different sources. The average of price is around 50€ with standard deviation of 10€
Now lets plot how those samples are distributed along with normal distribution graph:
As you can see the probability of distribution is Gaussian with mean of 50 standard deviation 10. For later intuition we will be using this Gaussian distribution expressed as probability density calculated by formula:
We already know mean and standard deviation from out data, so calculation and plotting is easy.
Now we can get back to statistical test.
For sake of simplicity lets say after one a year you decide to recheck the prices of Nokia 3310 and find that it average dropped to 23€.
Can we state that price is different? From the first glance it looks very different, but what if collected data sample is different? What if you collect same size of data sample and you get average price closer to 50€? The statistical test helps answering question weather the price is significantly different from previous – in other words is sample mean significantly different from null hypothesis. So how to draw a line where we can state that result is significantly different?
In order to have some limits we choose some significance level α which indicates the risk that the difference exists even if there is no difference. Normally this level is chosen as α = 0.05. This means that there is 5% probability of rejecting null hypothesis. Humanly speaking if we think that average price of Nokia have dropped after year, there is still 5% probability that it’s not.
So now when we have significance level we can draw those limits on distribution graph. On distribution graph Y axis is representing probability distribution, we need to calculate the area under graph to find actual probability limits. Since graph is double sided we need to split 0.05 significance level to both sides having 0.025 for each side. Total area under graph is equal to 1. In zoomed graph portion you can see that significance level is at price around 22€:
Because price our observed price is 23€ doesn’t fall in this critical region, so we must state that price didn’t change. So we cannot reject null hypothesis of average price being 50€. In other words, our observed average price of 23€ isn’t statistically significant to reject null hypothesis. Significance level can be different, for instance if we choose significance level α = 0.1, then definitely our observed new price will fall in to risky area and so we could state that price changed significantly.
Probability value P
It seems that we have covered the basics, but there is still one more important thing to discuss – the P value. Practically speaking P represents probability of obtaining observed data (in our case price being 23€). so we need to find area under distribution graph up to point 23€. One side are is 0.0023. Add bots sides we get 0.052. This is the value of P = 0.052.
Now we can rephrase our investigation. Comparing null hypothesis of Nokia price being 50€ to recent observation of average price 23€ we find that probability P=0.052 of price is lower than significance level α = 0.05 and so we state that we stick to null hypothesis and say that price isn’t lower that 50€. Hope this makes sense.
Final thoughts about statistical significance
As mentioned before, significance level isn’t fixed value. Before you test your hypothesis you must choose your level. The common ones are 0.05 and 0.01. You choose significance level with the thought of error rate that you reject null hypothesis even if it is true. So P value and significance level help you to quantify the results of observation and control the error in hypothesis test.
Sometimes instead of significance level e.g. 0.05 you will find notation of confidence interval equal 95%. This is practically same if you subtract 0.05 from 1 you get 0.95.
Confidence interval indicates uncertainty interval around null hypothesis point (50€) or in other words margin error.
You can play around with data above with this [Matlab live editor file.zip]