**Link**: Logistic regression coefficients Hypothesis testing and the null hypothesis

## What is p-value?

The **p value**, or probability value, tells you how likely it is that your data could have occurred under the null hypothesis. It’s between 0 and 1.

- Threshold: 0.05
- Can be used to confirm the Z-value
- e.g. When z < 2, and if p > 0.05, it also confirm it’s not statistically significant

### The intuition

If p = 0.05, it means only 5% of the time we would see the null hypothesis, thus, we should **reject the null hypothesis**.

In short:

- p > 0.05: failed to reject the null hypothesis / reject the alternative hypothesis
- p < 0.05: reject the null hypotheses

## Purpose

The p-value tell us **how often we would expect to see the extreme statistic**. It can help us eliminate some noises (e.g. if the coin is fair, if the drug result has other factors).

## Adjust the p-value

A small p-value when there’s no difference is also called **False Positive**. See P-hacking and false positives for more details.
We can adjust the p-value to a smaller threshold, if checking the difference is important to avoid false positive. We can also make the threshold larger, e.g. 0.2 means we are willing to get a False Positive 2 times out of 10 experiements.

## The limit of p-value

P-value does not reflect the **effect size**, aka how different they are. It only tells us **whether** it’s different.

## Two-sided vs One-sided p-value

Usually the *two-sided p-value*(or two-tailed p-value) are mostly common. See details in One-sided p-value(one-tailed p-value).

### When to use which p-value?

Example: Testing the effectiveness of a drugs comparing to current treatment

One-sided p-value is 0.03, while two-sided p-value is 0.06.

Which p-value shoud we use?

The one-sided p-value tests the hypothesis that the drug is **better than** the current. The two-sided p-value tests if the new drug is **better/worse/not significantly different** than the current one. The former is smaller because it doesn’t cover other scenarios.

Therefore, we should use two-sided p-value because we want to find out whether the drug is better or worse. Based on the similar logic, we usually use two-sided p-value whenever we can.

Note: from statistical best practice, we should always **decide what test and which p-value to use before doing the experiment**. Otherwise it would be

**p-hacking**and increase the chance of reporting fake results.

## Distribution of p-value

The distribution of p-value is also different when the samples come from the same distribution vs different distributions.