False discovery rates FDR

Link: P-value

The main idea §

This is one of the best ways to prevent P-hacking.

The concept is to do some simple math to adjust the p-value for all tests. After the adjustment, some False Positives in p-value result with small p-value would be > 0.05.

To refresh what is False Positives:

The term, False Discovery Rate (FDR) is used interchangeably with the “Benjamini-Hochberg method”.

The Benjamini-Hochberg method: sum of p-value histograms §

It adjusts p-values and make them larger to limit the numbers of false positivies by looking at the smallest p-value bin in Distribution of p-value.

Eyeball method: We draw a line at the top of the evenly distributed p-values, and extend the line to separate the false positives from true positives.

The math behind Benjamini-Hochberg method §

1. Order p-values from smallest to largest
2. Rank p-values
3. The largest FDR adjusted p-value
4. The next largest adjusted p-value, which is the smallest of the two options:

   1. The previous adjusted p-value

   2. Adjust p-value based on rank:

      $$\text{current p-value}\times \frac{\text{total num of p values}}{\text{current p-value rank}}$$
5. Repeat step 4