The main idea
This is one of the best ways to prevent P-hacking.
The concept is to do some simple math to adjust the p-value for all tests. After the adjustment, some False Positives in p-value result with small p-value would be > 0.05.
To refresh what is False Positives:
The term, False Discovery Rate (FDR) is used interchangeably with the “Benjamini-Hochberg method”.
The Benjamini-Hochberg method: sum of p-value histograms
It adjusts p-values and make them larger to limit the numbers of false positivies by looking at the smallest p-value bin in Distribution of p-value.
Eyeball method: We draw a line at the top of the evenly distributed p-values, and extend the line to separate the false positives from true positives.
The math behind Benjamini-Hochberg method
- Order p-values from smallest to largest
- Rank p-values
- The largest FDR adjusted p-value
- The next largest adjusted p-value, which is the smallest of the two options:
The previous adjusted p-value
Adjust p-value based on rank:
- Repeat step 4