Link: P-value

The main idea

This is one of the best ways to prevent P-hacking.

The concept is to do some simple math to adjust the p-value for all tests. After the adjustment, some False Positives in p-value result with small p-value would be > 0.05.

To refresh what is False Positives:

The term, False Discovery Rate (FDR) is used interchangeably with the “Benjamini-Hochberg method”.

The Benjamini-Hochberg method: sum of p-value histograms

It adjusts p-values and make them larger to limit the numbers of false positivies by looking at the smallest p-value bin in Distribution of p-value.

Eyeball method: We draw a line at the top of the evenly distributed p-values, and extend the line to separate the false positives from true positives.

The math behind Benjamini-Hochberg method

  1. Order p-values from smallest to largest
  2. Rank p-values
  3. The largest FDR adjusted p-value
  4. The next largest adjusted p-value, which is the smallest of the two options:
    1. The previous adjusted p-value

    2. Adjust p-value based on rank:

  5. Repeat step 4