What is the best way to compare probabilities?
If you have two different events with different (known) probabilities, what is the best way to compare the probabilities?
For example, the relationship between $0.5$ and $0.7$ is not the same as the relationship between $0.79$ and $0.99$. In both cases, the second number is $0.2$ more than the first; however, raising the probability of something from $0.5$ to $0.7$ will not have nearly as much of an effect as raising it from $0.79$ to $0.99$.
To me, it seems like there should be some sort of precise way to compare probabilities, but I have not been able to find one. I want to be able to quantify the comparison in a way that makes sense. Given two numbers, there are countless ways to compare them (subtraction, division, logarithms, etc), but I want to know which comparisons make the most sense in the context of probability.
My current idea is converting the probabilities to z-scores, so $(0.5, 0.7)$ becomes $(0,0.52)$, which is a gain of $0.52$. Also, $(0.79,0.99)$ becomes $(0.81,2.31)$, which is a gain of $1.5$.
$\endgroup$ 21 Answer
$\begingroup$Instead of the probability of an event occurring, you can equivalently consider the probability of the event not occurring. Thus a reasonable requirement for any comparison of probabilities is that you should get exactly the opposite comparison result if you compare the complements of the probabilities. That is, if $f:[0,1]\times[0,1]\to\mathbb R$ is to provide a comparison $f(p_1,p_2)$ between two probabilities, we should have $f(p_1,p_2)=-f(1-p_1,1-p_2)$. A natural ansatz for $f$ might be $f(p_1,p_2)=g(p_2)-g(p_1)$ with $g:[0,1]\to\mathbb R$ some transformation of the probabilities. Then the requirement becomes $-f(1-p_1,1-p_2)=-g(1-p_2)+g(1-p_1)\stackrel!=g(p_2)-g(p_1)$, and thus $g(p)=c-g(1-p)$.
One transformation that fulfills this requirement that has some nice properties and exhibits the behaviour that you want for your example is $g(p)=\log p-\log(1-p)=\log(p/(1-p))$. For your example, the result would be $f(0.5,0.7)=\log(0.7/0.3)-\log(0.5/0.5)\approx0.85$ and $f(0.79,0.99)=\log(0.99/0.01)-\log(0.79/0.21)\approx3.27.$
$\endgroup$ 1
