March 28, 2024
Lies, damned lies and statistics
the claims are true, actually!
Did you arrive here from Instagram? Don’t buy the 100% specificity bit? Don’t believe that the algorithms are extremely precise?
Sorry, boring math coming, leave now if you don’t like math.
Precision and Accuracy
This one is pretty basic, precision refers to repeatibility and reproducibility of the test. Dig out the high school math text book and find the picture of the dartboard. All the darts are close to each other, but nowhere near the target. The picture checker on the home page only rejects photos. The results are reproducible every single time, but what is its accuracy? Read on!
Sensitivity
Ah, this one is a bit more involved.
If we consider the tool’s ability determine whether a photo will actually be accepted or rejected by airliners.net (gotta spread the love here, too Jetphotos centric here); This can be represented by a 2 x 2 table of results. Let’s assume 100 photos are screened with the tool:
and let’s use that 60% acceptance rate
60 photos will be accepted at A.net and 40 will be rejected. This is represented by the total at the bottom of the table. If every one of those photos is screened with the tools, we get the following:
Outcome | Accepted at A.net | Rejected at A.net | Total |
---|---|---|---|
Accepted by jpscreener | 0 (True Positive) | 0 (False Positive) | 0 |
Rejected by jpscreener | 60 (False Negative) | 40 (True Negative) | 100 |
Total | 60 | 40 | 100 |
The tool simply rejects every photo (this is represented the right column).
The specificity of a test is the True Negative / (False Positive + True Negatives). In other terms: “How many pictures rejected by the actual site are rejected by the jpscreener tool?” All of them! 100% Specificity!
Wait a second, doesn’t that mean the Sensitivity of the test is 0%? Also yes, but what’s the fun in advertising that?
Accuracy
Note that nowhere do we mention the accuracy of the tool. This is a different value, which is the quotient of correct test results and the total tested: True Positive + True Negative / (The total). The accuracy of the tool is actually dependent on the rejection rate at the real site. In this example, the accuracy is only 40% (again, why would we advertise that!?)
Enjoy!
Have fun, have some tweaks or suggestions or did I do the math wrong? Contact us.
Should I give it a 0.0001% of an acceptance a la the Wonka Golden ticket? Maybe one day!