Metric to decide on compatibility?

We didn’t add this in the end as we couldn’t think of the best way to do it, but if anyone has any ideas…

Basically we’re looking for a score that tells us how likely a gem is to be compatible, now obviously we could do the number compatible divided by the number incompatible but then that’s likely to penalise the gems that have just been fixed.

One idea was to only count the last X (say, 10) comments towards the score, but any other ideas welcome.

Another is: fail is a -1, works is a +1, but weighted over time (so stuff that was recorded ages counts less than the stuff recorded yesterday)

Nice idea.

Possibly relevant:

“What we want to ask is: Given the ratings I have, there is a 95% chance that the average rating is at least what?”

http://www.evanmiller.org/how-not-to-sort-by-av…

still probably want to weight over time too.

Heh – I was just looking at the site and thinking we need to address this pretty soon. I’ve noticed a few (such as http://isitruby19.com/redcloth) where the initial reports are simply out of date now.