Peer Review Is Flawed

An interesting piece from TEAMVOYAGR over at CacheMania on the Peer Review system over at Opencaching.COM. This review system has been debated at length for quite sometime, over TEAMVOYAGR have an interesting example of one of it’s major flaws:

The first listing I looked at had a score of -126.  I thought perhaps this was a poorly executed cache so I looked at the listing.

Peer Review of Opencaching geocache OXZTMNT
I immediately assumed there must have been LOTS of reviewers marking it down, so was surprised to find that over half of the -126 was from the 4 you can see in the image. That’s some pretty serious reviewer weight being thrown around there.
In reading the reviewer comments all them them used the same option: From looking at the map, it looks like the coordinates might not be correct. Please double check them.  Well how bad are those coordinates?  I did what you would do and looked at the map.  I zoomed in as close as I could and sure enough the cache looked like it might be in the river.   Then I looked at the scale on the map.   Based on that scale the cache is about 5-6m in the river.   I don’t know about you but at best I get 3m accuracy on my GPSr and in most cases it’s closer to 5m.   Add in the fact that the arial imagery may be off slightly doesn’t that make it it more likely that this cache is in a tree than in the water?

It’s common sense really. How many times has a map shown the cache to be in one place and then you find it is somewhere else entirely?

And how often are the maps wrong? I recall a conversation on a forum regarding a recent geocache bomb scare in the US where the cache location had been viewed on Bing Maps. It showed a clear section, which lead to lots of questions as to why there was a problem…until someone noticed that the satellite imagery was from 2010 and since then an extensive shopping precinct had been built!

So I’m a little concerned that peer reviewers with so much weight are seemingly not taking this into account.

Why do I think the cache is in a tree?  Well because the cacher left a reviewer note that said “small pill bottle in hole of tree about 61/2 feet high”.    To be fair I don’t know if the reviewer note came before or after the cache was reviewed.  In either case wouldn’t it be prudent to revise a vote if there is a better understanding of the cache placement?

And that’s where the GC.com reviewing system works and the OX.com peer reviewing doesn’t. A reviewer and CO can have a discussion on the coordinates. In this case I can imagine the conversation might go something like this:

Reviewer: The coordinates appear in the water. Are you sure they are correct?
CO: The tree cover next to the river is pretty heavy but I averaged the coordinates with my Oregon 550.
Reviewer: Great. Perhaps a note in the listing saying there is no need to go in the water would be useful.
CO: Okay, no problem.
Reviewer: Published!

Surely that is more constructive than a CO having to go back, check coordinates and then convince not one but many peer reviewers, some of whom may not have been involved in the original attempt to list, that the coordinates are correct, especially if the coords still show as being in the water!



Skip to comment form

  1. sTeamTraen

    The whole “peer” (I prefer to call it “multi-stranger”) review concept is totally broken, right out of the box. As pointed out, there is the impossibility of having a discussion with the cache owner, who may be 5000 miles away, speak another language, and not want to talk to 10 people about the same issue with the cache (which in any case is a non-issue). Determining if a cache meets the guidelines or not is not generally a matter of democracy anyway.

    About 87% of caches at OX have OX codes which exactly correspond to published GC codes, so these caches have already been reviewed by a Groundspeak reviewer. Maybe Garmin could start these off with a review value of +300. 🙂

    Of course, the biggest problem with Garmin’s review system is that it makes it trivial to get the FTF. The #review page is accessible to anyone, and they can see every detail of your cache – in fact, with the new reviewer note system, they can see more about your cache before it’s published than afterwards (good thinking there, Garmin). Every cache is, in effect, “published” before it’s published.

    For example, imagine what would happen in the unlikely event that OX becomes sufficiently popular that someone holds an OX-only event with specific OX-only caches for that event. At Geocaching.com., you can ask your reviewer to publish the event caches at a specific time, and in my experience, the reviewer will generally try to oblige, within reason. At OX, for a 9am publication, you’d have to submit all the caches at exactly 9pm the evening before, and hope that /a/ nobody votes them down and /b/ nobody goes out and logs the FTF at 9:30pm.

    Anyway, all of this is academic. Since I wrote my guest piece here five weeks ago, less than 650 “new” (mostly cross-listed) caches have been reviewed and published at OX – that’s under 20 per day – while nearly 10 per day have been archived. But the OX fans all assure me that, despite the growth in cache numbers being at its lowest since the site was launched, this is all due to the site’s newness, and thousands of people will be flocking there Real Soon Now.

  2. Atlas Cached

    Another fantastic (non) fact filled smear of OpenCaching.


    FYI, once the CO sees the negative reviews and the reasons for them, they can make necessary corrections and leave an updated “reviewer note” for all to see, and the reviewers can change their votes accordingly.

    Perhaps you could try producing more objective reviews in the future.

    1. Cumbyrocks

      Thanks! 🙂 We aim to please.

      And again, they have to convince multiple ‘reviewers’, who don’t have the same requirements as GC.coms volunteers reviewers, that they are wrong and that they should come back and change their vote. I’d sooner have teeth pulled (and I’m not allowed anaesthetic so that means something).

      As for objective reviews – this is a blog and we are all humans, thus every comment, regardless of how ‘objective’ it seems, comes from a individuals subjective point of view, their own reality. By way of example: your seemingly unbridled passion for defending Opencaching.com.

    2. sTeamTraen

      Doubtless some balance will be struck when OpenCaching.com’s replies to NATN’s questions are published here, which I’m sure will be Real Soon Now.

    3. Michael Rogers

      Why do I feel the need to discount what you say because you are a bought and paid for cheerleader, er, ambassador for OC.com? Your opinion is hardly the mark of objectivity. This topic’s statements are spot on about OC.com…I’m sorry you’re too blind to see it.

  3. CraigRat

    Reviewing in all forms is flawed.
    I am yet to be convinced that reviewed caches on ANY site are of some superior quality to caches on unreviewed sites.

    Give people the tools they need to make their decisions themselves to self publish.
    A lot of the more bone-headed mistakes can be picked up programatically, and you can always add prompts to encourage more verbose descriptions and also flag proximity issues where known….. it’s not rocket science, some of us have managed to do it for years now 🙂

    Yes, people will abuse the system, but usually they get bored after placing one or two ‘naughty’ caches.

    Obviously, running a site that has no review has probably got me biased a tad 🙂

    I agree that having *multiple* people being able to review a cache up or down is a useless thing, especially considering some of the douchebaggery I’ve witnessed on OX.
    I’ve watched the site ‘grow’ and evolve since it’s inception and the users there that dominate the discussion seem to be very unpleasant……. just my observation….

    1. sTeamTraen

      Sites other than Geocaching.com can probably function OK without reviewers because they tend to have mainly fairly smart, experienced cache placers, whereas newbies tend to land at Geocaching.com, where they either become experienced cachers or lose interest. I would guess that >90% of people who use sites other than Geocaching.com are people who started off with Groundspeak, and perhaps have some issue with how their version of the game is run, but otherwise have a pretty good idea how to place a fairly decent geocache.

      Having seen some of the “WTF?” attempts at “geocaches” which people who are brand new to the game submit to Geocaching.com, I can say with some confidence that the most dominant site, whichever it is, will always need a review process. For one thing, without reviewers, a high-traffic site like Geocaching.com would quickly be inundated with spam caches, listed only to sell Viagra or pr0n sites. In general, you don’t see the problems which you don’t have.

      It’s kind of like running a very select restaurant, at the end of a long driveway out in the country. You probably won’t have too many homeless people coming in off the street and trying to score a free bag of fries. McDonald’s, or the local diner, has to be able to handle those people as well as keeping their customers happy.

Leave a Reply to Michael Rogers Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Switch to mobile version