Poor reviewing of Suggested Edits

Question

I find consistently poor reviews in the Suggested Edits queue, by a given reviewer.

I recently voted to reject a particularly unnecessary/harmful edit, and found that it was voted for approval by the reviewer mentioned above. No big deal, if this didn’t happen so frequently, but it does.

I don’t mean to call anyone out, so I’ve scratched out the names, but to compare the stats:

• I seem to reject ~14%
• Another rejects ~20%
• The third rejects ~2%

My question comes down to this: Does anyone review the reviewers?

And, I would suggest a feature that gives notification to a reviewer if their review statistics grossly deviate from the average (after the user has submitted enough reviews for a fair calculation, perhaps 100).

---

edit

Here are a couple of examples:

(comment/answer posted to the question)

(original post deleted and replaced with spam)

Answer 1

My answer is: YOU seem to be reviewing the reviewers, and that's great! That's what community involvement is all about.

There are a couple of ways of dealing with this.

• You can roll back the edit, optionally leaving a comment explaining what was wrong with it. However, this doesn't get back to the reviewer who mistakenly approved the edit.

• If you're seeing a pattern of this behavior, raise a flag so the moderators can take a look at the specific situation. But I know I'm not going to look through several hundred approved edits to determine which ones should not have been, so you're going to have to help us out in that regard.

Answer 2

I suggest you bring up a specific review on meta if you think it's particularly bad. I'm not a fan of public tarring and feathering, but I don't expect experienced users who read meta to go on a witch hunt because of a bad review. On a plus side, we would be able to discuss a specific example much more meaningfully, and the user in question would have a chance to explain their reasoning.

Statistics you have presented are rather meaningless: a Mr. Positive who only ever clicks on "Accept" or "Skip" would be deemed the worst reviewer of all times, yet they may be doing a very good job of getting good edits approved quickly.

Answer 3

I have lots of experience of this from SO, after doing thousands of reviews there. They have exactly the same problem with sloppy reviewers, though on a much bigger scale. Particularly when it comes to suggested edits. The term used is "robo reviewers", people who just click approve without thinking (check the term on https://meta.stackoverflow.com/ and you'll find plenty about the subject). The root of the problem is that users get a bit of reputation and badges for reviewing.

There are measures in place - the audit reviews. Failing enough of these will make you automatically banned from reviewing. First ban, a short period of time. Second ban, longer, etc. The audits are however easy to dodge for someone gambling the system on purpose.

Normal accept/reject ratio should from somewhere around 70/30 to 30/70, depending on how pedantic the reviewer is. People with accept ratio over 90% are clearly doing a poor job, because edits are rarely consistently of that high a quality.

Bringing the specific matter up on meta isn't really a good idea, since this should be handled by diamond moderators rather than "lynch mobs" of meta users.

---

What you can do upon finding that a review was incorrectly accepted:

• Check the accept/reject ratio of the users that approved it. Everyone makes mistakes, but someone with an unusal high accept ratio is a likely robo-reviewer.
• Upon finding someone you suspect of robo-reviewing, check some other reviews they have approved incorrectly.
• If you find a pattern of bad reviewing, flag for diamond moderator attention on the original post where the edit was incorrectly approved. Explain that you suspect user xxx of "robo-reviewing". Post a link to one of the suggested edits they incorrectly approved.
• Diamond mods will take it from there and investigate the user, then hand out a review ban if needed.
• Regardless of how the moderator errand goes (you'll not be notified of how it ended), you can help out by doing rollbacks of all incorrectly approved edits that you found.

Answer 4

Suggested edit stats are available in Moderator Tools, (https://electronics.stackexchange.com/help/privileges/moderator-tools), available to users at 10K rep.

Those tools have limited functionality. I find them useful to keep an eye on migrations when users bring up issues on meta, but beyond that, the tools aren't something I review with diligence. They're not particularly user friendly, and sometimes (like suggested edit review) the views/snapshots provided are not particularly useful.

I suggest that community moderation holds here. As Dave Tweed suggested, if you think an edit is bad, roll it back or edit the post further, or flag it. If that doesn't do it, Moderator's chat or meta is probably the right place. Also, aggressive reviewers tend to trail off after they've got the badge.

It's a bit difficult to criticize someone who feels like they want to sit on the review queue. It's a thankless task, and it's nice that new users feel like their suggestions will be acted upon in a timely fashion (whether rightly or wrongly)

Answer 5

The idea of needing several votes to approve an edit is to reach some sort of consensus among people who have different opinions.

You say you voted to reject a particularly unnecessary/harmful edit, this is your opinion other people have their own. Even though I agree with you in this case I do not think it is necessary to impose reviewing standards on those that have earned the right to express those opinions on the review queues.

The fact that you fell in the middle of the statistics spread in this particular case does not mean that there is no value to the votes of those that are out nearer the edges of the distribution.