15
\$\begingroup\$

I find consistently poor reviews in the Suggested Edits queue, by a given reviewer.

I recently voted to reject a particularly unnecessary/harmful edit, and found that it was voted for approval by the reviewer mentioned above. No big deal, if this didn’t happen so frequently, but it does.

user stats

I don’t mean to call anyone out, so I’ve scratched out the names, but to compare the stats:

  • I seem to reject ~14%
  • Another rejects ~20%
  • The third rejects ~2%

My question comes down to this: Does anyone review the reviewers?

And, I would suggest a feature that gives notification to a reviewer if their review statistics grossly deviate from the average (after the user has submitted enough reviews for a fair calculation, perhaps 100).


edit

Here are a couple of examples:

approved 1 (comment/answer posted to the question)

approved 2 (original post deleted and replaced with spam)

\$\endgroup\$
  • 5
    \$\begingroup\$ And does anyone review the reviewers that review the reviewers? *Takes a blunt* \$\endgroup\$ – Harry Svensson Oct 8 '18 at 8:27
  • 2
    \$\begingroup\$ Where did you find the Reviewer Stats? \$\endgroup\$ – pipe Oct 8 '18 at 11:30
  • 2
    \$\begingroup\$ @HarrySvensson Great follow-up question. \$\endgroup\$ – Blair Fonville Oct 8 '18 at 12:15
  • \$\begingroup\$ @pipe Try this: go to the suggested edits queue; click the history tab; click “approve” or “reject” from an old review; in the box where it shows the people who reviewed the question, click “more”. \$\endgroup\$ – Blair Fonville Oct 8 '18 at 12:19
  • \$\begingroup\$ Really, imho, don't bother, look at other stacks, there everything gets approved because there are countless people sitting in the review queue robo approving everything, and in the end it gets enough approval votes anyways. And then you get these tests where you should have approved/rejected whatever an edit, but where not perfectly in line with whoever set them up, and boom you lost priviliges. Isn't worth it. \$\endgroup\$ – PlasmaHH Oct 8 '18 at 15:37
  • 4
    \$\begingroup\$ With the added examples it's clear that this "reviewer" is abusing the website and should just be blocked. \$\endgroup\$ – pipe Oct 11 '18 at 9:36
  • 2
    \$\begingroup\$ Well your second example is not the first time it was rejected! The question about the best tool to hold a PCB had the same edit that myself and another reviewer rejected, yet someone else approved. I assume it was the same reviewer as your '2% rejected' reviewer as the approve/reject numbers are almost identical when I look. It seems that this person is likely just hitting accept without reading \$\endgroup\$ – MCG Oct 11 '18 at 13:47
10
\$\begingroup\$

My answer is: YOU seem to be reviewing the reviewers, and that's great! That's what community involvement is all about.

There are a couple of ways of dealing with this.

  • You can roll back the edit, optionally leaving a comment explaining what was wrong with it. However, this doesn't get back to the reviewer who mistakenly approved the edit.

  • If you're seeing a pattern of this behavior, raise a flag so the moderators can take a look at the specific situation. But I know I'm not going to look through several hundred approved edits to determine which ones should not have been, so you're going to have to help us out in that regard.

\$\endgroup\$
  • 2
    \$\begingroup\$ Great points. On the final point, if the mods had tabularized user-review statistics it would be easy to see those who are routinely make bad calls. It may not be a big deal, but I consistently see very obvious calls being approved by the user alluded to in the question. \$\endgroup\$ – Blair Fonville Oct 7 '18 at 15:18
  • 1
    \$\begingroup\$ @BlairFonville I would agree there. I deduced who the user was based on review numbers and looking at the stats of edits I had personally rejected and by the look of their activity, it is just badge farming, which is fine, but when it comes to making consistent bad calls in reviews, then it can be considered poor behavior as in my view it is almost intentionally mucking up questions \$\endgroup\$ – MCG Oct 11 '18 at 14:03
5
\$\begingroup\$

I suggest you bring up a specific review on meta if you think it's particularly bad. I'm not a fan of public tarring and feathering, but I don't expect experienced users who read meta to go on a witch hunt because of a bad review. On a plus side, we would be able to discuss a specific example much more meaningfully, and the user in question would have a chance to explain their reasoning.

Statistics you have presented are rather meaningless: a Mr. Positive who only ever clicks on "Accept" or "Skip" would be deemed the worst reviewer of all times, yet they may be doing a very good job of getting good edits approved quickly.

\$\endgroup\$
4
\$\begingroup\$

I have lots of experience of this from SO, after doing thousands of reviews there. They have exactly the same problem with sloppy reviewers, though on a much bigger scale. Particularly when it comes to suggested edits. The term used is "robo reviewers", people who just click approve without thinking (check the term on https://meta.stackoverflow.com/ and you'll find plenty about the subject). The root of the problem is that users get a bit of reputation and badges for reviewing.

There are measures in place - the audit reviews. Failing enough of these will make you automatically banned from reviewing. First ban, a short period of time. Second ban, longer, etc. The audits are however easy to dodge for someone gambling the system on purpose.

Normal accept/reject ratio should from somewhere around 70/30 to 30/70, depending on how pedantic the reviewer is. People with accept ratio over 90% are clearly doing a poor job, because edits are rarely consistently of that high a quality.

Bringing the specific matter up on meta isn't really a good idea, since this should be handled by diamond moderators rather than "lynch mobs" of meta users.


What you can do upon finding that a review was incorrectly accepted:

  • Check the accept/reject ratio of the users that approved it. Everyone makes mistakes, but someone with an unusal high accept ratio is a likely robo-reviewer.
  • Upon finding someone you suspect of robo-reviewing, check some other reviews they have approved incorrectly.
  • If you find a pattern of bad reviewing, flag for diamond moderator attention on the original post where the edit was incorrectly approved. Explain that you suspect user xxx of "robo-reviewing". Post a link to one of the suggested edits they incorrectly approved.
  • Diamond mods will take it from there and investigate the user, then hand out a review ban if needed.
  • Regardless of how the moderator errand goes (you'll not be notified of how it ended), you can help out by doing rollbacks of all incorrectly approved edits that you found.
\$\endgroup\$
  • \$\begingroup\$ I'm one of the reviewers who accepts the good ones and skips the bad ones, letting someone with more time deal with those that need help. \$\endgroup\$ – Scott Seidman Oct 12 '18 at 15:23
  • 3
    \$\begingroup\$ @ScottSeidman That doesn't make sense. Accepting and rejecting takes the same amount of time. The only time-consuming part is when there's a half-decent edit but further changes are needed to fix the post, so you have to go in and manually edit yourself. \$\endgroup\$ – Lundin Oct 12 '18 at 17:09
  • \$\begingroup\$ No diff. It's what I do, sensible or not. \$\endgroup\$ – Scott Seidman Oct 12 '18 at 23:16
  • 1
    \$\begingroup\$ @ScottSeidman Anyway, just because some people do like you for whatever reason, we need to peek at their review history before blaming them for robo reviewing, as per my answer. \$\endgroup\$ – Lundin Oct 13 '18 at 11:17
2
\$\begingroup\$

Suggested edit stats are available in Moderator Tools, (https://electronics.stackexchange.com/help/privileges/moderator-tools), available to users at 10K rep.

Those tools have limited functionality. I find them useful to keep an eye on migrations when users bring up issues on meta, but beyond that, the tools aren't something I review with diligence. They're not particularly user friendly, and sometimes (like suggested edit review) the views/snapshots provided are not particularly useful.

I suggest that community moderation holds here. As Dave Tweed suggested, if you think an edit is bad, roll it back or edit the post further, or flag it. If that doesn't do it, Moderator's chat or meta is probably the right place. Also, aggressive reviewers tend to trail off after they've got the badge.

It's a bit difficult to criticize someone who feels like they want to sit on the review queue. It's a thankless task, and it's nice that new users feel like their suggestions will be acted upon in a timely fashion (whether rightly or wrongly)

\$\endgroup\$
1
\$\begingroup\$

The idea of needing several votes to approve an edit is to reach some sort of consensus among people who have different opinions.

You say you voted to reject a particularly unnecessary/harmful edit, this is your opinion other people have their own. Even though I agree with you in this case I do not think it is necessary to impose reviewing standards on those that have earned the right to express those opinions on the review queues.

The fact that you fell in the middle of the statistics spread in this particular case does not mean that there is no value to the votes of those that are out nearer the edges of the distribution.

\$\endgroup\$
  • 7
    \$\begingroup\$ Fair point, but I’ve seen spam, answers made within the question, and entire posts deleted and replaced with a thank you, all approved by the same reviewer. It seems clear that the user isn’t even looking at the edit before clicking the accept button in many cases. Why even visit the review queue if you (in general) have no intention of doing so properly? \$\endgroup\$ – Blair Fonville Oct 8 '18 at 16:29
  • \$\begingroup\$ It takes multiple people to approve OR reject a suggested edit. And you can't do your own suggested edit in the review queue. So it's a non-issue. \$\endgroup\$ – Passerby Oct 9 '18 at 0:26
  • 3
    \$\begingroup\$ @Passerby The issue is that it’s kind of an abuse of the system. We have multiple people vote to form a concensus. Not so one can farm badges while the other two vote. \$\endgroup\$ – Blair Fonville Oct 9 '18 at 3:17
  • \$\begingroup\$ @Blair good thing badges are 100% meaningless. \$\endgroup\$ – Passerby Oct 9 '18 at 5:04
  • \$\begingroup\$ @Passerby Yes, I agree. And again, it’s not the purpose of the review queues. \$\endgroup\$ – Blair Fonville Oct 9 '18 at 5:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .