4
\$\begingroup\$

How can we improve the manner in which "poor" questions are dealt with? brings up many interesting points and issues, but I'll focus on one of them.

We see a number of these types of posts a year, and without much variance, they usually hit the "we have a problem" vs "we don't have a problem" answers.

I think its safe to say that the steady participants have a desire to see EESO maintain its standing as a valuable resource to those in need of help. I won't bother (here) about how to best maintain that status -- there are plenty of people with plenty of good ideas about how that should be approached.

The engineer in me, however, wants to put a metric or three on this issue. How do we know if we have a problem?? Can we develop a quick snapshot that we can apply over time that would help us keep track? What would such a mechanism look like?

Personally, I wouldn't look toward something obvious, like percentage of closed questions. I think its somewhat irrelevant. It doesn't make a difference if questions are closed. So long as a constant stream of visitors accompanied by a constant stream of new and good questions keep coming in, I assert that EESE is healthy (though maybe not perfect).

In other words, are there some canary in the coalmine type metrics that we should keep an eye on?

How about something like % of recent non-closed questions with no upvotes? This may help us toward calculating signal-to-noise.

Obviously hits, new users, and active members are something to keep an eye on.

Mabye if we can pin something down, someone familiar with the SE query tools can help make this a reality.

\$\endgroup\$
9
  • 1
    \$\begingroup\$ What about up-downvotes to that type of questions? ;) \$\endgroup\$
    – clabacchio
    Dec 8, 2014 at 15:53
  • 2
    \$\begingroup\$ For those that don't know, there is the StackExchange Data Explorer ( data.stackexchange.com ) where you can execute SQL queries on the entire question and answer base of StackExchange. My SQL query crafting skills are very poor (and I don't really have time to improve them). \$\endgroup\$
    – W5VO
    Dec 8, 2014 at 16:22
  • 3
    \$\begingroup\$ The percentage of questions asked by those below 100 rep might be interesting to compare to other metrics. A high new-user fraction can explain why some other metrics are high. It might also be interesting to track how people participated in the site after their first experience. I suspect we are suffering from more drive-by posters than before. The number of users that don't earn 100 new rep within one month after asking first question might tell us we have a drive-by problem. \$\endgroup\$
    – Olin Lathrop
    Dec 8, 2014 at 16:24
  • \$\begingroup\$ I like @OlinLathrop's idea of the return ratio, although it can be interpreted in different ways. For instance, as a metric of satisfaction (more depending on the site itself) or as an indication of willingness to participate (more depending on the single user) \$\endgroup\$
    – clabacchio
    Dec 8, 2014 at 16:26
  • 1
    \$\begingroup\$ @clabacchio Drive-by posting is a nuisance which increases the required amount of moderation effort and occasionally disturbs peace, it would be nice to know more about it. On the other hand, drive-by posting is largely out of our control or influence. More importantly, if we want to know the actual health of EE.SE, we should be measuring satisfaction, participation, turnover of contributors. How could we do this? We can monitor the monthly Top 100 (or some such stat) and watch for turnover. \$\endgroup\$
    – Nick Alexeev
    Dec 8, 2014 at 18:24
  • \$\begingroup\$ @NickAlexeev I like that one a bunch! Health can be measured by making sure there are new people to take over when the old guard get tired. Monitoring turnover on the top lists should hit that. \$\endgroup\$
    – Scott Seidman
    Dec 8, 2014 at 18:38
  • 2
    \$\begingroup\$ @Scott: No, it won't. By definition there will always be a top 100, so new people will always take over when a top user leaves. The fact that there are always 100 at top tells us nothing. The turnover rate actually tells you exactly the opposite, which is how many of the top 100 leave over some unit it of time. It seems to me that a lower number there would be better. \$\endgroup\$
    – Olin Lathrop
    Dec 8, 2014 at 19:17
  • 1
    \$\begingroup\$ @OlinLathrop -- good point, but I'd still like to see some sort of new blood number. Every successful volunteer board I've ever been involved with worries about replacing active board members. How about a target rate for new 5K or 10K users?? If we see a lull, the new blood isn't coming in. \$\endgroup\$
    – Scott Seidman
    Dec 8, 2014 at 22:35
  • 1
    \$\begingroup\$ @ScottSeidman, I'm new. But if we can classify members into active or inactive, then calculate the median tenure of active members, this would tell us a lot about the turnover. If the median tenure is low, that implies that we have high turnover. Not sure what is possible, what statistics we have access to. \$\endgroup\$
    – mkeith
    Dec 20, 2014 at 5:15

3 Answers 3

Reset to default
2
\$\begingroup\$

What about the metrics used for beta sites? Area51 lists stats that are used to judge the health of a beta. I don't see anything similar for non-beta sites, unfortunately. Here's an example. The stats and benchmark values are:

  • Questions per day: 10 is healthy, 5 or fewer needs work.
  • Questions answered: 90% is healthy, 80% needs work.
  • User base: Number of avid and total users. It's recommended at have at least 150 users with 200+ rep, 10 users with 2000+ rep, and 5 users with 3000+ rep.
  • Answer ratio: 2.5 answers per question is good, 1 answer per question needs some work.
  • Visits per day: 1500 is good, 500 needs some work.

Perhaps the user base numbers could reflect recent activity within the past few months.

\$\endgroup\$
2
  • 1
    \$\begingroup\$ I think those are fine for beta sites, on the way up, but by the time a mature site like EESE falls on the wrong side of these, it's too late. \$\endgroup\$
    – Scott Seidman
    Dec 8, 2014 at 22:31
  • 1
    \$\begingroup\$ +1 to Adam. I was going to propose similar things. Questions answered and Answer ratio are still informative metrics for a mature site like EE.SE. We are already talking about User base in this thread. These can be calculated for a week, month, quarter. We can use different thresholds, or just observe dynamics without thresholds. (Other metrics have issues. Questions per day: not a reliable sign of community health. Visits per day: depends more on search engines than on us, frankly.) \$\endgroup\$
    – Nick Alexeev
    Dec 9, 2014 at 17:32
2
\$\begingroup\$

I think closed questions are important. We should strive to have as few closed questions as possible, but realize that we will never reach 0%. But we don't want a site with 5 closed questions for every good question, so closed rate is important.

However, different close reasons point to different problems.

Different close reasons

(If you're familiar with the closing system you can skip to to sum up.)

  • Duplicates are just mistakes by people who didn't search carefully enough, but we shouldn't bother too much about them and there's not much we can do about them.
  • Off-topic votes come in different flavours:
    • Bluntly off-topic: question on recommendations for products or places to purchase them as well as questions on the use of electronic devices are mostly cases where people didn't bother to read the help center. It's annoying and if we get a lot of this we could look into giving the help center a more prominent place in the UI for new users.
    • Repair questions could possibly be reworded to become on topic but normally also this is end of story.
    • Wrong network site doesn't happen a lot as far as I know, but again we should look into emphasizing the help center if this becomes a problem.
    • Other reasons: lately many questions have been closed with a comment such as 'what part of the datasheet are you having trouble with?'. I hope moderators have a tool to monitor these other reasons to see if reasons should be added to the list of off topic reasons.
  • Unclear what you're asking are very important. Often, these questions could be rephrased to make it OK. We should therefore monitor how many closes of this type are followed by a substantial edit (whatever that may be) or hopefully a reopening. If this amount is low we should look into ways to show to users that a closed question doesn't have to be the final station.
  • Too broad: similar to the previous one, but less to worry about in my opinion, since less questions that are closed as too broad can easily be rephrased to fit into the SE framework.
  • Primarily opinion-based questions would often be welcome in chat if I'm not mistaken. When I'm voting to close a question as such I always consider adding that as a comment. We could monitor how many users with a question that was closed as primarily opinion-based come to the chat afterwards (good luck writing a data query for that). But anyway I don't see many questions closed for this reason at the moment, so it doesn't seem real important.

To sum up, there are some close reasons we don't need to care about: duplicates and most of the off-topic reasons. Mistakes like that happen and questions closed as such normally can't be rephrased to fit into the framework. Questions closed as unclear what you're asking and too broad could often be rephrased. We should therefore be monitoring in how many cases the OP (or someone else) edits the post after it was closed, and if it was reopened in the end. If this amount is low, we have a problem.

By looking how often users attempt to edit their question, we can see how much they're encouraged to improve this EE.SE's quality, which should be one of our cornerstone values.

This raises another point:

Substantial edits and upvoted comments

Part of what SE is good at is answering questions of specific people in specific situations. However, the network is also intended to provide some kind of more universal information source. To this end, questions but more importantly answers should be edited to provide more information, explain better, etc.

We could therefore monitor the amount of substantial edits (again, whatever that may be) related to either the amount of views (how many visitors decide to improve something) or to the amount of posts (how many posts are getting improved).

Often, information is added not by editing the answer, but by leaving a comment. Typically, comments that provide good information are voted up (we can forget about the upvoted jokes for the moment). We could monitor the average amount of comments with some minimum score per post (possibly, comments on answers could be more important than comments on questions).

Other points

I've mentioned two things that weren't mentioned before, but don't want to claim these are the only things we should monitor, or even that we definitely should monitor these. I just wanted to open up the discussion a bit.

I like the idea from Adam Haun's answer of using the statistics used for beta sites, but I'm not sure if they address the kind of problem we have here (which is, mainly, bad questions).

I also like Olin Lathrop's idea of separating statistics on reputation to see how we can help new users get to know the site better.

As a last point, I would say that when many questions of the type 'do we have a problem?' are raised, this may definitely be a pointer to us having a problem.

\$\endgroup\$
3
  • 1
    \$\begingroup\$ For some reason "Which IC should I use for implementing XYZ" questions are rejected, stating that the information is volatile and doesn't help the site in the long run. I don't think that's the case: when some IC becomes obsolete, someone will eventually write another answer that describes some other IC that is available. The need to implement XYZ is often not volatile; other people might want to do the same and will arrive at the question via google. How do you feel (I'm not ranting this time)? Or is there some other reason to reject the "which chip to use" questions? \$\endgroup\$
    – PkP
    Dec 10, 2014 at 16:52
  • \$\begingroup\$ @PkP interesting point. I would say the problem is that the problem is too localised and the question unlikely to help future visitors. If the question can be edited to a format 'how to do XYZ', without asking for a chip, that would already broaden it. \$\endgroup\$
    – user17592
    Dec 10, 2014 at 17:25
  • \$\begingroup\$ @PkP there used to be a close reason too localised - you can see the recommendations close reason as its successor in my opinion, although the change was more complicated. \$\endgroup\$
    – user17592
    Dec 10, 2014 at 17:26
1
\$\begingroup\$

I answered already, but this is a different matter and I didn't want to make that wall of text even longer.

On the Math.SE Meta they built a 'citizenship query': a query resulting in people who put the most effort in the site, with some score. This score is of course based on reputation, but also on badges for things you don't earn reputation with (reviewing, edits, flags, etc.).

The original Math.SE Meta post is here: How much experience should a potential moderator have? (Math.SE is currently in an election).
And here is the data query: Citizenship score based on certain badges and reputation

I'm not sure how we can use this to see whether we have a problem yet. But perhaps individual users can use this to see if they have a problem. Note that history is not taken into account (for example, reputation earned 5 years ago weighs as much as reputation earned yesterday, and the same goes for badges), which is a big con about this query, in my opinion.

With some modifications we could maybe use such a query also to get a citizenship score for the whole site.

\$\endgroup\$
3
  • 1
    \$\begingroup\$ I agree that lack of time-awareness is a big con. For example, Area51 has a commitment score for proposals, which decays at a rate of 10% over 6 moths. For more details, see update #2 in this meta.SE post. I wouldn't apply decay to badges, though, because most of the badges used in this citizenship score calculation can be awarded only once. \$\endgroup\$
    – Nick Alexeev
    Dec 16, 2014 at 0:54
  • \$\begingroup\$ There is some sort of "Active User" status that they apply to the longevity badges. What does it mean, and can we make use of that? \$\endgroup\$
    – Scott Seidman
    Dec 16, 2014 at 20:29
  • \$\begingroup\$ @ScottSeidman you mean the yearling badge? I think it doesn't mean anything there, it just means 'be a user for a year and earn 200 rep', but I can't find any reference on meta.SE for that. \$\endgroup\$
    – user17592
    Dec 16, 2014 at 20:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .