Re: The Quiz Evaluation Discussion/Debate
61. hozzászólás - 2010.05.03. 22:53:05
61. hozzászólás - 2010.05.03. 22:53:05
Here's some thoughts. Hopefully we'll get a good discussion going.
Number one, and I can't stress this enough, Miklos or some other admin should read the forums and engage in the discussion so that the community feels that our opinions are being heard and not just ignored. Looking back, I can see that Miklos used to post a lot more often than he does now, but even then, his posts were not the type that I am looking for here. Experienced players can and do answer questions like why player X can attack me but I can't attack him. What I would like to see is responses to suggestions, such as "I like that suggestion, and I will relay it to the appropriate people" or "That would be nice, but it would cost way too much to implement" or "I think that would make the game worse, not better". I'd even be happy with "Stop bugging me, you pathetic loser", as that would still show that he's paying attention.
Next, I am going to collect all the suggestions I have made in the past on this topic and put them here.
- Require each quiz question to be accepted by 2 out of 3 evaluators. This would require a lot more evaluators, so in practice it would probably mean accepting any and all volunteers to become evaluators. Miklos has explicitly rejected this idea in the past, but I believe his reasoning was flawed.
- Get more evaluators to cut down on the time it takes to get a quiz question evaluated. I think this one might have actually been done already. When I posted the suggestion, it was taking 3-6 days to get my questions evaluated. Now, it's usually about 1 day.
- Add a field for the evaluator to explain his reasoning / suggest fixes when rejecting a question.
- Add a field when submitting questions to say something to the evaluator that is not part of the question.
- Add controls on the hunt screen to allow users to flag existing questions for review by moderators. The existing process of posting the question to the Quiz errors thread is clunky and slow, and I'm sure most users never do it.
- Allow evaluators to set questions as provisional. Originally, my idea was this would send the question to the members to approve it or not. But I do buy Miklos's argument that the general player population has little to no interest in doing a good job of evaluating questions. So now I think this suggestion should be modified and combined with the 2 out of 3 idea. An evaluator should be able to approve or reject questions on their own if they think it's clear cut. But if it's a gray area, they should force it to go to a larger pool of evaluators.
Now, on to new suggestions.
- A pool of evaluators should be selected to draft explicit guidelines for what questions should be accepted and what should be rejected. Some guidelines have been posted, but since there is all this confusion, I think that is strong evidence that we need better guidelines. Once the guidelines have been drawn up, they should be posted on the forums. Thereafter, the committee should meet periodically to review the guidelines and make changes as necessary.
- For the categories of too specific and uninteresting lexical knowledge, the default should be to allow the question, not reject it. Especially if the correct answer is easily found using Google.
What does everyone think?
Number one, and I can't stress this enough, Miklos or some other admin should read the forums and engage in the discussion so that the community feels that our opinions are being heard and not just ignored. Looking back, I can see that Miklos used to post a lot more often than he does now, but even then, his posts were not the type that I am looking for here. Experienced players can and do answer questions like why player X can attack me but I can't attack him. What I would like to see is responses to suggestions, such as "I like that suggestion, and I will relay it to the appropriate people" or "That would be nice, but it would cost way too much to implement" or "I think that would make the game worse, not better". I'd even be happy with "Stop bugging me, you pathetic loser", as that would still show that he's paying attention.
Next, I am going to collect all the suggestions I have made in the past on this topic and put them here.
- Require each quiz question to be accepted by 2 out of 3 evaluators. This would require a lot more evaluators, so in practice it would probably mean accepting any and all volunteers to become evaluators. Miklos has explicitly rejected this idea in the past, but I believe his reasoning was flawed.
- Get more evaluators to cut down on the time it takes to get a quiz question evaluated. I think this one might have actually been done already. When I posted the suggestion, it was taking 3-6 days to get my questions evaluated. Now, it's usually about 1 day.
- Add a field for the evaluator to explain his reasoning / suggest fixes when rejecting a question.
- Add a field when submitting questions to say something to the evaluator that is not part of the question.
- Add controls on the hunt screen to allow users to flag existing questions for review by moderators. The existing process of posting the question to the Quiz errors thread is clunky and slow, and I'm sure most users never do it.
- Allow evaluators to set questions as provisional. Originally, my idea was this would send the question to the members to approve it or not. But I do buy Miklos's argument that the general player population has little to no interest in doing a good job of evaluating questions. So now I think this suggestion should be modified and combined with the 2 out of 3 idea. An evaluator should be able to approve or reject questions on their own if they think it's clear cut. But if it's a gray area, they should force it to go to a larger pool of evaluators.
Now, on to new suggestions.
- A pool of evaluators should be selected to draft explicit guidelines for what questions should be accepted and what should be rejected. Some guidelines have been posted, but since there is all this confusion, I think that is strong evidence that we need better guidelines. Once the guidelines have been drawn up, they should be posted on the forums. Thereafter, the committee should meet periodically to review the guidelines and make changes as necessary.
- For the categories of too specific and uninteresting lexical knowledge, the default should be to allow the question, not reject it. Especially if the correct answer is easily found using Google.
What does everyone think?
Pontszám: 5