Username: Password:
Page: 123456789
Author
Name of topic: Quiz: the aspects of evaluation
much likedMiklosMale
Administrator
1138 post
View profile
Quiz: the aspects of evaluation
1. post - 2009.07.15. 23:43:18
Quiz: the aspects of evaluation

To our greatest pleasure the quiz has grown into a game-in-a-game and has at least as many questions as the framework game. Here you can find information about how the questions are evaluated.

The primary goal of the quiz is to ask questions that can be answered by an educated, intelligent person. Almost everyone. You don’t need specialized knowledge. Of course, one can say you can google most answers in a minute but we don’t really want questions that can only be answered by using a search engine. Some of the questions should make you think, make you rule out answers you that you will have an idea which answer to choose.

The most “convenient” general knowledge topics:
- Elementary school general knowledge questions. It might sound silly, but everybody learned about these, so no one can say they are specialized
- General knowledge, outside school questions, such as, films, famous people, rock bands, buildings etc.
- Current issues. These are a bit more complicated as questions become out-of-date in a few years, but there are still a lot of current topics worth asking a question about.
- Mind twisters. Even though not general knowledge questions, these still qualify as they do not require knowledge but simple brainwork. Please don’t write questions that require a calculator or equations.
- Linguistic questions. Spelling.

Bad questions:

First of all, it is important to highlight that even though a question might be accepted by the players it can still be disqualified by the admins later. There may be several reasons for this:

- Questions with a lot of spelling mistakes are rejected at once. It is simply impossible to correct each and every answer and we cannot run a spellchecker on the server for the time being. If you are unsure, please use a word processor to check the spelling. And please use the question mark at the end of the question.
-Questions that require a very special expertise for which an average person cannot know the answer. Many make the mistake of asking such questions. Don’t ask questions about games, hobbies, comic books, professions that can be only answered by someone who has years of experience in them.
-Lexical knowledge. Some questions about dates are acceptable, but only those that belong to the first category. Everyone should know the dates of the World Wars but not when a celebrity bought her first bicycle. What’s more, if you ask for a date, it is much more civilized to leave a bigger gap between the dates, than having 1 day between the four dates. In my personal opinion, the most acceptable method is to list four events and ask which one happened first or last.
- Extremely difficult questions are not good either. If you ask a question for which an average person might know the answer and you give four almost identical choices he will most certainly choose the wrong one. Please note that if 75% or more of the answers are incorrect the system will automatically disqualify the question. The purpose of this method is also to filter the correct questions where four good alternatives are listed but the person who asked the question selected the incorrect answer. If you ask a question that is already in our database, the question will be evaluated as wrong, but you will have the option to replace it with another without any penalty. But anyway, try to avoid asking trivial questions, e.g. When did WWI start?

- Odd-one-out questions and questions about capitals are filtered. We have enough questions about capitals, thank you very much. Odd-one-out questions are always ambiguous, because any of the answers can be accepted from a certain perspective. Getting back to filtering, if your question is rejected, please don’t submit it again using spelling mistakes, because it will be rejected as well. Please accept that the system will not accept it. And please also note that any questions containing vulgar/offensive words are also rejected.

- Maths. How much is (1+5)*3? Which is more, (4+3)*2 or (17/2-1)? Questions that require logic are OK, but trivial questions and equations that require a calculator are not.

- Foreign languages. You can ask what a certain famous thing was named after but you should not ask for plain translation of different words.

Questions to be deleted:
Simply put, when the player asks a question with the intention of submitting a correct one but he does not manage to, the question is evaluated as “wrong”. The “to be deleted” tag is given to those question which are incorrect on purpose. Examples are trivial questions, violating the netiquette or those that cannot be answered: What colour is the sky? Which Albanian folk band has not released an album? What colour was Amy Whitehouse’s hair in 2006? and so on.

Of course you can ask questions about Doomlord as well, but only ask questions that do not require the character to be high level and any player can answer, as a level 2 might also get the question: What skill does learning level 14 give? or What is the protection of the Nerubian robe? and won’t be able to answer.

Some examples for good and bad questions (please don’t submit them!)

‘How many members did the Beatles have?’
Although it is a bit trivial, still a good question, because everyone should know the band.

‘Which was the third album of the Australian Pigmies?
No one has ever heard of them, much less of their albums. Bad question.

Too special questions:
‘What is the width of the standard rails?‘
‘Who is the brother of the cartoon hero Glacius?’
‘What is the average wingspan of the Eastern mating mantis?’
‘Who was the coach of Manchester United in 2004?’

‘What is the name of the female protagonist in the Harry Potter books?’
Almost everyone knows the books, so you can ask such a question.

‘What is the title of the fourth book of the Harry Potter series?’
‘The goblet of fire, the qublet of fire, The calice of fire’
The difficulty is OK, but it is based on visual deception (q an g being similar) and contains a spelling mistake, calice instead of chalice. Bad question.

‘How old is the oldest woman on earth?’
There might be up-to-date information that becomes invalid in a month, should she die or become a year older. Try to avoid questions that might become out-of-date.


Of course we don’t expect anyone to submit correct questions all the time - I have also written questions that I would reject today. If 50% of your questions are rejected you get a warning and if this ration persists, you will lose the chance to ask questions. Also you can get the evaluation ‘to be deleted’ maximum three times. After the third occasion you will lose the opportunity to submit further questions permanently.

This is not a penalty, the SE reward is not significant. There are players who simply are unable to ask correct questions. That is all OK, they can still enjoy the other parts of the game.

Evaluators are also evaluated both automatically and manually. The aim is to let those players evaluate questions who give a realistic assessment. There is no penalty for “bad evaluation” you simply won’t be asked to evaluate further questions, however some system for a reward is on the planning table good assessors.

Note that now you have to option to choose the reason why the question was rejected. This way the player who asked the question will have some idea about what he had done wrong.
This post was modified by Miklos at 2009.07.16. 11:47:03
Score: 5
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
2. post - 2009.08.31. 22:53:13 (Reply to post #1 of Miklos.)
Please create an international team of players to review these questions. 9 people to review every question... 2 North American, 2 European, 2 Asian, 1 each from S. America, Africa and Australia. If 7 to 9 people accept the question, it's added, if 0 thru 4 approve, then gone. If it's 5 or 6, then there can be QUICK discussion from both sides.

Right now, you have absurdly obscure questions that were accepted (and questions with wrong answers), yet you have also rejected questions based on criteria that you accepted for previous questions.

The Quiz portion of the game is one of the best parts of any online game... However the method under which the questions are evaluated and passed/failed is horribly poor and inaccurate. Therefore, I suggest moving to the group-system outlined above... Let an international pool of PLAYERS tell you which are good or bad questions.
Score: 6.67
likedValkyrieMale

474 post
View profile
Re: Quiz: the aspects of evaluation
3. post - 2009.09.01. 08:29:27 (Reply to post #2 of Viridel.)
I support the initiative and volunteer
Inter urinas et faeces homo nascitur
Score: 5
much likedMiklosMale
Administrator
1138 post
View profile
Re: Quiz: the aspects of evaluation
4. post - 2009.09.01. 13:34:17
The original concept of the quiz evaluation was: let the players do it. Whenever a new question was posted, it was asked from 30-40 players, and all of them had the opportunity to evaluate it. I don't remember the exact numbers, but it was something like: 8 "delete" -> it was deleted, 12 "wrong question" -> it became a wrong question, 15 "good question" -> it became a good question. The result was a disaster. People let pass many bad questions, while some very good - but more difficult - questions were rejected. So we replaced the system with the current one: quiz admins evaluate the questions. Quiz admins are people who asked many good questions, understand, accept and follow the evaluation rules, and earned our trust.
Of course, it is always a bit subjective if a question is good or bad. If I say, the "people evaluate" system was 50% faulty, the quiz admin system was much better, like only 1-3% faulty. Sometimes the quiz admins make mistakes, but very rarely. And if a mistake is made, you can still complain here in the forum, or in email. If a bad question was accepted, we can correct it afterwards. If you think your question was rejected, you can still ask the reason.
Maybe the system could be further optimalized - as you suggest, a question is not evaluated by 1 admin, but a team of admins. However, to do this, we need a lot more admins, especially when the number of players grow. Also, to become a quiz admin, volunteering is not enough. If you are active on the forum, you have intelligent posts, and you also post good questions, then we are happy to give you the appointment. But until then, we can't just appoint anyone who volunteers, or case 1.) happens.
Score: 5
likedValkyrieMale

474 post
View profile
Re: Quiz: the aspects of evaluation
5. post - 2009.09.01. 14:42:00 (Reply to post #4 of Miklos.)
I am active on the forum, as you can see
Also, I find my self intelligent enough, and with good knowledge of English language
Only my first question got deleted, so I think they are ok (despite some minor mistakes, of which I find some justified, some not)
That leaves the matter of trust

Btw, you say that the questions get corrected, but I still come across reported mistakes
Inter urinas et faeces homo nascitur
Score: 5
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
6. post - 2009.09.01. 19:30:06
Well, first of all, you don't need "volunteers", you need active, intelligent members. I would suggest those who do well in the Quiz Olympics be considered - as they are obviously not afraid to go out and find (accurate) answers to questions. Also those who post on these boards with insightful and accurate comments should be considered.

As such, I would like to be selected for one of the two North America positions.
Score: 5
averageMosquito665Male

116 post
View profile
Re: Quiz: the aspects of evaluation
7. post - 2009.09.06. 12:19:40
all bad questions should be considered like the ones with spelling mistakes. just because someone feels that a question is too hard a player shoulnt lose the chance to get SE. hard question? write a new one. but waiting another 100 correct answer just because 1 or 2 people think a question is too tough for them is not right.

let people re-write the questions!!!

think about it pls.
This post was modified by Mosquito665 at 2009.09.06. 12:38:15
Score: 7.33
much likedstekkos

1356 post
View profile
Re: Quiz: the aspects of evaluation
8. post - 2009.09.06. 13:22:23
I dont know if the rules you stated are followed but the way most peoples questions are rejected without any explanation (for example it would be nice if you STATE THE SPELLING MISTAKE!!!!!!!!!!!!) Just by saying rejected for spelling mistakes with no further explanation, while questions with HUGE spelling mistakes are in the database, dissapoints people. Also this "Specialised group" rejections should stop. When you have questions about Drizzt, senator Kenety, Dragonlance, etc., how can you reject questions like "What is the legal drinking age in USA"???? I dare you to leave it in and see how many people will answer that one correctly (I am not a US citizen by the way and i do know the answer). Personally i will stop posting questions, it is really frustrating to get rejected for silly reasons and is messing up a really good idea you had
This post was modified by stekkos at 2009.09.06. 13:23:28
Score: 7.50
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
9. post - 2009.09.06. 18:32:39
Cute... I post something, and the Mods delete it... Well, since the Mods only check every few days, here we go again:


National Hockey League (NHL) Team - Anaheim ___?

Eagles
Ducks
Mallards
Penguins

The question is valued as "Bad, incorrect question"!
According to evaluators only a special group of people might know the answer.


Ok, tell me why I should stay in this game. You have ABSURD crap from Europe littering the quizzes. You have wrong answers allowed into the database in the first place...

You're saying this question is too "difficult", when you just go to Wiki and go "NHL Anaheim"... Yet there are questions about Dragonlance heroes, F1 winners (F1 is less popular in North America than NHL is in Europe - by far), and other stuff that can't be looked up easily or at all (that French "Unbribable" guy, for instance).

I don't care about Literature, Mythology (at one time, not not any more) or insignificant European historical figures... But you know what happens when a question about them comes up??? I LOOK IT UP!!! There is no such thing as "too specific" when the resources are readily available for anyone to answer any properly phrased question.

I'm one foot out the door... Give me a damn good reason why I should stay - because if I have to answer everyone else's questions about topics that do not interest me, but am not allowed to post what is important to me - I have no reason to stay.
Score: 6.50
much likedstekkos

1356 post
View profile
Re: Quiz: the aspects of evaluation
10. post - 2009.09.06. 20:31:31
He makes exactly the point i am trying to make. Stop catecorizing the questions. Let people post questions about any subject. Most will put questions that can be answered if you just google it. And even if you get it wrong, you get the answer anyway for the next time it pops up
Score: 7.50
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
11. post - 2009.09.10. 22:27:31
Please look at the below pair of questions. They ask the same fundamental question - NHL Team Name. One has been approved, AND has had 5 correct answers out of 6. The second "clone" was rejected outright for being "too obscure"... HOW CAN IT BE TOO OBSCURE WHEN THE FIRST QUESTION IS AT 83.3% ACCURACY???

And who are you to say what is too obscure and what isn't?!? Isn't this a game for the players? Shouldn't the players be telling you what good and bad questions are? If you want to play Hungarian Trivial Pursuit - then go play Hungarian Trivial Pursuit!!! Just don't make an INTERNATIONAL game about it.


National Hockey League (NHL) Team - Atlanta ___?
Hawks
Falcons
Thrashers
Jays

The question is valued as "Bad, incorrect question"!
According to evaluators only a special group of people might know the answer.

National Hockey League (NHL) Team - Anaheim ___?
Eagles
Ducks
Mallards
Penguins

5 good and 1 bad answers have arrived for this question.


STOP BLOCKING GOOD QUESTIONS WITH EASY-TO-FIND ANSWERS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Score: 5.33
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
12. post - 2009.09.11. 19:20:14
And now I don't even get a reason for questions being rejected. Because they KNOW they can't say "only a special group"... They just don't want to accept my questions. Is this really a game people want to support?


National Hockey League (NHL) Team - Boston ___?

Bears
Bruins
Grizzlies
Polar Bears

The question is valued as "Bad, incorrect question"!
Score: 3
averageViridel

796 post
View profile
Re: Quiz: the aspects of evaluation
13. post - 2009.09.12. 09:18:48
Well all - I quit.

The Quiz system has alienated me to the point where I refuse to be part of this game any more.

For example, questions that have been accepted:
What NBA team does Ray Allen play for
Who is the leading scorer in NBA history [set ~20 years ago]
What NHL team is celebrating their 101st anniversary
What NHL team won the 2008 Stanley Cup
Who won the 2008 F1 Championship
What is the width of a soccer net
What was the nickname of the Hungarian soccer team that went undefeated in 35 matches in the 1950's

Examples of questions that I have submitted that have been rejected:
National Hockey League - Anaheim ____ (repeat for Atlanta, Boston and Buffalo, as well as soccer's Tottenham, just to test)
Who is the Career Goaltending Wins leader in the NHL [set last year]
What is the size of a water polo net


Now, do you see the similarities between the two sets? Would you consider the first set or the second set to be more "only a specialized group would know the answer". At first glance (and 2nd, 3rd and 4th) the second set is clearly the easier set of questions, with answers as easy or easier than the first set to research online. However, it has become clear that my questions are not even being fairly reviewed - merely rubber-stamped "Reject" - largely because I am focusing on a largely North American sport.

Note that I asked 6 Water Polo questions before one was rejected - all of which FAR more difficult to research an answer to than ANY of the NHL questions I have posted. This is, for lack of a better term, racism against their North American audience. Hungary is a Water Polo powerhouse - therefore, those questions are passed thru the system with ease. Yet the NHL questions are instantly blocked. This is a form of (at worst) Racism, and at minimum, Prejudice.

And this doesn't come close to addressing the massive disparity between ridiculously special-knowledge questions involving Dragonlance, European historical figures, Nobel winners, obscure literary works, Television shows seen in only one country and the myriad of other absurd questions (not to mention all the questions with the wrong answers).

I would like to call for all those who are frustrated with the quiz system that this game so blatantly abuses, to simply walk away. Hopefully all you have invested is your time, and you will realize as I have that time is precious, and frustrating time is purely wasted time. There are many hundreds of other free online games, and one such as this that clearly makes no attempt to be fair and non-prejudicial does not deserve one penny of your cash, nor one minute of your time. This is YOUR game - because without you, there is no game. Therefore, please take ownership of your right to fight Racism and Prejudice, and not support this game a moment longer.
Score: 2.33
averageMosquito665Male

116 post
View profile
Re: Quiz: the aspects of evaluation
14. post - 2009.09.12. 10:13:41
bye!
Score: 2.33
averageSauron

2 post
View profile
Re: Quiz: the aspects of evaluation
15. post - 2009.09.13. 23:25:30 (Reply to post #2 of Viridel.)
Could you please give reason why my QUIZ was considered "BAD" incorrect question? Would appreciate a good reason why. I believe this to be a very good question about a very good series of books and the movies.

Thanks Kindly
Sauron
Score: 5
likedPogaxaFemale

227 post
View profile
Re: Quiz: the aspects of evaluation
16. post - 2009.09.14. 18:03:54
please post your rejected question with answers into the "My quiz question was rejected" topic.
Score: 5
everyone's favoritemaleka

19 post
View profile
Re: Quiz: the aspects of evaluation
17. post - 2009.10.15. 20:45:57
The quiz should be there to be educationnal. So why is there a limited type of questions that we can ask? Who cares if everybody doesn't know all the answers.
When my question is wrong, ok, don't pu it, but when it is because a ''limited group of person wont know the answer'' I find it ridicoulous! I would rather have a wrong answer and learn something than know all the answers and don't learn anything!

Plus...

Why do we have the same questions twice in a row or in the same hunt? Isn't there enough question in the database?

thanks
Score: 7.50
much likedHatsepsutFemale
Administrator
131 post
View profile
Re: Quiz: the aspects of evaluation
18. post - 2009.10.28. 11:55:49 (Reply to post #17 of maleka.)
Quote: maleka - 2009.10.15. 20:45:57
Why do we have the same questions twice in a row or in the same hunt? Isn't there enough question in the database?


Unfortunately, the random generator sometimes has strange choices. Now we have approximately 3000 questions, the same was chosen twice. Bad (or good) luck?
Score: 5
much likedstekkos

1356 post
View profile
Re: Quiz: the aspects of evaluation
19. post - 2009.11.01. 06:40:35
Evaluation of your questions so far: 39 good, 7 bad, 0 deleted. This is the current status of my quiz contribution. and now i get this message:

Attention: Please try to ask respondable questions, and take care to mark the good answer. If you do not take care, you may lose the opportunity to write questions in the future, and you may also lose the opportunity to acquire soul energy this way.

Do i seem like i do not try to get the questions right? Most of my bad ones are "only a special group....". Since i cannot predict every time what a "special group" means to evaluators, i fail sometimes and loose the question. But warning me like that, even if all 7 of my bad questions were wrong answers, is unfair. You ignore completely that i posted 39 good! Doesnt that count for something, at least to show that i am not randomly throwing questions around not caring for the answer????????
Score: 7.50
everyone's favoritemaleka

19 post
View profile
Re: Quiz: the aspects of evaluation
20. post - 2009.11.25. 21:38:34
I had the same response and I.m at 24 good 12 bad and 0 deleted!!!
Score: 5
Page: 123456789