[Discussion] Episode Rating Polls.

[felix]

Originally Posted by Sazelyt

Polls can be useful in many ways. For instance, if a series is going through a painful period (e.g., Naruto), and you want to get the general idea of the way the anime is going, you can check the poll results and see whether it is better than expected, getting better or getting worse, etc.

Uhh...... no.
For one you can just read 3-4 posts and you get pretty much a good idea on how good or bad it is.

BTW, you can tell just by the number of people who voted in it if its really good or bad. For the most part.

Originally Posted by Sazelyt

With so many options, you can see the diversity, and separate the quality viewers than the pre-teen-type (not actually pre-teen, but the type that likes anything given) viewers. Additionally, if you know the voters (the way they rate the story - you can get that idea if you have read their posts for some time), and have a general idea of how they approach the anime (quality- and story-wise), you can see their votes and get a really accurate idea of the anime - from many perspective.

Note
Bellow I made some lists, don't take them too seriously as I made them more as an example. Also when I say ability, mostly I'm refering to will power.

And if it's so important why don't we just have a thread for it.
Also, I'm personally seeing this pattern at the moment:

• You got the trolls 1% to 3% generally -- They just don't like it, but seem to keep watching it.
• You got the reserved 30% to 50% -- they'll rate and sometimes bable about past ratings and future expected ratings, this would sound logical but I'm only hearing figures and comments such as: "I gave this a 9 cuz I want to give next episode a 10" or "I'm saving my 10 for X event" or "I didn't feel like giving it more then X cuz I gave Y to...."
• You got the Rate Perfect/Excellent or Don't Rate At All; around 30% -- they'll rate the episode 9 or 10, but if they don't like it, by just a little, they won't rate it at all.

Also the way you imply it, to rate an episode one might require:

• Knowledge of the production team
• Knowledge of the production budget
• Knowledge of the director. (past project and reputation)
• Some insight on how it will (as in should) play out
• For Visuals:
  • The ability to decipher what is an optical illusions and what is just messed up animation.
  • The ability to spot variation in luminosity, contrast as well as color balance for individual scenes, and the ability to interpret it.
  • The ability to spot consistency and inconsistency in art style, in other words: good memory to remember how much of cheap animation was used, when and how; how much expensive animation was used and again when and how, everything reported to a time scale.
  • The ability to judge good/bad timing of individual scenes. (seconds)
  • The ability to spot how many static scenery scenes were used. (scenes that are just one big pic that either scrolls zooms or just does both)
  • The ability to judge the best scenario, or at least to have a realistic opinion on what the best scenario might be. (if the production team used their time and budget correctly)
  • The ability to spot and differentiate between animation style, and the knowledge of the goods and bads related to it.
  • The ability to spot unrealistic animation and realistic animation. (Ex of unrealistic: a character holding a giant 10ton sword without displacing any ground; a character double-jumping, a character tacking/blocking a blow but not being being pushed back, character's cloths not suffering the effects of friction or any other effects related with the environment around them etc etc etc - basically the ability to spot fraud presented as non-fantasy material)
  • The ability to judge cartoony animation and weather it helped or not.
  • The ability to spot & judge the quality of scene recycling. Or better put, animation recycling since most of the time, they won't recycle the hole scene but pieces of the characters/objects animation. (using tricks like zooming in or out to hide it)
• Story Wise:
  • The ability to interpret current events in relation to a all past events.
  • The ability to spot & judge story-candy. Meaning your not just noticing character X does this and that or character Y does this and that, but also remember what character X and Y did before, what meaning this has and how realistic it is. For instance in some shows some characters may at some point just simply act totally not like themselves for the only purpose of providing the audience something to munch on or just because they were a plot lock, sorta speak.
  • The ability to judge story distribution over episodes.
  • The ability to judge cliff-hangers. (their quality, frequency and necessity)
  • The ability to judge scene timing. (as in relative to other scenes, from things like how they jump from scene to scene, to the delays used in-between scenes)
  • The ability to judge and spot story pace. This means being aware of the length of scenes, and the patterns in timing related to it. How much filler scenes are used.

etc etc etc etc etc etc

A poll simply gives you a number representing the combined total of whatever the person who rated took into consideration. Which I would think should be at least 4-5 things at least regardless what they are. (and I'm not limiting the options to the lists above) That's if it were to be even remotely useful. But polls on animesuki considering what criteria is used might as well be used to judge if a episode is "circus attraction" or not.

Originally Posted by Sazelyt

Just ask yourself that question? If you want to buy a record, which kind of review do you find as more useful, if you have to check a lot of sources - just like/dislike rating - or an average of 1-10 scale rating? If I were to given a choice between those two, I prefer the 1-10 scaling system, which gives a better information on the quality.

Better information on the quality ehh~

Lets say you have your average series. And you have a poll.
Of course it's AnimeSuki style Episode rating poll.

Let's say 1000 people watched the series and participated in the poll. Out of those you have 300 people that's 30% who voted as accurately as possible, tacking into consideration at least all the listed above (yes, that's right I'm giving you impossible statistics in your favor) and presuming optimal formula to calculate total rating.. and of course minimum displacement in-between their choice of vote. (so they didn't all vote the same thing but really close)

And you have the other 70%. You got 35% Only-Vote-Best (note: I said above participated not that they voted, consider not voting an option) 40% Rate-The-Rating 5% are trolls or vote before they see it and 20% have no idea how exactly they should vote it (mixed feelings) but do it anyway.

Even with these chances (lets not kid ourselves 300/1000 that knowledgeable and perfect is a ridiculous impossible number) even so, you have far far greater chances of making the right choice by just throwing a coin, then judging the poll. Statistically speaking of course, since you want to judge by statistics.

---

Replies to some other issues..

Originally Posted by Sazelyt

If you know the site RottenTomatoes, they actually combine both rating systems, as the number of positives (more like - positive and negative or like and dislike), and the average rating (1-10 or F-A or *-****). This way, they can give a better idea on the product.

Sorry I don't understand, which would be the both rating systems and what do you mean by the positive negative thing.

Originally Posted by Sazelyt

In short, "I like" cannot differentiate between, great/good/not-bad/, and "I dislike" cannot differentiate between awful/bad/not-good. With the grading system, you can balance your options on how you can evaluate the anime.

But that's not the point.
The point is if a scale that differentiates between them is actually practical.
What good is a system that has 99% error probability.

My argument with the simpler version was that it has a far higher chance of getting it right.

Originally Posted by Sorrow-K

I'm not sure if I'd advocate getting rid of them, since I can't be any more than speculative regarding what sort of impact it'd make in the discussion surrounding episodes.

But perhaps keeping them when they offer such a easy way to spam and no practical information isn't that great of an option either. I agree, they can be removed, but it might be harmful, best if they are moved/changed or simply replaced by something better.

Originally Posted by Sazelyt

There is another side to that rating. Those people may give 1 to a really good episode, if they expect action and see a highly-emotional or serious episode. If the system could have automatically eliminated those kind of ratings (from a human-based perspective, you can ask the person who has voted to list their reasons, and if the reasons are not logical, you can just remove their vote), you could get a pretty accurate rating for the series, but that will not happen. But, at least, if there is lots of diversity in the opinions, you can get an idea of what kind of episode it is, based on people's expectations.

That's why I said it's like 2 polls in one.
Your suggestion on handling votes is not only a truck load of work but would also mean Moderators would have to be flawless and perfect. I also think we don't share the same opinion on what a discussion implies. I mean even presuming perfect mods, if nobody had an opinion that was different then their standard then what would be the point of the Episode Discussion Threads in the first place.

[npal]

You forgot to mention second wave trolls, the trolls who flame the trolls who consistenly vote low and/or exhibit other troll behavior in the aforementioned thread.

Hope that made sense

There's a ridiculous phenomenon where people draw vote charts regarding a particular member and flame that member because of his voting pattern. Of course, many of those second wave trolls are trolls in their own right elsewhere, but I'd like to point that out.

[SeijiSensei]

Back in the 1980's I worked on a project to develop a method of rating television programs using viewer opinions gathered through surveys. For the curious, I posted some observations in the "ratings" thread about the problems of aggregating opinion scores like those discussed here. There are also some excellent observations by rooboy on the subject there as well.

[Sazelyt]

Sorry, for keeping the post as simple and short as possible.

Originally Posted by Cats

Uhh...... no.
For one you can just read 3-4 posts and you get pretty much a good idea on how good or bad it is.

BTW, you can tell just by the number of people who voted in it if its really good or bad. For the most part.

That wouldn't be any different than checking only 4 of the votes out of maybe 40 votes. Not enough information for sufficient evaluation.

Note
Bellow I made some lists, don't take them too seriously as I made them more as an example. Also when I say ability, mostly I'm refering to will power.

And if it's so important why don't we just have a thread for it.
Also, I'm personally seeing this pattern at the moment:

• You got the trolls 1% to 3% generally -- They just don't like it, but seem to keep watching it.
• You got the reserved 30% to 50% -- they'll rate and sometimes bable about past ratings and future expected ratings, this would sound logical but I'm only hearing figures and comments such as: "I gave this a 9 cuz I want to give next episode a 10" or "I'm saving my 10 for X event" or "I didn't feel like giving it more then X cuz I gave Y to...."
• You got the Rate Perfect/Excellent or Don't Rate At All; around 30% -- they'll rate the episode 9 or 10, but if they don't like it, by just a little, they won't rate it at all.

Also the way you imply it, to rate an episode one might require:

• Knowledge of the production team
• Knowledge of the production budget
• Knowledge of the director. (past project and reputation)
• Some insight on how it will (as in should) play out
• For Visuals:
  • The ability to decipher what is an optical illusions and what is just messed up animation.
  • The ability to spot variation in luminosity, contrast as well as color balance for individual scenes, and the ability to interpret it.
  • The ability to spot consistency and inconsistency in art style, in other words: good memory to remember how much of cheap animation was used, when and how; how much expensive animation was used and again when and how, everything reported to a time scale.
  • The ability to judge good/bad timing of individual scenes. (seconds)
  • The ability to spot how many static scenery scenes were used. (scenes that are just one big pic that either scrolls zooms or just does both)
  • The ability to judge the best scenario, or at least to have a realistic opinion on what the best scenario might be. (if the production team used their time and budget correctly)
  • The ability to spot and differentiate between animation style, and the knowledge of the goods and bads related to it.
  • The ability to spot unrealistic animation and realistic animation. (Ex of unrealistic: a character holding a giant 10ton sword without displacing any ground; a character double-jumping, a character tacking/blocking a blow but not being being pushed back, character's cloths not suffering the effects of friction or any other effects related with the environment around them etc etc etc - basically the ability to spot fraud presented as non-fantasy material)
  • The ability to judge cartoony animation and weather it helped or not.
  • The ability to spot & judge the quality of scene recycling. Or better put, animation recycling since most of the time, they won't recycle the hole scene but pieces of the characters/objects animation. (using tricks like zooming in or out to hide it)
• Story Wise:
  • The ability to interpret current events in relation to a all past events.
  • The ability to spot & judge story-candy. Meaning your not just noticing character X does this and that or character Y does this and that, but also remember what character X and Y did before, what meaning this has and how realistic it is. For instance in some shows some characters may at some point just simply act totally not like themselves for the only purpose of providing the audience something to munch on or just because they were a plot lock, sorta speak.
  • The ability to judge story distribution over episodes.
  • The ability to judge cliff-hangers. (their quality, frequency and necessity)
  • The ability to judge scene timing. (as in relative to other scenes, from things like how they jump from scene to scene, to the delays used in-between scenes)
  • The ability to judge and spot story pace. This means being aware of the length of scenes, and the patterns in timing related to it. How much filler scenes are used.

etc etc etc etc etc etc

A poll simply gives you a number representing the combined total of whatever the person who rated took into consideration. Which I would think should be at least 4-5 things at least regardless what they are. (and I'm not limiting the options to the lists above) That's if it were to be even remotely useful. But polls on animesuki considering what criteria is used might as well be used to judge if a episode is "circus attraction" or not.

Actually not the combined total. I don't think no one evaluates an episode that deeply. If it is some kind of combination, it is more like an automatic processing of the information acquired by that person's mind.

In simple words, it is more like how a person perceives the episode. Some people automatically take into account many points, so while watching the episode, they can balance many of the important items required to rate an episode accurately. Some people do not have (have not developed) such processing power, so they just keep it as simple as possible, that's why I said pre-teen to represent that kind.

Better information on the quality ehh~

Lets say you have your average series. And you have a poll.
Of course it's AnimeSuki style Episode rating poll.

Let's say 1000 people watched the series and participated in the poll. Out of those you have 300 people that's 30% who voted as accurately as possible, tacking into consideration at least all the listed above (yes, that's right I'm giving you impossible statistics in your favor) and presuming optimal formula to calculate total rating.. and of course minimum displacement in-between their choice of vote. (so they didn't all vote the same thing but really close)

And you have the other 70%. You got 35% Only-Vote-Best (note: I said above participated not that they voted, consider not voting an option) 40% Rate-The-Rating 5% are trolls or vote before they see it and 20% have no idea how exactly they should vote it (mixed feelings) but do it anyway.

Even with these chances (lets not kid ourselves 300/1000 that knowledgeable and perfect is a ridiculous impossible number) even so, you have far far greater chances of making the right choice by just throwing a coin, then judging the poll. Statistically speaking of course, since you want to judge by statistics.

Again, throwing the coin would give you only two choices - kind of like death or alive option, you cannot allow anything in between. That kind of rating is not enough for many people.

Also, do not forget that, the people who vote without taking into account many things might also provide an insight to the episode. It may not be useful for me, but it might be useful for some other person. If you have checked the ratings at imdb you will notice that there are zillions of movies that have ratings around 6-7 out of 10. If you check how the voters vote, based on their age groups, their gender, etc., you can get a better idea on how the movie is evaluated. This is the same for the ratings. If you have an idea on the voter, you can get a better idea on the episode.

Also, many of those voters, may not post a detailed comment on that episode explaining every single detail they took into account while evaluating the episode. That is always the case. You do not usually see a page long comment on how the episode is, and even if you remove the ratings, that will never change. so, why not make it easier for these people to evaluate the episode.

Sorry I don't understand, which would be the both rating systems and what do you mean by the positive negative thing.

Just check the site, to see how their ratings work. And, positive/negative is just a representation of good or bad. If it is above 0, then it has in general good reviews. And, anything above 50 out of 100 counts as good.

But that's not the point.
The point is if a scale that differentiates between them is actually practical.
What good is a system that has 99% error probability.

My argument with the simpler version was that it has a far higher chance of getting it right.

First, the poll results are not given only in terms of like 5.6 out of 10. It gives the result as it is with poster's name. Second, the error probability is not that high. The ratings you see everywhere use a similar system. If the error rate would be that high, then they wouldn't have used that.

That's why I said it's like 2 polls in one.
Your suggestion on handling votes is not only a truck load of work but would also mean Moderators would have to be flawless and perfect. I also think we don't share the same opinion on what a discussion implies. I mean even presuming perfect mods, if nobody had an opinion that was different then their standard then what would be the point of the Episode Discussion Threads in the first place.

Actually, if there would be such perfect moderating, they would only remove the votes that only serve as a spam. Any vote that can logically be supported would still be there. That is the same case with discussion. If there is a spammer, you wouldn't wanna see his/her posts, that would only degrade the quality.

[felix]

Originally Posted by Sazelyt

Sorry, for keeping the post as simple and short as possible.

Likewise,
Also, sorry I've cut off some pieces in quotes to avoid making a excesively long post. >_>

---

Originally Posted by Sazelyt

Actually not the combined total. I don't think no one evaluates an episode that deeply. If it is some kind of combination, it is more like an automatic processing of the information acquired by that person's mind.

Sounds more like a wheels of bias spinning. T_T
And like I said in the note, don't take it too seriously.

Originally Posted by Sazelyt

In simple words, it is more like how a person perceives the episode. Some people automatically take into account many points, so while watching the episode, they can balance many of the important items required to rate an episode accurately. Some people do not have (have not developed) such processing power, so they just keep it as simple as possible, that's why I said pre-teen to represent that kind.

I wouldn't go into age groups..
There is no such well defined difference and I think I've seen some real oldies that are totally immature. =P

Originally Posted by Sazelyt

Again, throwing the coin would give you only two choices - kind of like death or alive option, you cannot allow anything in between. That kind of rating is not enough for many people.

That's because I was thinking you were thinking of using polls to chose weather or not to watch a show (hence the 2 choices: yes/no ). But I guess I must have been mistaken.

Originally Posted by Sazelyt

Also, do not forget that, the people who vote without taking into account many things might also provide an insight to the episode.

Sometimes maybe, for some series maybe but some people just go with the flow, if anything the poll might totally influence their entire opinion.
Sometimes reading one of these posts it feels like I'm reading the current results of the poll. -_O

Originally Posted by Sazelyt

Also, many of those voters, may not post a detailed comment on that episode explaining every single detail they took into account while evaluating the episode.

Exactly, they go into a thread and after voting either loose any urge to post or just post something to do with the rating. That's why it might be a good idea to move the episode rating somewhere else. In a separate review thread perhaps.

Besides, it's not the purpose of the thread to decide anything.

Originally Posted by Sazelyt

First, the poll results are not given only in terms of like 5.6 out of 10. It gives the result as it is with poster's name.

Actually, if there would be such perfect moderating, they would only remove the votes that only serve as a spam.

So T_T it's not a 5.6 out of 10, it's a 5.6 out of 10 given by X or Y. I doubt you know people that deep. I also doubt anyone knows themselves that well to tell you, after like a week or two, why 5.6 out of 10.

Also concerning the perfect moderator for poll thing.
If for example the results were filtered based on everyone's buddy list. In the sense that in Episode Discussion Threads, or discussion threads in general, nobody can see the poll votes of everybody in the forum, unless they just happen to have in their buddy list the hole forum. I think that would either stop or slow down the trolling. But its going to be a very very long time before something like that ever gets implemented by Vb.

Just so there's no confusion for the above; if for example nobody in your buddy list voted then you would see 0s everywhere, like you're the first to vote. Its not evil =P okey maybe a little. But it would be for their own good.