Studio Deen: Studio discussion, speculation, and whatnot

[RRW]

i disagree of using rating as indication of studio quality. is bit to bias compare to professional review. one can say "OMG it have good soundtrack 10/10" other can say "Yuri!!! BAD 1/10"

for comparison
Simoun have good rating (around B-A) for its DVD review by ann

[sople]

Originally Posted by donquigleone

Isn't the quality in proportion to how many, and by how much, people will enjoy the work?

No.

Originally Posted by donquigleone

What other slightly objective approach can you use to even try to gauge quality? Individual opinion is not that useful as people will always disagree, and you can always find someone that enjoyed even turds like Dragonaut. The best you can do is to average a large number of people's opinions and try to derive a number from it.

If you want to claim something is good or bad or great or shit, you have to watch it yourself. Basing your opinion off of anything else completely invalidates it.

[Reckoner]

Originally Posted by donquigleone

Isn't the quality in proportion to how many, and by how much, people will enjoy the work? So surely rating lists as a informal survey of a large number of people's opinions is a decent gauge of how much they enjoyed a series, and hence how good it is.

Hell no. Going by this I'll have to start considering American Idol one of the greatest American TV shows of all time, a sad time indeed.

[DonQuigleone]

Originally Posted by sople

No

At least say why?

If you want to claim something is good or bad or great or shit, you have to watch it yourself. Basing your opinion off of anything else completely invalidates it.

I watched Maria Sama ga miteru and I found it dull. Does that mean it's bad? No, it's not my kind of show, but I know that many other people like it. Now some things are inherently bad (stuff that's poorly animated), and we can easily say "this is bad", but when it comes to the line between mediocre and good it's a lot more difficult, and there are 100 more variables at play.

What MAL or ANN says is not my own personal opinion. I am saying that it's related to the average opinion, the closest thing we have to what the anime community thinks as a whole. If the list says people rated poorly, than most viewers (who took the trouble to rate) thought it was mediocre.

I'm not saying that a rating causes it to be good. I'm saying the rating is a reflection of properties of the anime itself. If an anime is a masterpiece, people will recognize that fact and rate it accordingly. Some people will disagree and rate it lower. An interesting thing to look at is on ANN, where they show the proportions people vote in. For the vast majority the opinion follows a bell curve.

However we're debating the quality and general perception of Studio Deen's works. You say Simoun is good, I saw it and didn't like it, but this is just a personal disagreement, the real point is that Deen produces Mediocre shows, and the fastest way to tell if Deen's shows are, in fact, mediocre is to survey people's opinion on every title they've made. We can use the MAL or ANN rating as a convenient shortcut

So I point to it's low rating, which show that most people think that's it mediocre while you have no proof but your own word

Originally Posted by Reckoner

Hell no. Going by this I'll have to start considering American Idol one of the greatest American TV shows of all time, a sad time indeed.

If you did a comprehensive survey of most americans, American Idol would probably be rated much lower then you think. Also these ratings work best if they are done by people like you. People on MAL are internet based anime fans, and we are also the same. Their opinions will generally be similiar to our own.

@RRW
When there's a large number of ratings (in the hundreds) those abberations tend to be filtered out. However it's probably also a good idea to compare like with like. Yuri is not a popular genre, so many may rate it lower because of that. In practice this doesn't tend to bear out, niche interests often tend to be over-represented at the higher ends of rating lists, like LoGH, for instance.

Also most professional reviewers do not have particularly more valid opinions then their audience (but they are much better at writing about it). Also professional reviewers and audience reactions tend to correlate quite well and are usually in agreement. Compare IMDB ratings (which are audience ratings) and Rotten Tomatoes (Aggregated critics ratings). Most of the time the two will generally agree.

[Reckoner]

Part of the problem with rating lists like MAL are that 1) Informal survery is informal, and 2) The metrics employed are not universal for every person.

If you look at people's lists on MAL, you'll see people who rate tons of shows 10's, 9's, and 8's, and you'll see lots of other people (like me), who tend to spread around the ratings so as not to inflate the ratings too much (Thereby taking away meaning from them).

A 9 for me is not a 9 for another person. I could very well rate a show 8/10 and possibly enjoy it more than a person who rated it a 9/10.

It's just extremely flawed, especially when you consider the types of fans certain genres attract. Things like Clannad gets tons more viewers than say relatively obscure shows like Mushishi. The sample size is already much larger for that.

It doesn't pan out really. Most of what you'll learn from these lists are simply what were the most popular animes of the time. It is not indicative of quality.

[DonQuigleone]

Originally Posted by Reckoner

Part of the problem with rating lists like MAL are that 1) Informal survery is informal, and 2) The metrics employed are not universal for every person.

If you look at people's lists on MAL, you'll see people who rate tons of shows 10's, 9's, and 8's, and you'll see lots of other people (like me), who tend to spread around the ratings so as not to inflate the ratings too much (Thereby taking away meaning from them).

A 9 for me is not a 9 for another person. I could very well rate a show 8/10 and possibly enjoy it more than a person who rated it a 9/10.

It's just extremely flawed, especially when you consider the types of fans certain genres attract. Things like Clannad gets tons more viewers than say relatively obscure shows like Mushishi. The sample size is already much larger for that.

It doesn't pan out really. Most of what you'll learn from these lists are simply what were the most popular animes of the time. It is not indicative of quality.

I wouldn't take the numbers as absolutes. Instead they need to be judged relatively. Generally everyone will rate high quality shows higher, some will rate more extremely then others, cuains their opinion to be over-represented then those who are more measured, but this should balance out if the sample is large enough.

And as I said you should take it relatively. My personal approach is to say that the top 10% (or so) of Anime on MAL are likely to be good. I did a bit of research previously and found this 10% mark to occur around 8 for both MAL and ANN, so you can informally judge anything scoring above 8 to be in the top 10% of Anime and "good".

Within that I don't think ratings count for anything. At the top a few percentage points count for too much, so are easily manipulated. Just because Clannad After Story is rated top doesn't actually make it the best anime ever. Likewise around the border it's a bit fuzzy. So a 8.01 or a 7.99 could be in the "good category", a 8.6 certainly is, and a 7.4 is likely not.

Also Mushishi actually does extremely well on rating lists, as the rating is not related not to the number of raters, but the rating. Rating lists actually may over-represent stuff that is niche. On MAL Naruto rates at 7.7, and Bleach at 7.9, very low considering these shows actually being the most popular 2 anime shows.

[Reckoner]

Are people who frequent ANN or MAL representative of the whole view base? What about anime casuals who don't really post on the internet that much or at all and just watch it in Japan on TV?

Western and Japanese perceptions are often different.

In the example of Mushishi, you noted that it may be overrepresented. That is why I mentioned it as an example of a bad survey. Tons of people probably don't like this show, though among certain critics, it is valued highly. Genre bias.

Popularity trumps "quality." People will think that what is popular is all that there is to watch. This is true in all areas of entertainment. Anything that gets a little momentum ends up being highly acclaimed regardless of the actual quality.

Saying that the top 10% is usually "good" is pretty much moot to me, because 90% of everything is almost certainly crap. Usually what I and I think many others care about is distinguishing between these top 10%.

[Malkuth]

Originally Posted by donquigleone

Isn't the quality in proportion to how many, and by how much, people will enjoy the work? So surely rating lists as a informal survey of a large number of people's opinions is a decent gauge of how much they enjoyed a series, and hence how good it is.

OMG! of course not! Let me try to state two reasons:

First problem is how the rating question in both those sites (and many more) is presented... in complete absense of context. What are the users rating, animation, story, enjoyment, value... what version are they rating, original, speed-subs, crap-dubs... how biased are they when they are rating, getting pissed at a disloyal adaption, getting over-exited with yuri, shocked to see sex scenes, emotional for rating an anime they saw when they were kids (just check the rating Kabamaru has in MAL for LULZ)... so which one, or some, or all of these. In absence of context they're choices are at best... well... random

Second and most important reason is that popularity is not correlated with quality. Studio 4°C has excellent animation quality, but almost none knows what they have published; many hentai anime have amazing plot, but everyone discards them because they have sex scenes; similar with yaoi, since most anime viewers are homophobic...

If anything popularity and high ratings are related to political correctness (as defined by the majority of the fans frequenting the site), that's also the reason we have such diverse popular anime when you consider TV ratings, BD/DVD sales, foreign anime sites, 2chan, etc.

Originally Posted by donquigleone

What other slightly objective approach can you use to even try to gauge quality? Individual opinion is not that useful as people will always disagree, and you can always find someone that enjoyed even turds like Dragonaut. The best you can do is to average a large number of people's opinions and try to derive a number from it.

Opinions are by definition subjective, averaging them does not make them objective. It is a useful oversimplification to say that if most people like a show, it is more likely that would like it too. But keep in mind that when you begin to form your own opinion by watching more shows, this oversimplification loses its meaning, since you begin to understand more what you like and what not.

By the way, I enjoyed Dragonaut as much as Death Note, Code Geass, and Clannad, for being such fails, becaming wins, from my perspective

Originally Posted by donquigleone

I'm not saying rating lists are perfect, far from it. They often have an unfair bias against stuff that's older, for instance. But if you can suggest a better approach I'm all ears.

Rating lists give you some idea of what the popular consensus was (and you are more likely then not to be part of that consensus most of the time), and only 2 or 3 of Deen's shows have performed well by that standard. Those 2 or 3 also happen to be the ones most complimented anecdotally as well (higurashi and Maria sama ga miteru).

Finally, be mindful that popular consensus is a dangerous thing... many crimes have been committed with crowds cheering in their own ignorance

[DonQuigleone]

Originally Posted by Reckoner

Are people who frequent ANN or MAL representative of the whole view base? What about anime casuals who don't really post on the internet that much or at all and just watch it in Japan on TV?

How representative they are is certainly up for debate. However for our purposes it works well, as being anime fans ourselves we are more likely then not to agree with their assesments, if we were to watch the same thing, helping us find out what is "good"

Western and Japanese perceptions are often different.

Also true, but Anime rating sites are international. Obviously there's an english language bias, but English being the second highest language of choice for Anime fans (behind Japanese of course), I'd say many ratings are taken internationally, so it's less of a "western vs. Japanese bias" as a "International vs. Japanese bias". As far as I know a lot of english speaking Anime fans are Singaporean, Filipino etc.

In the example of Mushishi, you noted that it may be overrepresented. That is why I mentioned it as an example of a bad survey. Tons of people probably don't like this show, though among certain critics, it is valued highly. Genre bias.

This is counter to your earlier "American Idol" asertion, that the popular sludge is best. The only thing I can say is that it tends to be a self-righting system. Generally if something unexpected rises high on the list people outside the original base will begin to watch it, and if it's undeservedly high will begin to sink subsequently, or maintain it's position if it's deserved.

Popularity trumps "quality." People will think that what is popular is all that there is to watch. This is true in all areas of entertainment. Anything that gets a little momentum ends up being highly acclaimed regardless of the actual quality.

Which is why lists can often go around the hype. High Quality stuff that people have never heard of (like Beast Player Erin) often gets known through being represented on the lists.

Saying that the top 10% is usually "good" is pretty much moot to me, because 90% of everything is almost certainly crap. Usually what I and I think many others care about is distinguishing between these top 10%.

You are absolutely correct. I am trying to distinguish between the two. To say "these titles are the top 10% of anime" we need to assess them somehow and make a judgement. It's not assessing a primary characteristic (the show itself), which would take too long if you wanted to assess every anime ever made by watching all of them.

Instead we assess a secondary characteristic that correlates with the primary, IE people's reaction. This is approach is a bit more inclusive then exclusive IE there will be almost no good shows below the 8.0 or so cutoff, but there will be some bad shows above it. It can be used to eliminate 80-90% of anime from consideration before you more closely consider the rest.

I'd say if you just watched Anime randomly you'd only come across a "good" one 10% of the time. If you use my approach and watch anything above 8.0 you'll find that jumps to about 70%-80%. If you set a higher cutoff like 8.5 it will jump to maybe 90-95%.

If you can give me an approach for figuring out what the top anime titles are that's more accurate and faster, I'm open to it.

[Matrim]

Isn't the quality in proportion to how many, and by how much, people will enjoy the work?

No. That would be popularity, not quality. Quality is subjective. More people saying that something is great only makes it more popular, not objectively better than something else. If we could measure quality objectively, producing art would be a much easier and less risky venture that it is now.

I'm saying the rating is a reflection of properties of the anime itself. If an anime is a masterpiece, people will recognize that fact and rate it accordingly.

What's masterpiece for me may not be what's masterpiece for you. And there are other factors that come into play - marketing of the anime, whether the subs are good or exist at all, the genre, etc. And the rating system is rather flawed considering a rating of say 7.683 gives a series a rank of #840 on ANN. I would venture that an incredible number of shows are rated between 7 and 8.5, which makes the whole rating thing more than a bit pointless. Too many people voting ten or nine for series that aren't that special (I am guilty of that myself). Many people who vote without watching the whole series.

[Archon_Wing]

Pro tip: Self selected samples are usually not good for anything.

[Reckoner]

Originally Posted by donquigleone

How representative they are is certainly up for debate. However for our purposes it works well, as being anime fans ourselves we are more likely then not to agree with their assesments, if we were to watch the same thing, helping us find out what is "good"

I take it you never have taken a class in statistics? One simple flaw of this survey which is fatal is simply the fact that not enough people give two shits to actually make accounts of these websites and vote. Completely flawed.

Originally Posted by donquigleone

Also true, but Anime rating sites are international. Obviously there's an english language bias, but English being the second highest language of choice for Anime fans (behind Japanese of course), I'd say many ratings are taken internationally, so it's less of a "western vs. Japanese bias" as a "International vs. Japanese bias". As far as I know a lot of english speaking Anime fans are Singaporean, Filipino etc.

An English website gets mostly English speaking people form English speaking countries. This forum included. Regardless of the interpretation, it's another big flaw.

Originally Posted by donquigleone

This is counter to your earlier "American Idol" asertion, that the popular sludge is best. The only thing I can say is that it tends to be a self-righting system. Generally if something unexpected rises high on the list people outside the original base will begin to watch it, and if it's undeservedly high will begin to sink subsequently, or maintain it's position if it's deserved.

Umm.. No it isn't. I was just giving an example of how your logic can be twisted. The Mushishi example further illustrates the inadequacies of such survey techniques despite the show in my opinion actually being a very good one.

Originally Posted by donquigleone

Which is why lists can often go around the hype. High Quality stuff that people have never heard of (like Beast Player Erin) often gets known through being represented on the lists.

It is why lists further propagate the popularity of certain animes over others.

Originally Posted by donquigleone

You are absolutely correct. I am trying to distinguish between the two. To say "these titles are the top 10% of anime" we need to assess them somehow and make a judgement. It's not assessing a primary characteristic (the show itself), which would take too long if you wanted to assess every anime ever made by watching all of them.

Instead we assess a secondary characteristic that correlates with the primary, IE people's reaction. This is approach is a bit more inclusive then exclusive IE there will be almost no good shows below the 8.0 or so cutoff, but there will be some bad shows above it. It can be used to eliminate 80-90% of anime from consideration before you more closely consider the rest.

I'd say if you just watched Anime randomly you'd only come across a "good" one 10% of the time. If you use my approach and watch anything above 8.0 you'll find that jumps to about 70%-80%. If you set a higher cutoff like 8.5 it will jump to maybe 90-95%.

If you can give me an approach for figuring out what the top anime titles are that's more accurate and faster, I'm open to it.

This idea is very flawed. It often assures popularity, and nothing else. 90% is shit, and that 90% is not necessarily the bottom 90% of the ratings.

[DonQuigleone]

Originally Posted by Malkuth

OMG! of course not! Let me try to state two reasons:

First problem is how the rating question in both those sites (and many more) is presented... in complete absense of context. What are the users rating, animation, story, enjoyment, value... what version are they rating, original, speed-subs, crap-dubs... how biased are they when they are rating, getting pissed at a disloyal adaption, getting over-exited with yuri, shocked to see sex scenes, emotional for rating an anime they saw when they were kids (just check the rating Kabamaru has in MAL for LULZ)... so which one, or some, or all of these. In absence of context they're choices are at best... well... random

Kabamaru is certainly a weird one. If subs actually existed for it, I'd say it would be a LOT lower. But when you have a large enough sample size these biases tend to get eliminated. For instance you may imagine DBZ would have a similiar bias, but it does not get rated highly anywhere.

Second and most important reason is that popularity is not correlated with quality. Studio 4°C has excellent animation quality, but almost none knows what they have published; many hentai anime have amazing plot, but everyone discards them because they have sex scenes; similar with yaoi, since most anime viewers are homophobic...

I think you'll find if you actually examine rating lists this is not borne out. Junjou Romantica (Yaoi) gets rated highly on MAL, Niche titles generally are actually OVERrepresented. Hentai is a special case, as satisfaction with Hentai is a lot more to do with how much it turns you on then anything else.

If anything popularity and high ratings are related to political correctness (as defined by the majority of the fans frequenting the site), that's also the reason we have such diverse popular anime when you consider TV ratings, BD/DVD sales, foreign anime sites, 2chan, etc.

A lot of very popular Anime aren't politically correct at all.

Opinions are by definition subjective, averaging them does not make them objective. It is a useful oversimplification to say that if most people like a show, it is more likely that would like it too. But keep in mind that when you begin to form your own opinion by watching more shows, this oversimplification loses its meaning, since you begin to understand more what you like and what not.

Its an objective assessment of a subjective thing (Anime quality). Also such an approach works very well for trying things you may not do otherwise, or things that don't get word of mouth exposure.

Originally Posted by Matrim

No. That would be popularity, not quality. Quality is subjective. More people saying that something is great only makes it more popular, not objectively better than something else. If we could measure quality objectively, producing art would be a much easier and less risky venture that it is now.

We can't define quality, but it can be measured. If we judged based on "numbers of viewers", we get popularity. If we judge by how much those viewers actually like it, we're actually close to how much quality it has,

What's masterpiece for me may not be what's masterpiece for you. And there are other factors that come into play - marketing of the anime, whether the subs are good or exist at all, the genre, etc. And the rating system is rather flawed considering a rating of say 7.683 gives a series a rank of #840 on ANN. I would venture that an incredible number of shows are rated between 7 and 8.5, which makes the whole rating thing more than a bit pointless. Too many people voting ten or nine for series that aren't that special (I am guilty of that myself). Many people who vote without watching the whole series.

That's why we have to take the scores relatively. The top 10% of anime scored (or so) is what's good. That cut off, as I've said earlier, is around a score of 8.0

If the top 10% of anime had a score above 6 then I'd define the cutoff as 6. Most people rate in an inflated way as you said. So long as you take that into account the system works.

You could argue that MAL and ANN would be better if the system operating on just a 4 or 5 point scale. But what would be shown to be good would be roughly the same.

EDIT: @reckoner: If you really want to test it out, take all the anime that you would consider great, and see where they lie on MAL or ANN, they will probably all have a score over 8.0. I've watched a lot of Anime before, and I've seen very few shows that I though were amazing that MAL or ANN did not put above a score of 8. There is stuff that's rated highly that I would extremely disagree with (Gintama on MAL), but I never said it was perfect.

[DonQuigleone]

I'm taking too much time to write all this stuff.

I'll acknowledge that there are imperfections in my system, but there is no better one. If you can give me a better way to give an approximation of what shows are or aren't good I'm open to them.

I'd also have to say that this approach does not work nearly as well with other fandoms. TV.com's list is shit for instance, and IMDB only marginally works, and doesn't work at all for obscure movies.

For anime it works well as most anime fans have seen a wide variety. Quite uncommonly there are a lot of Anime fans who "watch everything".

Anyway, please give me a better approach to trying to determine whether something is good or bad, without spending the time to watch it (IE you want to know before you watch...).

EDIT: and I have studied a bit of statistics, thank you very much... And this is the only statistical approach that's available for solving this problem. Everything else is subjected to even more bias.

[Akito Kinomoto]

Originally Posted by donquigleone

I'll acknowledge that there are imperfections in my system, but there is no better one. If you can give me a better way to give an approximation of what shows are or aren't good I'm open to them.

I'd also have to say that this approach does not work nearly as well with other fandoms. TV.com's list is shit for instance, and IMDB only marginally works, and doesn't work at all for obscure movies.

Why are you placing more value in the site ratings for one medium of entertainment over another site's scores for another form of media? I'm not even talking about TV.com; it could have been any other site and even for any other medium such as music, video games, ect. Regular TV goers, especially those not knowledgeable in animedom, can just as easily claim the scores on MAL or ANN to be ludicrous.

[DonQuigleone]

Originally Posted by Akito_Kinomoto

Why are you placing more value in the site ratings for one medium of entertainment over another site's scores for another form of media? I'm not even talking about TV.com; it could have been any other site and even for any other medium such as music, video games, ect. Regular TV goers, especially those not knowledgeable in animedom, can just as easily claim the scores on MAL or ANN to be ludicrous.

Personal experience. I find MAL and ANN align "well enough" with my own taste.

For IMDB it's okay for English Language films, but falls down for all but the most mainstream of foreign films, as there's simply not a large enough sample.

TV.com's list has a lot more spamming of votes.

It's more down to the website then the medium. For instance I find ANN's list to be superior to MAL's. I don't know why this is, perhaps it's due to the type of voter a particular website attracts.

However this is more for finding new things to watch then making absolute pronouncements on the quality of something. I think the actual order MAL or ANN puts stuff in is often ludicrous, but it the general place relative the vast majority of other anime is about right. For instance FullMetal Alchemist: Brotherhood is probably not 2nd best anime ever made, but is it a great series? Yes. However there is a neophile bias in the system, this tends to decrease with time, so I'd say the system only really works well for titles that are over 2 years old.

If something is rated no.1 it does not mean it's the best ever. In fact it's impossible to pin anything as being the best ever. Instead you can seperate Anime into Classes: <6 is awful, 6-7.8 is mediocre, 7.8-8.2 could be good, 8.2-8.7 very likely to be good, >8.7 is extrememly likely to be good. All the above are modified by how you feel about certain genres etc. If you despise Mecha and Camp you'll never like Code Geass, for instance.

[SeijiSensei]

Originally Posted by Archon_Wing

Pro tip: Self selected samples are usually not good for anything.

+1

I have a long discussion about the problems with measuring reactions to programs based on self-selected samples over in the ratings thread.

[Akito Kinomoto]

Originally Posted by donquigleone

Personal experience. I find MAL and ANN align "well enough" with my own taste.

...You set up a system for seeking high-quality works based on your own taste?

Edit: I mean, you basically said the set-up you use for finding good shows is the best one, IE, looking at the top brand from MAL/ANN, then you say they fall in line more with your own taste. You basically, in effect, called your own preferences the best. It goes without saying that there's no such thing as 100% unbiased, but anyone trying to do so should avoid statements like that, right?

[DonQuigleone]

Originally Posted by Akito_Kinomoto

...You set up a system for seeking high-quality works based on your own taste?

Well it would be fairly pointless if the system recomended me works I didn't actually like, right?

@SeijiSensei: I think you're absolutely right, and that's why IMDB and TV.com's rating system doesn't work well. Everyone has too large a discrepancy in how they rate things. For a rating system to truly work most of the people rating have to have similiar rating habits, and failing that not have one type of rater be predisposed to one type of show. This does not tend to happen with Anime sites as many Anime fans watch a large number of different types of shows, so such selection biases are evened out.

But to be fair to ANN, I think it shifts down the weight of people who frequently give high, low, or extreme scores to things. You can see this on ANN where often the bars are not directly proportional to the numbers next to them, and these scores are not valued as highly. So in a sense it is lessening the problem of confirmation bias. I don't know if MAL does this as well. My guess is they don't, which may explain some of the weirder scores there. For instance look at Gintama on ANN, if you look at the stats there are 246 10 votes and only 186 9s, but the system actually rates the 9s higher, presumably because many of those Gintama votes are from accounts that only rated a small number of things "10/10". The system does even out those with abberant voting records. And I'd say ANN has the better list overall.

So I think it's unfair to say the lists are totally vulnerable to confirmation bias.

[CrowKenobi]

Originally Posted by SeijiSensei

+1

I have a long discussion about the problems with measuring reactions to programs based on self-selected samples over in the ratings thread.

I think SeijiSensei is right, let's discuss everything ratings in the ratings thread since the Studio Deen thread isn't the place for it.