Question to Encoders: File Sizes getting out of hand?

[Access]

Originally Posted by Wadjet

So really my question is this: Is it just encoders have gotten lazy or bigger files is just how it is nowadays? Are some encoders just taking the HQ raws and muxing them with a sub script?

I love the h264 codec but does it cause video files to take up more space than say xvid?

It is perfectly natural that file sizes are increasing, just the way things evolve over time as people try to outdo one another in terms of quality and with BT making distribution easy, there is no downward pressure on file sizes. In 2000/2001 a typical episode was 110-160mb, some a bit smaller. Sure there are 'repeat offenders' out there, but largely no one cares as distribution today is easy, and space is always getting cheaper. But seriously it's not a big deal, if you feel a need to save or collect files a 1500gb hard drive is only a little more than $100. In 2000 a 500gb drive was what (?) $200. or so? So the cost to store stuff has gone down, not up, likewise we now have DVD-Rs that cost as little as CD-Rs did back then.

BTW better quality always compresses better, b'cos randomness or noise is a waste to encode. In the early days (ie. pre-2000) if you were doing video capture, the best captured to raw uncompressed (or with lossless compression) meaning that raw file sizes were in the 10-20gb range for an episode. Compression was, ideally, the final step, not an intermediate one. Today, stuff that is captured has already been compressed, decompressed, and so on.

[cyberbeing]

Originally Posted by Access

In 2000 a 500gb drive was what (?) $200. or so?

Either you accidentally left off a zero or you weren't living in the same 2000 as me. In the year 2000, 500GB worth of storage space would have run ~$2000, with the largest capacity hdd available being 80GB at the end of the year.

The fact that storage space is dirt cheap nowadays is an understatement. If you are still using a 80GB or smaller hdd in this day and age (yes, even you laptop users), I think it's about time you move into the modern day and upgrade.

[Zalis]

Originally Posted by Daiz

More like file sizes getting reasonable. About damn time, I say. 230MB 720p releases look like shit most of the time.

That's mainly because people have been intelligent enough to move from 400p to 480p for SD. Also about damn time.

As a disk-space-miser/SD downloader, I've also noticed some size creep. True, the move to 480p does account for some of that. But let's take a look at the SD versions of Eclipse's Hayate no Gotoku between the 2 seasons:

Season 1, episode 1:
Video -- 704x400 29.97 fps XviD (797 kBit/s)
Audio -- 2.0 Stereo MP3 CBR (160 kBit/s)
Total Size -- 175 MB

Season 2, episode 1:
Video -- 704x400 23.976 fps XviD (1168 kBit/s)
Audio -- 2.0 Stereo MP3 CBR (128 kBit/s)
Total Size -- 233 MB

I don't doubt that mentar has a good explanation for this, but as a leecher who's not terribly well-versed in encoding technicalities, I see the following:
S2 has the same resolution as S1, but lower framerate and lower audio bitrate. Yeah, S2 has 371 kbps of additional video bitrate, but that's really not something I notice offhand. What I do notice is that if S2 were 52 episodes, it would take as much space as the HD version of S1.

I'd rather see more h264/mkv/175MB SD releases, but I guess cases like Pumpkin Scissors proved that XviD/avi is or at least was more popular with the low-enders.

[Dark Shikari]

One of the most hilarious things I've noticed is that people make a big deal about how you should use x264 for HD anime, but for SD you can "get away with Xvid."

What makes this funny is that due to how the H.264 transform works, x264 likely has a significantly greater advantage over Xvid at SD than at HD.

[TheFluff]

Originally Posted by Dark Shikari

One of the most hilarious things I've noticed is that people make a big deal about how you should use x264 for HD anime, but for SD you can "get away with Xvid."

What makes this funny is that due to how the H.264 transform works, x264 likely has a significantly greater advantage over Xvid at SD than at HD.

see

Originally Posted by TheFluff

SDTV broadcast shows (or upscaled SD shows capped off "HDTV") aren't really worth bothering with, they'll look like shit no matter what you do (...). No one cares about SD anyway.

that said, if you're making xvid encodes of a show that is only available in SD you are (of course) retarded.

on the other hand there is no real point in releasing an SD version when there is HD available; the only reason to do so is to make owners of shitty set-top boxes and very ancient computers happy, so the SD version might as well be xvid in avi since that's what they want anyway.

[Mentar]

Originally Posted by Zalis

As a disk-space-miser/SD downloader, I've also noticed some size creep. True, the move to 480p does account for some of that. But let's take a look at the SD versions of Eclipse's Hayate no Gotoku between the 2 seasons:

Season 1, episode 1:
Video -- 704x400 29.97 fps XviD (797 kBit/s)
Audio -- 2.0 Stereo MP3 CBR (160 kBit/s)
Total Size -- 175 MB

Season 2, episode 1:
Video -- 704x400 23.976 fps XviD (1168 kBit/s)
Audio -- 2.0 Stereo MP3 CBR (128 kBit/s)
Total Size -- 233 MB

I don't doubt that mentar has a good explanation for this...

Sure I do. Season 1 was based on Raws, in other words, on sources which were already recompressed, and back in those days, mostly recompressed with ASP codecs (XviD). With them and their level of detail retention, 233 for h264/mkv and 171 for XviD/avi was a reasonable decision, even though 171 was already borderline, leading to quants of 7 and sometimes even 9. Consequently, this is how S1 frames looked like in the AVI release:



In Season 2, we now have transport stream sources, and thus, a much higher quality to work with. Here's a screenshot of the same kind, scenery-wise:



The differences are glaring. Much more details are visible here. The size of the lossless screenshots is telling, too: 213k for Season 1 compared to 398k for Season 2. And so, since I had to adjust the size for the mkv release anyway (staying 233m would have caused severe quality loss), I decided to upgrade the avi size aswell. And lo and behold, we landed perfectly in the sweetspot of xvid encoding: Quants 2-5 for P- and B-frames, and occasional Quant 1 for I-frames. Exactly where you want to land as encoder.

[Kristen]

Screenshot comparisons between 10.9 GB Lagarith Lossless and 350 MB h.264:

http://i40.tinypic.com/16i5fg8.png
http://img15.imageshack.us/img15/8407/29077h.png

Taken from a high motion scene. Going to make 500 MB in about 5 hours, once I get back from class.

There is a bit more blocking on the 350 MB that the 10.9 GB Lagarith lossless, but honestly, I don't think when playing anyone would be able to tell. Correct me if I'm wrong, please.

For reference, these are the x264 compression settings used:
cabac=1 / ref=6 / deblock=1:1:1 / analyse=0x3:0x133 / me=umh / subme=9 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=1 / mbaff=0 / bframes=8 / b_pyramid=1 / b_adapt=1 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc=2pass / bitrate=1864 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00

[comatose]

High motion scenes don't really matter, because nobody will notice. I bet those white lines in the background change completely every frame (probably alternates?). Even websites that do comparisons (like ji-hi) avoid these scenes because of this.

You should be testing on a scene with a detailed background to see how the detail was affected, and if stuff like grain were smoothed out.

[jalapeno]

Can't say I noticed an increase in file size, but then I don't care.

I'd attribute it to the increase in broadcast resolution. Also there are more groups than good encoders.

If you want small files, convince people to settle for less than possible quality. Or subbers to leave that for commercial releases.

BTW are there shows that need HD? SHAFT stuff occasionally has very tiny screentext, but I suppose they rework that for the DVD.

[Kristen]

High detail background (ground she's standing on)

Lossless: http://i44.tinypic.com/23idjqv.png
350 MB: http://i43.tinypic.com/21jt5hc.png

(Sorry that it's not the exact same frame. I think my encode dropped this frame for whatever reason...)

[Daiz]

bframes=8 / b_adapt=1

Slighty offtopic: Use b_adapt=2.

More on topic: Why don't you just do CRF encoding? It'd be a lot wiser than doing every episode with the same filesize. Sure, you might have to experiment a bit to find a satisfactory CRF value but once you find it you don't have to worry about it anymore.

[Kristen]

Originally Posted by Daiz

Slighty offtopic: Use b_adapt=2.

More on topic: Why don't you just do CRF encoding? It'd be a lot wiser than doing every episode with the same filesize. Sure, you might have to experiment a bit to find a satisfactory CRF value but once you find it you don't have to worry about it anymore.

I know I should use b_adapt=2, and I've been doing it in my K-On encodes, but

Originally Posted by Kristen

threads=3

really slows it down a lot. And I didn't want to spend 8 hours on this encode with MERange of 32 and b-adapt 2 like I find ideal on a 720p encode when I have 14 encodes backlogged at me, and another 2 coming on Thursday. If I had a better computer, I would too. >.<

Anyways... With my crf encoding tests, I've found that it generates a lot larger file at the same average quant than with 2-pass. Like, with K-On, I tried crf 18 and I got a P-quant of 16.8 at 200 MB. I then tried 2-pass at 100 MB, and I got a P-quant of 16.7. These include 20 MB of audio, so it's more 80 MB vs. 180 MB.

[Tofusensei]

I've noticed i7 procs make a tremendous speed advantage in x264 encodes. If you are not using one, I highly recommend upgrading. Put some of that donation money to good use.

[Dark Shikari]

Originally Posted by Kristen

Anyways... With my crf encoding tests, I've found that it generates a lot larger file at the same average quant than with 2-pass. Like, with K-On, I tried crf 18 and I got a P-quant of 16.8 at 200 MB. I then tried 2-pass at 100 MB, and I got a P-quant of 16.7. These include 20 MB of audio, so it's more 80 MB vs. 180 MB.

CRF should be within 1% of 2-pass at the same bitrate in every regard. If you're getting any significant difference, the issue is almost certainly a PEBKAC error.

[Waryas]

really slows it down a lot. And I didn't want to spend 8 hours on this encode with MERange of 32 and b-adapt 2 like I find ideal on a 720p encode when I have 14 encodes backlogged at me, and another 2 coming on Thursday. If I had a better computer, I would too. >.<

Slightly offtopic : That's why you should encode on a server
100mbps upload, you can directly mux/release as soon as it's done encoding.
A quad server with 100mbps upload (metered or unmetered, doesn't matter for encoding) is like 100e
A i7 OVH is 99e.

inb4bashforusingawindowsserver

[Vide]

Originally Posted by Heibi

You know, after reading this thread, all you people should just be happy that the file size isn't ZERO. i.e. no FREE fansubs. Remember, the people doing the translation, editing, timing, and encoding aren't really getting paid. So sit back, pop a beer, soda, wine bottle or favorite beverage and enjoy the FREE show.

That is no excuse when there is room for improvement. Just because something is free doesn't mean that you can't point out things that could be improved upon. Not really related to the matter at hand, but generally speaking.

As for the topic at hand; personally I don't mind the current file sizes. Increase them as much as you want for all I care. But then again I have plenty of free space, and an umetered Internet connection. Not everyone does, but perhaps they should start trading USB flash drives with friends that do. Not really a good excuse these days, we've come a long way since 56k modems.

[Heibi]

Originally Posted by Vide

That is no excuse when there is room for improvement. Just because something is free doesn't mean that you can't point out things that could be improved upon. Not really related to the matter at hand, but generally speaking.

As for the topic at hand; personally I don't mind the current file sizes. Increase them as much as you want for all I care. But then again I have plenty of free space, and an umetered Internet connection. Not everyone does, but perhaps they should start trading USB flash drives with friends that do. Not really a good excuse these days, we've come a long way since 56k modems.

True, improvement is always good. As long as the picture and sound are good the size isn't that important. Of course giant files(500+) for 23 minute shows probably is a bit much.

I was just pointing out that it seemed a little ridiculous to complain about a free product. That's like someone giving a starving man food and he complains that there is too much bread.

[Waryas]

Of course giant files(500+) for 23 minute shows probably is a bit much.

You never have too much bitrate when you're compressing to a lousy format.
The difference will be there (even if they aren't visible at first glance)

[Vide]

Originally Posted by Heibi

I was just pointing out that it seemed a little ridiculous to complain about a free product. That's like someone giving a starving man food and he complains that there is too much bread.

Well, the bread could also be bad, and he could get sick. Then if he have no health insurance he would most likely die alone on the streets. Now, downloading free fan subtitles won't kill you, but the translation could be way off mark, timing just awful and the encoding gives you eye-cancer.

Don't take my posts too seriously though, I'm not trying to be mean, honest.

[Daiz]

Originally Posted by Kristen

I know I should use b_adapt=2 blah blah

That's why you use lower amount of B-frames. B-frames 3 & B-adapt 2 is generally superior in all cases when compared to B-frames 16 & B-adapt 1.

Anyways... With my crf encoding tests, I've found that it generates a lot larger file at the same average quant than with 2-pass. Like, with K-On, I tried crf 18 and I got a P-quant of 16.8 at 200 MB. I then tried 2-pass at 100 MB, and I got a P-quant of 16.7. These include 20 MB of audio, so it's more 80 MB vs. 180 MB.

You're doing something wrong. Also, you shouldn't blindly trust quants (just like you shouldn't trust any automated metric), you need to look at the actual video.