CCET - The Core Community Encoders Toolkit (Proposal)

[gumbaloom]

Ok,

So codec packs have been done to death but the war has pretty much in the fansub community been won by the CCCP and they're doing a mighty fine job wiht it too.

However it's just struck me what about all us encoders in the world?. Don't you find it a pain having to find all the various bits and pieces you need for running encodes and dealing with all the various stuff you have to do like audio codecs, AVS filters, x264 encoders and what not.

Wouldn't it be good if there was a community project which assembled all of the main encoders, tools and so on in to one exe and installed them all for you to cut down on time and effort whenever you get a new machine or reinstall? Granted each encoder has their own individual encoding method and special filters that they'd never disclose under pain of death or injury but theres a lot of common filters and tools and encoders that everyone uses. Why don't the CCCP people get together to make a CCCP like toolkit for all us encoders out there.

Discuss!

-gumbaloom

[XyFreak]

The answer is simple:

Every encoder is (should be) able to get his stuff together.
If he can't he isn't fit to be one....

Question: What if the encoder just doesn't want to redownload all that stuff after formating?
Answer: There's WinRAR, 7-Zip ..... and even windows has some batch-file-support

[martino]

Originally Posted by XyFreak

Every encoder is (should be) able to get his stuff together.
If he can't he isn't fit to be one....

Totally agreed. And if you can't even get a dozen of tools yourself (took me about 5-10 minutes to get about 20 or 30 different tools on my new machine -- is it really that much work?), are you willing to spend time making up a filter chain or something similar? Sounds to me like not really...

[TheFluff]

For me personally the pain of setting up shop on a new computer isn't so much the installation of the tools, it's tweaking them (moving avisynth filters around between autoload/nonautoload folders, modifying paths in .avsi's, hacking YATTA's filterlist.txt etc. etc.) so for me personally this wouldn't be that useful. I can imagine it'd be sort of convenient to have all the tools in one place though.

[gumbaloom]

oooo talk about negative reaction o_o. "If you can't find the stuff yourself and configure it yourself you're not worthy to join our ranks"

To a point I agree but at leas for me I find configuring a machine an annoyance and feel that my time could be better spent doing more constructive work like working on actual fansubs.

And well n00b encoders have gotta start somewhere right or do we want to keep the art of encoding a black art that is only accessible if you have been indoctrinated by one who has already been given the knowledge?

-gumbaloom

[TheFluff]

Originally Posted by gumbaloom

"If you can't find the stuff yourself and configure it yourself you're not worthy to join our ranks"

I really don't like that sort of reasoning either, but the problem with this proposal is that there's so many ways to do the same thing. H.264 encoding with meGUI or x264 CLI (and a .bat file)? XviD with xvid_encraw or VfW in VirtualDub? Vorbis with OggDrop, BeSweet, some CLI utility, something else? What lossless codec, lagarith, ffvhuff, original Huffyuv, VBLE, something else? Use YATTA or write Avisynth scripts yourself? What Avisynth plugins should be included? And so on and so forth; the pack would not only be huge but there'd be a monstrous amount of configuration options to pick from. I think if you really want to do this you'd be better off writing a script customized for your own needs.

Actually what I'd like to see more is a good fansub encoding guide with links and instructions on how to use all the common tools.

[Mentar]

gumbaloom: The general idea is good - I like it! However, I wonder if there's a realistic chance to achieve that.

With playback, the task is _relatively_ straightforward: Come up with a set of tools and filters which properly and efficiently play back material of different formats. And realizing THAT (with CCCP) was already a hell of a task.

Now, with encoding, things get much more murky. The part which I'm worried about the most is the avisynth filter selection. Here, MANY mutually exclusive filter versions are floating around. Take the critical masktools, for example: I know for a fact that if I ever updated it to the "most recent" version, alot of my avisynth scripts (the famous .avsi files) would simply cease to work. Likewise, there are many, MANY warpsharp.dll versions out - almost all of them incompatible - which produce very different results.

Who is supposed to decide which versions to include? And even if whoever selected it would be the God of Encoding and 100% fair, I wouldn't use a preconfigured pack out of fear it would break my scripts. There is no "right" or "wrong" on this level, but endless choices.

Therefore, such a pack could only be aimed at "new" encoders. And even then, I'm almost too curious which tools would be selected, and how it would match up against my routine which has developed over all these years. I bet that a discussion over something so trivial as how to mux your encodes could be very very long ^_^

The best chance I'd see would be to have one person (or a very very small group) start with a setup, and then maintain it. A committee would kill the idea before it sees the light of day.

[ReAn]

Originally Posted by gumbaloom

Ok,

So codec packs have been done to death but the war has pretty much in the fansub community been won by the CCCP and they're doing a mighty fine job wiht it too.

However it's just struck me what about all us encoders in the world?. Don't you find it a pain having to find all the various bits and pieces you need for running encodes and dealing with all the various stuff you have to do like audio codecs, AVS filters, x264 encoders and what not.

Wouldn't it be good if there was a community project which assembled all of the main encoders, tools and so on in to one exe and installed them all for you to cut down on time and effort whenever you get a new machine or reinstall? Granted each encoder has their own individual encoding method and special filters that they'd never disclose under pain of death or injury but theres a lot of common filters and tools and encoders that everyone uses. Why don't the CCCP people get together to make a CCCP like toolkit for all us encoders out there.

Discuss!

-gumbaloom

Although my ENCODING] Tools are minimal since what I do for anime these days involves very simple encoding techniques (Mainly LAGS Overlays) I have an approach to maintaining & migrating my toolsets between new machines.

I've set up a local SVN repo on my main box, Inside this repo I house a variety of command line tools I use on a day 2 day basis as well as installers for various things, such as CCCP / VLC / Winamp / Etc...

I've named the installer files in a simple way: <Appname>-Installer.exe

Every time I move to a new version of an installer or find a new command line app I overwrite the existing file and commit to the SVN repo.

This updates the HEAD revision with the newest installer, but the key to the setup is, if down the road I need a specific older version of any said app/codec/etc... I can just pull it out of the repo since I label all my Commits w/ changes & versions numbers.

When i go to set up a new machine, I check the REPO out onto a usb key and im set, run my installers and copy the command line apps and I'm done.

--------------------------------

As far as an encoding standpoint in concerned, the config is the only part that wouldn't be covered w/ this, but frankly even with CCCP, a fresh install I find myself re-doing most of the initial configs to suit my setup, tweaking settings etc...

So in my opinion a CCET would just end up in the same kinda situation. I could save a couple steps by combining it all into one installation but frankly im going to have to agree with XyFreak.

[gumbaloom]

ok...

So granted there are a set of common tools right?

Common Tools -
mp4box
vdubmod
x264.exe
megui

codecs -
xvid
ffdshow
x264
lamemp3

filters yes that's a tricky one. I'm not condoning that there is a particular "setup" in the pack more that all of the uuber common tools and codecs get installed and then the rest of the filters and what not is left in a nicely organised "fansub" directory. Like a directory with all the different warpsharp dll's eg warpsharp2.1.dll warpsharp_blahversion.dll

The point is not to advocate a particular setup or force people to do things a certain way more to have the most absolutely common tools/files in one place from which is kept updated even just an installer with access to an online repository or something from which encoders can configure it to their own needs and add their custom avsi scripts and custom special filters etc etc etc

-gumbaloom

[ReAn]

Originally Posted by gumbaloom

Common Tools -
mp4box
vdubmod
x264.exe
megui

codecs -
xvid
ffdshow
x264
lamemp3

Your list is missing Lagarith / HuffyUV (anyone actually still use that?) as well as AviSynth, VobSub(TextSub) which does get installed with CCCP, but you might want to include it. VDub (standard)

I'm sure we're missing a bunch more too, but right now I can't think of any.

[gumbaloom]

I wasn't trying to give a definitive list just an example of what I meant by common tools. Like I said theres a lot of stuff that EVERYONE uses like your xvid's your x264's your vble's and what not. Theres a lot of common ground which if formalised in to a pack with all the extra (publicly available) filters.

Like how often have you screamed when you realised you had the wrong version of vsfilter.dll ?. Wouldn't it have helped having a directory with all all the different vsfilters all in one place

I could imagine the avs filter directory to be like

AVS FILTERs
-
->VSFilter
---VSFilter227.dll
---VSFilter229.dll
->WarpSharp
---WarpsharpVersionX.dll
---WarpsharpVersionY.dll

and so on and so on.

-gumbaloom

[Yumi`]

If you're going to make an easy pack you might as well use mp4box, x264, lame and xvid_encraw from megui's autoupdate, no need to include them (again).
Ffdshow and vsfilter are already included in codec packs, which will be undoubtedly be used by ppl who'd turn to such packs.
As for filters, it'd be better providing a good library link, since there is a massive choice (clicky).

[Doughnuts]

I'm probably not fit for this discussion because I only know the basics about encoding, but from my PoV, I think such a pack would be ideal even if it is only aimed at beginners.

The main problems I've had so far are not knowing what software/plugins to use, even though there are plenty of guides telling me (the negative effect of that is that every guide tells me something different). One such problem I had was when I wanted to hardsub something, I googled some information on it, and found what appeared to be plenty of different VDub filters, none of which worked with ASS. It was only hours later that I found out I could stick .vwf onto DLLs I already had. And yes, I felt very stupid, but I wonder how many others have read the wrong guides.

Like TheFluff mentioned, a good guide would be nice, except that guides are usually written by one person, and they go out of date (which software to use etc). It's left up to that one person to maintain it, otherwise it becomes an annoyance to would-be encoders looking for reliable information. A wiki-like system would be ideal for keeping such guides up to date though.

The problems the experience encoders are pointing out (different choices, configurations etc), I think are exactly the problem that could be solved. A single app that could monitor your installation and configurations and record all that information into a single file. As such, if you migrate to another machine or whatnot, you would only need to load the one file and have it automatically download, install and configure.

[edogawaconan]

thing is, unlike ds filters that is the main component in "codec pack", most encoding tools doesn't mess up with registry (with exceptions being: avisynth (installation only), vfw/acm codecs, and... what else? vdub settings?)...
(my point is: most of configurations and tools can be made portable)

[pichu]

Again, I only like to read the first post and ignore all of the other posts, so this is my two cents.

Try to develop a website to provide encoding tools, automations, and guides for specific encoding jobs. It will probably be similar to that of doom9 and more intuitive than your encoding pack.

Developing filter chains requires experiences, which I doubt your encoding pack can offer... unless you provide programs that can analyze the video and plot out which filter chains are needed. But then, no one can really trust in AI, heh.

Also, the most challenging part to encoding, i believe, is to develop filter chains, to inverse telecine... more than using the codecs, encoder, and the muxer. As for providing filters and codecs, I don't think it will be useful to new encoders, as they don't know how to use them. xD It will certainly be useless for experienced encoders, as they are so used to their stuff already.

[checkers]

For video, there are three tools everyone will use: avisynth, x264 and a frontend (includes things like a .bat wrapper).

• x264 is easy to include, so no problem there.
• Avisynth can be done reasonably well (which is all you should aim for). Include all the big plugins that aren't incompatible with anything. These days, that's everything, only old plugins fail. If you include those top 20-40 plugins, most of the bases are covered. avisynth scripts should be distributed separately unless released as pseudo plugins like didee's stuff.
• After this you have to pick a frontend, which is getting into religious territory and therefore probably a bad idea unless you invent the new JesusEnc.

Yes, this misses out stuff, but this should never be an exhaustive pack. Stuff like ancient scripts with ancient dependencies should be done by hand, all the sane stuff can be done automagically.

Here's my list of essential avisynth plugins. Remember, the only bad thing about including extra filters (assuming they are sane & compatible) is that the download takes 10 seconds longer:
awarpsharp
deflicker
degrainmedian
dfttest
dup
ffmpegsource
fft3dfilter
fft3dgpu
gradfun2db
hqdn3dlimitedsharpenfaster
masktools1
masktools2
mvtools
removegrain package
tdeint / TIVTC package
ttempsmooth
yadif
warpsharp


Someone else can do audio, but I just feed everything through MeGUI these days. It does everything via directshowsource.

[Dark Shikari]

Add Deen to the list. An ancient denoiser, but still shockingly effective for anime in some cases when used correctly.

You'll also want some popular scripts, like Didee's incredibly effective TemporalDegrain or FastLineDarken().

[Quarkboy]

Originally Posted by Dark Shikari

Add Deen to the list. An ancient denoiser, but still shockingly effective for anime in some cases when used correctly.

You'll also want some popular scripts, like Didee's incredibly effective TemporalDegrain or FastLineDarken().

I second a vote for deen!

Although frankly I think it's overkill except with older/crappier sources. For any caps from the past 2 years or so there are better things, IMHO. But for TV rips from the early 2000s or earlier it's very useful.

[Nicholi]

I second the removal and complete forgettance of VDubMod's existence. Of course replaced with AVI-Mux GUI if you really need something to mux to AVI.

As to the package itself, there are so many small apps you could add to this list that it would be insane to host. Besides the abundance of AviSynth filters. I wouldn't bother.

[Yuudai-kun]

I think the idea isn't bad, though probably too much work and effort.

If anyone is interested, i found a package long time ago that is very similar to what this thread is about:

Click

I haven't actually used this package so i don't know if it's any good, but the guide for filtering helped a noob like me