AnimeSuki Forums

Register Forum Rules FAQ Members List Social Groups Search Today's Posts Mark Forums Read

Go Back   AnimeSuki Forum > Anime Related Topics > General Anime > Fansub Groups

Notices

Reply
 
Thread Tools
Old 2006-01-13, 16:57   Link #1
Suchy
Junior Member
 
Join Date: Jan 2006
Fast Hardsubs Updater (aka FHU)

I want present You a programme which can be usefull for users and hardsub groups.

Sometimes fansub have contained mistakes or errors in subs, audio, tags, etc.
The most often solution is publish next version of hardsub. That force users to redownload whole file (ex. 233 MB). Of course we can release component files, and user can remux it oneself, but the file will have other CRC in each cases (It' depends of remuxing, time and computer) and it's uncomfortable. That's why We can't share this files on p2p and not always user knows how to remux files oneself.

Fast Hardsubs Updater (FHU) is aimed to make it possibile to create an updating patch for the following files: MKV, OGM, MP4, MPG, MPEG, M1V, M2V, MOV, RMVB, RM and AVI (AVI is rather in case of making the patch modifying RAW into fansub); or the repairing patch (which means without the remuxing process). Actually target extension is always MKV.

This updating patch is supposed to enable you to update the files in such way that the updated file can be easily made seeded on p2p (so, that this file would always have the same CRC, no matter who patches it and on what computer). This will let you popularize only the part that needs to be altered instead of making all the corrected files accessible. Thanks to this, people who’ve got the older version of the file will not have to download e.g. 233 MB but only 1-2 MB. The updated file can be successfully popularized on p2p networks without fear of occuring the hashing error.

The repairing patch is aimed to correct the file downloaded in a wrong way, or if the file has been released with minor errors.

The priority is to save the patching person’s work. The programme is relatively easy to understand (foolproof :>), yet to prepare a patch you need to have some knowledge, so I suggest reading added ReadMe with understanding

The programme contains checking algorithm, which verify stages of updating - from begin to end.

- updating files
- checking CRC of updated file
- lang support (PO standard)
- remuxing files
- and other

More about FHU is contained in ReadMe

Programme: Fast Hardsubs Updater (FHU)
Actual version: 1.1.2.108 stable (Download Changelog)
OS: Windows

Author: Suchy
Idea: Suchy & Yang
Beta testing: Yang
Special thanks for: Anihilator

Last edited by Suchy; 2006-02-06 at 06:48.
Suchy is offline   Reply With Quote
Old 2006-01-13, 17:00   Link #2
NightWish
…Nothing More
*Administrator
 
 
Join Date: Mar 2003
Age: 44
Quote:
Originally Posted by Suchy
I wondered in which section of Doom9 I should write this post.
I want to apologize for mistake if I choosing wrong section.
Err, I think you missed the "section" by miles... you're so far out you're not even on Doom9 anymore...
NightWish is offline   Reply With Quote
Old 2006-01-13, 17:05   Link #3
Suchy
Junior Member
 
Join Date: Jan 2006
err.. ups
it's true..
There shouldn't be Doom9 but AnimeSuki.
I posted it on Doom9 too, that's why i made this mistake.
I'm sorry..

P.S.
Corrected
Suchy is offline   Reply With Quote
Old 2006-01-13, 17:09   Link #4
NightWish
…Nothing More
*Administrator
 
 
Join Date: Mar 2003
Age: 44
One of your links was broken, the other has banned material on it... even if it is just patches of banned material. I'll leave the thread for now in case the Fansub Groups take interest... though I personally can't see the advantage of it.
NightWish is offline   Reply With Quote
Old 2006-01-13, 17:39   Link #5
Suchy
Junior Member
 
Join Date: Jan 2006
Oh... pardon. I forgot 'bout there not allowed licenced anime.
It's good rule and I appreciate it so I don't have any pretensions for removing links to licenced things.

P.S.
And valid link to FHU (i checked it - 100% valid):
FHU 1.1.1.104 stable
Suchy is offline   Reply With Quote
Old 2006-01-13, 17:51   Link #6
Ledgem
Love Yourself
 
 
Join Date: Mar 2003
Location: Northeast USA
Age: 38
Interesting concept, but I can foresee a difficulty with this when it's applied to the "real world": how would people know if they were downloading an old version of a file or a patched version? One nice thing about the CRC32 value is that it lets you know if your file is corrupt, or if you have a different version. In patching files so that they're updated/corrected yet bear the same CRC32 value of the original file, you're just making it that much more difficult to discern whether a file has been patched or not.

I like the idea and can see that it'd be very useful for people who are of limited bandwidth, but that CRC32 issue bugs me. You'd have to go in and view the file for yourself to see if it had already been patched or not - and that's assuming that the fansub group was kind enough to mark somewhere that "this has been patched." It'd just become too difficult to keep track of versions, in my opinion. New versions aren't released often enough that it'd be a major issue to fansubs overall, but it'd make keeping track of versions difficult.
__________________
Ledgem is offline   Reply With Quote
Old 2006-01-13, 18:23   Link #7
Suchy
Junior Member
 
Join Date: Jan 2006
An example situation:
1) Manganime releases hardsub of Bleach:
[Manganime]Bleach - ep1 [v1].mkv
2) Pepole download it (ex. via torrent from our tracker)
3) Now, we noticed mistakes in translation, etc.
We fix this, and release version2:
[Manganime]Bleach - ep1 [v2].mkv
4) Now on tracker is only version 2 (older wrong release is removed by us because we don't want distribute version with errors anymore).
5) Users whose didn't download Bleach yet, can download version2 from begin.
Users whose have already downloaded version1 can download smaller patch and update it to version2

It's Sipmle.

Created patch can have a name: Bleach - ep1 (v1 to v2 update).exe or similiar. Moreover FHU check source file and if it's not file [Manganime]Bleach - ep1 [v1].mkv then FHU didn't update file. But if file is good FHU updates it and after finishing FHU check CRC to estimate if updating was succesfull. If not, then, removing tem files and on HHD leaves our version1


P.S.
Updates are only form version A to Z, not from A, B, C to Z

P.S.S.
You misundestanding me.
CRC are the same but in updated version (not for previous and updateg version, it's imposibble)
If You remux version1 to version2 by oneself version2 will have diffrent CRC in any case when You remux file (remux CRC is AABBAABB, remux the same files 5 minutes later and CRC is diffrent). That's why various users willn't share files on p2p after handmade remuxing.
FHU remux it in such way, that any user will have the same CRC of destination file.

For example:
version1 of file X have CRC: AC12AC12
After update we have version2 of file X , always with CRC: EF34EF34

Of course diffrent patches will give us diffrent CRC.
For Example:
Bleach update v1 to v2, gives AAAAAAAA -> BBBBBBBB
but
MaiOtome update v1 to v2 gives CCCCCCCC -> DDDDDDDD

Last edited by Suchy; 2006-01-13 at 18:39.
Suchy is offline   Reply With Quote
Old 2006-01-13, 19:21   Link #8
bayoab
Senior Member
 
Join Date: Nov 2003
If I understand this program right, it requires there to be soft subs and just edits the sub file, remux's the file, and then pads it to be the correct crc. (By automating it, it makes sure it remux's the same way everytime and therefore prevents a different file from being created)? If so, this only will work with softsubs. (Or does this modify the binary data by actually modifying the file length and pushing the data into the correct place?)
bayoab is offline   Reply With Quote
Old 2006-01-13, 19:59   Link #9
Suchy
Junior Member
 
Join Date: Jan 2006
Exactly! This is overall the simpliest scheme of working FHU.
Of course not only subs can be updated, but wrong sound streams or wrong tags, chapters, attachments, etc. can be remuxed.

Actually it allows to update hardsubed video stream too (without remuxing), but only when source (before update) and destination (after update) files have exactly the same lenght. If lenghts are diffrent even about a byte it can works unpropertly.

But in next version I'm going to include updating any types of data, both diffrents lenght and the same lenght. Something like a binnary diff from Linux systems. I hope it will be sucessfully done.

P.S.
It can be used to faster redist. in distro team too.
For ex:
Coder encode ep and send it to distro team (usually via net). Everybody from distro have this version, and now coder must remux the file (it's no important why). He remux and sends patch to distro team.

There is another posibility. Group A (english) made DVD-Rips. Group B (for ex. finnish group) want release this anime, but they haven't source. If the have agreement from Group A for using their video streams, situation is very nice. These finnish users whose have already download DVD-Rips form Group A, can download patch and update this release to release which was made by finnish group. Moreover, they will have exactly the same files the users whose downoad finish release in traditional way so they can shareit via p2p.
Suchy is offline   Reply With Quote
Old 2006-01-14, 03:29   Link #10
lamer_de
Member
 
 
Join Date: Jun 2003
Location: somewhere far beyond
Quote:
Actually it allows to update hardsubed video stream too (without remuxing), but only when source (before update) and destination (after update) files have exactly the same lenght. If lenghts are diffrent even about a byte it can works unpropertly.

But in next version I'm going to include updating any types of data, both diffrents lenght and the same lenght. Something like a binnary diff from Linux systems. I hope it will be sucessfully done.
Please have a look at the following 2 programs / projects:

Zidrav: http://sourceforge.net/projects/zidrav
PAR2: http://parchive.sourceforge.net/

They allow byte patching, and par2 even works if there's a (little) offset. So, yeah, such tools already exist ^_^

Why aren't they used more (I've seen at least 1 zidrav or similar patch, but don't remember from which group)? Because it's usually more hassle for both sides (encoder & leecher) to create/apply the patches than just to reencode/click on a BT link.

I haven't looked at your program, but an automatic softsub remuxer seems like a good idea. But then, I don't remember seing a v2 softsub release ever

CU,
lamer_de
__________________
Proud to be a Warezubber!
lamer_de is offline   Reply With Quote
Old 2006-01-14, 05:11   Link #11
Cythraul
Member
 
Join Date: Dec 2005
At least for mkv, automatic softsub remuxing can be done with a simple batch file and a bundled mkvmerge. I know at least two groups that have done it.

And I remember applying a few zidrav patches, but they were from a now dead dvd-ripping group.
Cythraul is offline   Reply With Quote
Old 2006-01-14, 08:12   Link #12
Suchy
Junior Member
 
Join Date: Jan 2006
FHU, have included zidrav already. That's why updating the same filesize files is possible. We can remux any stream of file because mkvmerge is included too. In future I want include more updating algorithms like mentioned PAR2 (thx for info, lamer_de ). Man whose will create patch, will be can choose algorithm update form all available in FHU. It allows create possible small patch (for ex. when filesizes are the same then use zidrav, but when there are some offsets then use PAR2 or diff, etc.).

The most important thing for typical user is easy-for-use patch.
He musn't know how it works and how to do it.
Just run patch and choose file to update.

P.S.
Yes. Bundled MkvMerge, softsub files and batch is the simpliest way to update, but in this situation CRC of target file will be other in any updating so sharing updated file in p2p is impossible. Moreover, we don't know if file updated propertly or if we update valid file
Suchy is offline   Reply With Quote
Old 2006-01-14, 15:11   Link #13
phib
Junior Member
 
Join Date: Oct 2004
Quote:
Originally Posted by lamer_de
Please have a look at the following 2 programs / projects:

Zidrav: http://sourceforge.net/projects/zidrav
PAR2: http://parchive.sourceforge.net/

They allow byte patching, and par2 even works if there's a (little) offset. So, yeah, such tools already exist ^_^
There's also xdelta 1, available on most *nix platforms and also on windows.
phib is offline   Reply With Quote
Old 2006-01-15, 03:19   Link #14
gumbaloom
Live-eviL Staffer
 
Join Date: Jun 2004
Location: Sailor Moon Musical Fansubbing Land
Send a message via AIM to gumbaloom Send a message via Yahoo to gumbaloom
If this program requires softsubs you will have a limited audience amongst fansubbing groups because most of us release hardsubs not MKV's. Your tool would probably have a greater audience amongst the "DVD Backup" scene than fansubbers.


-gumbaloom
gumbaloom is offline   Reply With Quote
Old 2006-01-15, 05:12   Link #15
Suchy
Junior Member
 
Join Date: Jan 2006
Actually this program require softsubs when You want update remuxed subs (or require other streams if You want remux other streams). If You want update file by binary comparison (upadting classic AVI hardsubs) You can do it also, but only when sizes of source and target file are the same. In this situation remuxing is disabled, and You use zidrav only.

I wrote earlier that I'm going to implement more updates algorithms (PAR, BinaryDiff, XDelta) in future so updating any files (and without including softsub file into a patch) will be possible.
Suchy is offline   Reply With Quote
Old 2006-01-15, 06:47   Link #16
gumbaloom
Live-eviL Staffer
 
Join Date: Jun 2004
Location: Sailor Moon Musical Fansubbing Land
Send a message via AIM to gumbaloom Send a message via Yahoo to gumbaloom
It's very difficult to do encodes of a fansub and get a V1 and a V2 to be EXACTLY the same byte size. That's nigh on impossible to do.

-gumbaloom
gumbaloom is offline   Reply With Quote
Old 2006-01-15, 08:16   Link #17
Suchy
Junior Member
 
Join Date: Jan 2006
Yes, that's why actually it can be used to fix borken ovarheads, etc., wrong download, wrong hasing after downloadu (it happens, when someone have broken RAM).
Suchy is offline   Reply With Quote
Old 2006-01-15, 16:14   Link #18
Ledgem
Love Yourself
 
 
Join Date: Mar 2003
Location: Northeast USA
Age: 38
Quote:
Originally Posted by gumbaloom
It's very difficult to do encodes of a fansub and get a V1 and a V2 to be EXACTLY the same byte size. That's nigh on impossible to do.

-gumbaloom
It's a misconception that Zidrav requires the exact same bytesize to work properly. While you can use Zidrav to patch files that way, it can be used exactly as if it were BitTorrent as well. Please see my post regarding proper operation of Zidrav.

I don't know that Zidrav would be applicable if the file were a whole different encode, though. I doubt it.
__________________
Ledgem is offline   Reply With Quote
Old 2006-01-15, 16:53   Link #19
Suchy
Junior Member
 
Join Date: Jan 2006
Hmm.. it's werid.
I used 3.2.0 version (because, when I start make FHU, thi the newest source of zidrav) and there is only cdt and cdp files.
Later I check 3.5.0 version, and there is still CDT and CDP, moreover (if I good remember) author added filesize checking (when I try do patch for files with diffrent sizes, then I've got an error).

Thanks.
Suchy is offline   Reply With Quote
Old 2006-01-15, 21:15   Link #20
Ledgem
Love Yourself
 
 
Join Date: Mar 2003
Location: Northeast USA
Age: 38
The Zidrav I referenced says that it is version 3.2.0.0 in Windows Explorer, but if you open it, the window header says "ZIDRAV v3.5 - U.G.I. Incarnation with More Things to Play With." As I referenced in my post, most people use Zidrav purely for the .cdt and .cdp file. I'm not sure why people overlooked the proper way of using it, but as I mentioned in my post, the way to really make the program shine is to use it the way I referenced.

The primary difference is in who makes the cheksum file: most people assume that the person with the damaged file has to make the checksum file (that's the .cdt file), and then the person with the completed file makes the patch file (.cdp). This completely cuts out one file-swapping step. The better way to use Zidrav - where it can apply even if the filesizes don't match - is to have the person with the completed file make the checksum file (.cdt), then have the person with the damaged file use the make request feature (generates a .cdq file) and so on. I think my instructions in the post I referenced are clear enough, so I won't replicate it here.

And yes, I did use Zidrav to patch a file that way, which was what led me to make that post in the first place. For some reason a file was corrupt, but the bytesize was different. Using Zidrav in the way I mentioned, my friend generated a 60 MB patch file (full filesize was 130 or so) and the file was then playable and the CRC32 value matched.
__________________
Ledgem is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 03:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
We use Silk.