Automatic Timing

[heartnetkung]

Hi everyone,

I'm a university student in Thailand studying a field related to computer science. I and my friend came up with the idea of computer-generated subtitle timing. We implemented our subtitle-making software, called "Subtitle Mate" with that "auto-timing" feature. The timing process would become: put in your video, wait, and get your timing done! (the screenshot is provided in the link)

Our inspiration on the software came from the fansub. We are able to watch anime from your hard working and determination. We want to help out by simplifying the subtitle making process. Removing the tough timing task.

We also hope our software would relieve language-barrier problem. ANYONE can translate ANY video clips, from youtube or from your video, by himself to share such content, idea, and inspiration across the globe.

Now our preliminary system is operational and got awarded National Software Contest 2010 in Thailand. The software is scheduled to release in late 2011 as free open source software. However, our system needs lots of improvement in accuracy (around 80%**).

Our improvement would require cooperation from many companies and organizations. We need to convince those organizations to provide their resources. So, we conducted our early PR in facebook and forums to survey how many people like this idea and also to prove that our work will be useful. If you like the idea or want more information, please visit our facebook page (and like the page too XD). Your support would help us to convince those organizations. Thus, increase the auto timing's accuracy.

http://www.facebook.com/pages/Subtit...5386018?ref=ts

The screenshots of the program and all features are provided in the facebook.

We are looking forward to hear comments and suggestions from you guys. They can be left here or in the facebook.

Thanks a lot

**The 80% is not that bad. All dialogs are timed very precisely in 0.1 second unit. Most faults are the false catch of sound effect, and op/ed. However, the accuracy test is conducted with our very limited resource.1

[jfs]

Try a couple of episodes of .hack//SIGN and FLCL.

[the.Merines]

Timing is a tough task? To do it well, perhaps...

For some reason, I want to liken this to importing a script into MS Word, fixing any squiggly lines and calling it edited, but that might be too harsh. It's an interesting idea, but I don't think I'd ever personally advocate its usage on any project I'd work on. I'd probably spend more time fixing the autotimed script than it would take to do it myself manually.

[ThaEagle]

Does it take an already translated script or similar, and time that? Or do you have to add in the dialogue by hand later on?

Somehow it feels like just adding dialogue manually could take a long time, but then again, I have no real experience in timing :p

[cat928]

I think it does pre-timing? (I know a small number of translators prefer pre-timing)

Translation could be liberal. When the translated script don't match the audio, the computer can never do a good job. Timing still need a brain.

I think if the interface is good and provide a word processor feel, some translators may use it.

[heartnetkung]

to jfs

thank your for your interest in Subtitle Mate.

I have tried using the auto timing with some episodes of .hack//SIGN as you have requested.
However, the outcome was not very satisfying. This problem results from the prototype AI that we created.

Our algorithm is based on machine learning AI. That is, you teach the AI which is human sound and which is not. Do it for million times. Then we use that AI to classify our input. The first problem is we don't have that million sound to teach. The second is that we need ultra pure human voice and ultra pure sound effect. These sound can only be found in movie and music production company.

The second problem is due to human error. The samples that teach AI about human sound is created by my friends and i. We acutally do the timing for the audio files to try to mark down which part of the audio contain human sounds. This method can lead to errors since it is very hard to distinguish the actual start and end of the voice. As a consequence, ultra pure sounds are needed. The current human voice detection is just a prototype which is not fully worked yet. However, it is proved to work with news report and school-life anime.

Currently, we are in the process of contacting movie production company to ask them for some ultra pure samples. Once we got our hand on those samples, we will be able to create a much better human voice detection AI and the auto timing feature will be complete. Please look forward to our final product for we believe it will benefit all timer.


to thaeagle and cat928

Our design instruct the user do the auto-timing first. Then, do the translation within our program. We provide a feature to play that particular subtitle (jumping to the start time, play and then pause at the stop time). Or do u suggest otherwise? At this early stage of the development, we can implement a wish-list feature or change our design easily. Your suggestion would help us a lot.

PS. please help support us by liking our facebook http://bit.ly/9VE5H2 ^^
PS.2 the tested video will be prepared and uploaded with our generated subtitle asap. please wait a little longer~*
PS.3 sorry for the late respond.

[Quarkboy]

The problem has never been, to me, about feasibility.

The only use I see for this is in 100% automated systems, like an auto closed caption concept.

For actual translations, the amount of time it takes to time and translate an episode is only ~20% more than to just translate it, at least for an experienced worker.

The auto-timer would never be 100% flawless, and would never be able to break lines at ends of sentences and clauses intelligently because that would require a full voice recognition engine/grammar parser.
The time it would take to check over the auto-times for errors would be about the same as it would doing it by hand, in my opinion.

In other words: This system would only save time when quality is secondary to speed. So for real time captioning it could be great, but not for actual subtitling.

[jfs]

Originally Posted by Quarkboy

The time it would take to check over the auto-times for errors would be about the same as it would doing it by hand, in my opinion.

This.
My experience with timing is that if it isn't perfect the first time around, it's no more work to re-do it entirely from scratch than it is to go over it and fix the problems.


And yes I deliberately pointed out .hack//SIGN and FLCL as two examples of anime that has a soundtrack with a lot of vocal in it, something I believe an automated system won't easily be able to tell apart from the actual speech; at least not without some voice recognition and sentence parsing AI as well, as Quarkboy also points out.

What a system like this might be useful for, apart from real-time captioning, could be to assist in checking whether everything relevant has been translated. I.e. detect the ranges with potential dialogue in and compare their overlap with the actual timed dialogue.
Another thing that could be interesting would be assisted karaoke timing. Since the Japanese language is mostly regular in its phonology, it should be possible to match written lyrics to the music/voice with some decent precision.

[Random32]

I'm not that sure, but isn't speedsubbing a time where quality is second to speed?
DVD rips will probably still be manually timed, but shaving and hour or two off a speedsub is a good thing even if there's a loss of quality.

[TheFluff]

Originally Posted by Random32

I'm not that sure, but isn't speedsubbing a time where quality is second to speed?
DVD rips will probably still be manually timed, but shaving and hour or two off a speedsub is a good thing even if there's a loss of quality.

any decent speedsubber will have a decent timer, and a decent timer can time an entire episode in less than half an hour so you're not really saving any time at all

[cat928]

Originally Posted by heartnetkung

to thaeagle and cat928

Our design instruct the user do the auto-timing first. Then, do the translation within our program. We provide a feature to play that particular subtitle (jumping to the start time, play and then pause at the stop time). Or do u suggest otherwise? At this early stage of the development, we can implement a wish-list feature or change our design easily. Your suggestion would help us a lot.

When you mention auto timing, people think of something else. Isn't it better to rename it to something like automatic pre-translation timing?

The concept is interesting. It's probably helpful in some shows. And maybe you can eventually make another youtube transcribe audio subtitle.

If your interface is like aegisub's translation assistant, I am quite sure that your project would fail. Since no competitive translator do line by line translation. The demand is probably on a real time colaboration A.S.S. editor, which has audio(and video?) capacity and notepad feel. You may look for that direction instead. Maybe it's easier to create a fork from aegisub?

[cyberbeing]

Around 6 years ago, silverfire coded a mIRC script which did semi-automated timing. Anybody still have that script? It would be interesting to see a comparison when heartnetkung's program is released.

[Free]

Bravo to heartnetkung and your team! I hope we'll see an alpha/beta release in the near future.