OverLua, new tool for masochistic karaokers and TS'ers

[jfs]

For the last couple of days I've worked on a new tool primarily intended to make karaoke production harder and more painful, but also able to produce much more advanced (ie. over-the-top, ugly etc.) effects and I think it's reached a state where it's ready for public consumption now.

I've already announced it and posted several versions on the Aegisub forum.

The tool's name is OverLua and it's function can best be described as Lua scripting to paint on video frames. It works as an Avisynth filter that calls a Lua script for every video frame requested.

I shouldn't need to say this, but I'll say it anyway: This tool is very hard to use correctly and requires a lot of programming experience. Also it's still very alpha-quality-like, buggy and incomplete, yet still useful enough to produce actual effects.

So, some links:
Thread on Aegisub forum - this is where I'll be posting new versions, don expect me to update this thread for every posted version.
Version 0.7 alpha download - you might also need some runtime libraries from Microsoft to use it.


Sorry, I don't have any interesting screenshots of OverLua in action yet. Update: Now I do have a somewhat interesting screenshot, see below. But to get you started, here's some very hacky code to read an ASS file with karaoke timing:

function parsenum(str)
	return tonumber(str) or 0
end
function parse_ass_time(ass)
	local h, m, s, cs = ass:match("(%d+):(%d+):(%d+)%.(%d+)")
	return parsenum(cs)/100 + parsenum(s) + parsenum(m)*60 + parsenum(h)*3600
end

function parse_k_timing(text)
	local syls = {}
	local cleantext = ""
	local i = 1
	for timing, syltext in text:gmatch("{\\k(%d+)}([^{]*)") do
		local syl = {dur = parsenum(timing)/100, text = syltext, i = i}
		table.insert(syls, syl)
		cleantext = cleantext .. syltext
		i = i + 1
	end
	return syls, cleantext
end

function read_input_file(name)
	lines = {}
	for line in io.lines(name) do
		local start_time, end_time, style, fx, text = line:match("Dialogue: 0,(.-),(.-),(.-),,0000,0000,0000,(.-),(.*)")
		if text then
			local ls = {}
			ls.start_time = parse_ass_time(start_time)
			ls.end_time = parse_ass_time(end_time)
			ls.style = style
			ls.fx = fx
			ls.rawtext = text
			ls.kara, ls.cleantext = parse_k_timing(text)
			table.insert(lines, ls)
		end
	end
end

If you wonder, you'll have to hard-code the ASS file name in the Lua script, at least for now.

Spoiler for Screenshot:

So here's a screenshot of the graphics produced by the 'test4.lua' script:

Of course you can do much more than this shows, it's just a little taster
What's shown here: Gaussian blur; the outer border of the text is strongly blurred. Actually it's two offset borders slightly overlapping and blurred afterwards. A copy of the original video frame is taken, blurred and used as fill for the text. Finally a dark red and very transparent red fill is laid over the text, darkening and reddening it.

If you want to test this out, please do so. I'm also very open to suggestions for improvements and new features. (The C++ source is also included so you can hack at it if you want. It's GPL 2.)

Also, I will only answer questions if they show that the person asking clear knows what (s)he is doing and the question is well asked. I'm not going to teach anyone programming. (If the above sample code looks completely alien to you you're probably not in the target user group for OverLua.)

Another small sample (adapted from the included test2.lua file) to round off. This code simply draws a variably blurred triangle on the video.

function render_frame(f, t)
	local surf = cairo.create_image_surface(f.width, f.height, "argb32")
	local ctx = surf.create_context()
	
	ctx.set_source_rgba(1, 0.5, 0.5, 0.75)
	ctx.move_to(10, 10)
	ctx.line_to(10, 200)
	ctx.line_to(300, 10)
	ctx.close_path()
	ctx.fill_preserve()
	ctx.set_source_rgba(0,0,0,1)
	ctx.stroke()

	raster.gaussian_blur(surf, (1-math.cos(t*10))*2)
	
	f.overlay_cairo_surface(surf, 0, 0)
end

Quick edit: Oh yeah, this tool is not related to Aegisub or Automation at all, it completely independent.

[Quarkboy]

Hmm, now THIS is something that could theoretically turn into a legal substitute for After Effects. However, without a GUI interface it's useless for anything other than extreme masochists.

But it seems like one could take this as a backbone and write a GUI interface without too much trouble. If you add in some motion tracking code and iteration... Well, it would be powerful enough to do a lot of things that After Effects could do.

[jfs]

It can't request frames for processing and I don't think OverLua itself would be the most suitable platform to implement motion tracking through. What you could do, however, was have an external tool perform the motion tracking and store the result to a file which an OverLua script could then read in.

By the way, I've checked the source into the Aegisub SVN repository now if anyone feels like following development from there Also 0.6 should soon be ready.
I introduced a bad bug in 0.5 actually, where the RGB24-surface-onto-video-frame composition fails completely. (And the ARGB32 composition might also be broken, but more subtly.)


Edit: Posted version 0.6 now. It fixes a bunch of bugs and introduces two new features: VFR support (in the Avisynth filter - translate frame numbers to timecodes using a Matroska timecodes file) and a parameter that passes a freeform string (eg. path to an ASS file) to the Lua script. See the Aegisub forum for details.

[MexFX]

I hope this well asked questios :P

I'm trying to draw the text via pixel dots :

function render_frame(f, t)
local text = overlua_datastring or "Bingo !"
local w, h = f.width, f.height
           surface = cairo.image_surface_create(w, h, "argb32")
contex = surface.create_context()
draw_text(contex,30,240,50,text,1,0,0,1)
 
for i = 0 , w do
for j = 0,h do
--[[ I really don't know why I'm doing this >< ....
if I didn't test every pixel , the script never run , at the same time , it's no diffrent 
if I test "r" , "g" or "blue" .....they all give me what I want ....
well , Idon't care , since it work fine ]] 
 
local  r, g, b = surface.get_pixel(i, j)
if r == 255  then --the original text is pure red
 
 
draw_text(contex,7,i,j+30,".",1,1,0,1)
 
 
end
 
end
end
 
f.overlay_cairo_surface(surface,0,0)
 
end
 
function draw_text(ctx,font_size,xPos,yPos,text_to_draw,red,green,blue,alpha)
ctx.select_font_face("Disney Comic", "", "bold")
ctx.set_source_rgba(red, green, blue , alpha)
ctx.set_font_size(font_size)
ctx.move_to(xPos, yPos)
ctx.text_path(text_to_draw)
ctx.fill()
end

it works fine and draw the text via pixel dots , but there's a problem

-I want to know how to get rid of the original text & keep the the "pixel dots" one ...

I tried to give it zero alpha , but the both disappear >_< ..EDIT : also make two surface for each one then kill the original won't work(actually it's stupid:P)

-is there any better way to draw text via pixel dots

- any hints of how to make something :

Spoiler for this:

[jfs]

You will want to use two surfaces for it, one temporary for the source and then the destination.

Drawing pixel dots is probably better done with eg. ctx.rectangle(x-0.5, y-0.5, x+0.5, y+0.5). The 0.5 things are because of the pixel grid model cairo uses. Whole coordinates mark the center of a pixel and so the "grid between the pixels" is at +/- 0.5.

Remember that cairo ARGB32 surfaces use premultiplied alpha. This means that if you have a pixel that is 100% red but 75% alpha it will get stored as 75% red, and in turn that 0% alpha will mean the pixel gets stored as black no matter what.


I haven't actually tried to make that effect with "text split into dots that spread apart" but I have a general idea of how to do it. First, you of course need a function to model the spreading, depending on how you want it to look. The gif you posted seems to use a kind of "drops in water" spreading model, which will probably need to be pre-calculated in some way.
Depending on how large the dots should be, instead of pixel a colour from the source surface (which has the simple rendered text) and then drawing on the target, it might be a better idea to use the source surface as a pattern for the drawing on the target, and displacing it as required.


Maybe I should push a new binary out soon... I've added a few (just two, actually) functions that might be interesting but I'm not entirely sure yet.

[MexFX]

hmmm...
so that effect seems quite difficult

OK, I think I have to try what you said ...
thanks for fast reply ...I'm looking forward the new function

[MexFX]

loOoOoOl
it was stupid ... trying to achive that effect is really painful
I think I have to wait to another time to do something like that .


BTW : did you mean ctx.rectangle(x-0.5,y-0.5 , 1.5 , 1.5 ) ?
using the same x,y value for both 1th , 3th ,2th & 4th just draw huge rectangle .

jana

[jfs]

Hah, actually ctx.rectangle(x-0.5, y-0.5, 1, 1), forgot you specify (left, top, width, height) and not (left, top, right, bottom).

[triviper]

i was playing around with your application and had a question...

how do i apply the effect created to a .ass, i am still a little uncertain, can you modify test4.lua and show/send me how it would look? I just get the filepath showing in the video so i assumed it had to be a problem with the lua file obviously.

[MasterID]

Take a look at the samples : here.
There are functions to parse ass files.

[triviper]

thanks, i will look into it


EDIT: i am trying to apply test4.lua effect in sample1.lua but without much success, i either get the subs with no effect or i get the data="" from avs with effect

[TheFluff]

Originally Posted by triviper

how do i apply the effect created to a .ass, i am still a little uncertain, can you modify test4.lua and show/send me how it would look? I just get the filepath showing in the video so i assumed it had to be a problem with the lua file obviously.

OverLua doesn't apply anything to .ass files. It's a method to use Lua scripting to draw things on video; currently it does the drawing through Avisynth since it's only implementation is as an Avisynth plugin.

Coincidentally, you can write a script that parses an .ass file and draws the text (with or without fancy effects) on the video, but the .ass file has nothing to do with the drawing itself. If you wanted to you could just as well put the text you want to draw in the Lua script itself in some sort of data structure but that would be kind of dumb.

[SilentTweak]

Would OverLua carry any of the negative effects of VSFilter, like crazy dots, artifacts or crashing, if the Lua script outputs something too crazy? Since it's different from how VSFilter draws things out, I wouldn't think so, but are there any?

[TheFluff]

Originally Posted by SilentTweak

Would OverLua carry any of the negative effects of VSFilter, like crazy dots, artifacts or crashing, if the Lua script outputs something too crazy? Since it's different from how VSFilter draws things out, I wouldn't think so, but are there any?

It does not have any known rendering bugs, no. Note that Overlua itself only does some specific raster operations; all the vector drawing is done with the rather well established and well tested open source library Cairo.

[jfs]

To a very large extent, the quality of graphics rendered using OverLua depends entirely on the effects programmer, ie. you.
It's quite possible to write some code that renders totally distorted images or just starts some infinite allocation (eventually crashing the process.)

Think of OverLua as a gun turret that tracks your feet until you understand the mechanism to install a new tracking program, to use a classic (and already overused) analogy.

I agree this may sound weird, but OverLua is extremely hard to use correctly, until you fully understand the logic in it, then it becomes very straightforward to use.
Myself, I find it blindingly obvious why things should be done the way I designed OverLua to be used, but at the same time I can (mostly) understand why many people won't easily understand the concepts.

[ReAn]

You see, Im a TSer, Im a massochist, and i've got a very high level of programming experience, however......................

I'm also lazy i'll bookmark this for later investigation.

[tin2tin]

What you're doing here is very interesting from an Avisynth perspective too.

Drawing of vectors over video is very nice. Is it possible to load images/video as is now? I've tried, but so far unsuccesfully. It would be very useful for ex. coding an antialiased pan, zoom and rotate script(Ken Burns) for Avisynth.

Is is still beeing developed? CheesyFX - is that a joke?

Anyway if you want testers for functions outside the subtitle functions - you might want to bang the drum for this fine project over at the Avisynth'er hangout: forum.doom9.org.

[jfs]

Nah, nothing new happened here for a long time. I don't expect anything to happen soon. I still want to do CheesyFX, but differently. (As a subtitle renderer that uses Lua scripts as the subtitle file format. Based on Kumaji. When I get around to do that.)

[tin2tin]

Is it possible to load images/video as is now?

Do you plan make CheesyFX work in Avisynth too?

[jfs]

Technically, if you implement the image reading completely in Lua, it's possible, but not something you'd want to do. There's no library functions to do it.
Of course any video filter I make would be get an Avisynth interface, it's just about the most convenient way to both use and test things.