In ASS, the X axis is horizontal (left to right), the Y axis is vertical (top to bottom) and the Z axis is perpendicular to the image - that is, imagine it as "sticking out" of the screen and "going into" it.
Normally, the origin of the rotation depends on the alignment of the line, in the same way as \pos. If we think of the line of a box, \an5 will place the center of the rotation in the middle of the box, and thus a rotation around the Z axis by 90 degrees will make the "box" vertical instead of horizontal. Observe:
(The other alignments place the point of rotation in different corners of the box, yielding different results.)
Similiarly, \frx and \fry with \an5 will make the text rotate around the other axises, like so:
Now, things get interesting. For a 640x480 image, the \org of an \an5 line obviously defaults to 320,240, but the \org override allows us to modify this. If we move the origin of the rotation, the line will be "positioned" in the same place as before, but now rotate around ANOTHER point than its center. With \frz, this produces the effect that it rotates in a circle (with the radius defined by where the \pos coordinate is) on the image, around the invisible point specified by \org. If you place the \org really, really far away (outside the image), you get something like in Sylf's picture, where a small \frz moves the line a considerable distance. For vertical movement, place the \org outside the image's left or right borders, and for horizontal movement, place it outside its top or bottom borders. If you place it sufficiently far away, the small "tilting" of the line won't be noticeable.
As for \frx and \fry... This is where stuff starts to get difficult, because now we have to think in 3D, despite the image only having two dimensions.
Moving the \org of a \frx rotation will (similiar to \frz) cause the line to rotate around the X axis, but with the center of rotation somewhere else than the line's center. If you place the \org's Y coordinate on the same Y coordinate as the original line but move the X coordinate in either direction, this will merely have the effect that the line rotates around itself, but kind of "slanted" in the direction you moved the X coordinate. The interesting things start to happen when you move the Y coordinate as well. Then, the line will rotate as if its top or bottom border was glued to an invisible cylinder lying horizontally across the image, with its radius decided by how far from the orginal position the \org's Y coordinate is.
\frx works in a very similiar fashion, only around Y axis.
\frx and \fry may both also be used for movements by abusing \org; figuring out in what direction you should move the \org coordinates is left as an exercise to the reader.
EDIT: ah, Sylf was faster, and had a simpler explanation to boot. :/