RAID for long-term storage of video

[Ledgem]

At work, our current project is to important video from tape sources and then convert them to DVD. Rather than trashing the raw imported video, we're saving them. When the RAID gets full, we'll disconnect it, store it, and set up a new one to continue operations.

We're currently using a striped RAID (Mac OS gave the option to set this up as a "concatenated disk set" but it still reports that it's a striped RAID regardless), and it's almost full. My friend used to manage most of this stuff, but he graduated, leaving me in charge of operations. His advice was to mirror the current RAID for further redundancy, and then set up a new one to continue progress. The director will determine whether the budget will allow for the mirroring RAID, but I was wondering if I should continue to create RAIDs the way we've done thus far.

Mac OS doesn't give a whole lot of options for the RAID; you can basically choose to stripe (RAID 3 or 4 I'd hope; maybe RAID 0) or mirror (RAID 1). We're going with striping either way, but I do have a choice with the filesystem. We've been using mac OS Extended+Journaling (HFS+Journaling), the Mac OS default, but other options include MS-DOS (? FAT32 perhaps? If so that's not an option) and UNIX filesystem (not sure which one). While I'd imagine that none of these would offer a performance boost to the Mac OS default, but would any of these be a better choice in terms of being able to recover data should something go wrong with the RAID (hardware failure)? Since we're all using Macs as work systems, locking into the Mac OS partition format probably isn't too big a deal, but it does make me slightly nervous... I guess I'd prefer something that can be accessed by more systems more easily. Any thoughts?

[grey_moon]

How long is long term? When you say disconnect, I am guessing you mean you will remove the disks and store them?

There are 2 issues I would raise with this method of archiving.

First is will the disks be supported in the future? Hardware gets updated at a fast rate and nothing worse then a few years down the line you pull out the archives and find the disks don't fit any of your kit.

The second which I actually asked a SAN seller and he never got back to me on, is I have heard that the method of storing data on magnetic disks does fade over time. If this is the case, what is the data integrity lifetime of your disks and is there any process to refresh it? (not that copying several terrabytes of data makes me jump with joy).

Some questions in general are:

How much value is the data worth? If it is valuable and there is a need for a good long term archive then I would say the good old "SAN" backup -> "cheap SAN" backup -> "tape" method. But this is the most costly solution.

I would not advocate backing up to dvd media at all. I haven't seen the life time spec's of blueray or hddvd yet, but I remember how much they promised when cds came out (indestructible my arse).

In terms of RAID, that should be used more for redundancy during operation more then as a backup solution. Just an example, if the raid controller goes it can trash all of your data across all the drives.

[TakutoKun]

As grey_moon mentioned, what is the value of your data? With that being said, you will have to establish a fault-tolerance/redundancy for your data. I will highly recommend that you steer clear of RAID 0 as if a single drive goes, you lose everything. The same goes for RAID 3 and 4 - if the parity drive goes, you will lose everything. I would be aiming at RAID 1, 5, or 0+1. In terms of redundancy, you should look at 0+1 as it uses two RAID controllers and will mirror your data as well as stripe it. If you use RAID 1, I think you will suffer a major performance loss and a disk space loss of 50%. It is relatively the same for RAID 0+1, but you will still achieve performance through striping. In regards to efficiency and speed, RAID 5 will allow you to distribute parity among at least 3-drives losing 1/3 of the space.

[jpwong]

I'm no expert on this, but on Raid 3, you'd still need 2 Hard disk failures to actually loose all your data (same as Raid 5) since 3 and 5 are essentially the same thing, just they distrobute parity bytes differently.

I suppose the only danger in using a Raid with dedicated parity disk would be if the parity drive fails and you don't notice and a second drive fails afterwards. That's the situation where you would loose all your data.

In order of preference if you only plan on using 3-5 disks in your raid would be Raid 5, Raid 3, Raid 10 (0+1), lastly Raid 1.

matradley has otherwise covered all the benefits and problems with the various raid modes.

[Syaoran]

After losing a lot of data in a sequence of hard disk failures I went for the hardware RAID 5 solution. 4x250GB SATA.
In my case it's not video that's stored, but mostly source code and huge database files.

Dunno what you mean by valuable video, but if it's TV station like, don't they've special storage devices for it? I remember seeing a few solutions on CeBIT a few years ago. Impressive.

For video that's valuable to me (not anime), I have 2 copies on DVDs and one on an external hard disk.

A CD/DVD should keep data for about 10 years, stored in perfect conditions. Reality is different... I once got a batch of discs that changed colour (brownish) and became unreadable. And those were never exposed to sunlight or a heat source. All boxed in a dark closet.
On the other hand I've also discs that are more than 5 years old and often used and still perform as well as if they were burned yesterday.

CD/DVD with important stuff gets duplicated once in a while avoid losing the content.

[Ledgem]

These are all external drives, chained via Firewire (I've only seen Macs do this, but I think it's part of the firewire spec anyway). I don't really know how important or for how long they want to keep them, but that they're all in their own enclosures makes storage somewhat easier (as compared to having a naked drive). The original project was to simply make DVDs, but perhaps the director realized that HDs were cheaper than he thought, and he also wants us to have the source video around so that later on HD-DVDs/Blu-Ray discs can be made?

We'll probably mirror the "big" RAIDs (at 1.25 tb, it was big when we made it two years ago, but I guess that seems modest now), but HD lifespan is a good point. Tapes are still being used for data? I remember having an Iomega Ditto, it was such a pain to use. What's the upper limit of tapes these days? Last time I checked they were only at 300 gb, which is nothing. It can still be used, but that's going to be painful

Also, how do you set what RAID type it is? Using Mac OS' internal tools only gives me those two (three) options. Are there special tools that you need, or is this standard on Windows/Linux and Mac OS is just dumbing down the options?

[grey_moon]

Oh wow I imagined that totally wrong, I thought the disks were in a raid device.

So the mac os raid works over firewire?

So from what i read you can have x number of external hdds which are all raided to make a virtual storage pool. I have no experience at all with this type of setup, but it does sound a little fragile to me (imagines a cable knocked out of one of the disks).

Did your friend mean mirror the current data onto another storage set?

"His advice was to mirror the current RAID for further redundancy, and then set up a new one to continue progress. "

I guess he did, as I don't think it is possible to change the basic nature of a raid without rebuilding it (could be wrong here).

So I guess he means mirror the data as in back it up to another storage device, which is a sensible move. Now with redundancy you have a safety net whilst making any changes.

Sorry to say but my experience is rather lacking in this scenario

Tapes are still being used for archiving, but they normally are the second tier in a backup solution. The models I was looking at were expensive fast san, which is nearline (shadow?) backup to cheap san and then slowly to tape.

The lifespan I was on about isn't the life of the media (hdd platters last for yonks), but the life of the data on the media. I remember reading about it an age ago and when I raised it with a sales person who was trying to offload a hdd based archiving solution he didn't come back with an answer.

[TakutoKun]

Originally Posted by jpwong

I'm no expert on this, but on Raid 3, you'd still need 2 Hard disk failures to actually loose all your data (same as Raid 5) since 3 and 5 are essentially the same thing, just they distrobute parity bytes differently.

I suppose the only danger in using a Raid with dedicated parity disk would be if the parity drive fails and you don't notice and a second drive fails afterwards. That's the situation where you would loose all your data.

In order of preference if you only plan on using 3-5 disks in your raid would be Raid 5, Raid 3, Raid 10 (0+1), lastly Raid 1.

matradley has otherwise covered all the benefits and problems with the various raid modes.

RAID 3 has a dedicated parity disk. RAID 5 has distributed parity - if one goes down, replace it and the other two will rebuild. However, with RAID 5, if two goes, hold on to your socks!

Originally Posted by Ledgem

Also, how do you set what RAID type it is? Using Mac OS' internal tools only gives me those two (three) options. Are there special tools that you need, or is this standard on Windows/Linux and Mac OS is just dumbing down the options?

Windows does some firewire chaining as well. One of my chains for some reason shows up as a SCSI0 in Linux! At any rate, which options does Mac OS give you? I am impressed with Mac's ability to offer such benefits.

[jpwong]

Originally Posted by Ledgem

Also, how do you set what RAID type it is? Using Mac OS' internal tools only gives me those two (three) options. Are there special tools that you need, or is this standard on Windows/Linux and Mac OS is just dumbing down the options?

You may only have those options, software based Raid really isn't the most ideal solution.

[grey_moon]

If your director is willing to invest, have you not considered a das/nas?

[Ledgem]

matradley: The options are to make a mirrored, striped, or concatenated RAID. I can also set the volume format (which I interpret to be the partition type), and advanced options allows me to set the RAID block size (defaults at 32k) as well as turn RAID Mirror Autorebuild on (it's disabled by default).

jpwong: The Mac OS RAID options aren't creating a software-based RAID, are they? Based on what I've read about RAIDs it is a pretty terrible way to go.

grey_moon: What does das/nas stand for? I've never heard of it before. I'll have to check with him about what he has in mind for this, how precious the data is to him/the institution, and how much he wants to spend. If I can present him with a few options, it'd be helpful.

[problemedchild]

Direct Attached Storage
Network Attached Storage

[jpwong]

Originally Posted by Ledgem

The Mac OS RAID options aren't creating a software-based RAID, are they? Based on what I've read about RAIDs it is a pretty terrible way to go.

Hehe, it's just how I read the post since it says "Mac OS' internal tools", which I take to mean something on the OS itself rather that a BIOS screen, but I suppose they could mean that. I have no idea what Macs do for RAID, but I have to setup my BIOS menu, not using a Windows tool.

[SeijiSensei]

Here's some NAS options from CDW sorted by price (descending). You'll see right away that they can get pricey (tens of thousands of dollars). On the other hand if your business depends on storing these files (as it sounds from your description) it might be worth it over the long haul.

As an alternative I'd build a Linux box using either a hardware RAID controller or the software RAID native to Linux. (I usually just use software RAID). I'd guess, being Unix systems, that Macs can connect using NFS? Linux has software to export files with most major protocols, Windows CIFS being the most common.

In your case I'd format the array with XFS (also now native to Linux), a filesystem developed by Silicon Graphics specifically to handle large files like video.

My current server arrangements typically have a RAID array with LVM ("logical volume management") overlaid on top. This enables you to avoid static partition sizes and create virtual partitions that can be resized. You can also take "snapshots" of the LVM volumes which is a nice way to do backups. This HOWTO has more information on LVM and snapshots.

[lamer_de]

Raid-based solutions are probably a good way to get some backup functionality for storage you work with, but that's not exactly a real backup strategy. There should be at least another copy somewhere, usually updated each day/week/month depening on how much important data you create (and often that backup is also doubled and stored in 2 different locations).

The NAS solutions are probably meant for usage in a datacenter, as to reduce the risk of physical damage (fire/water/terrorist attack/whatever), e.g. store the data in 2 different physical locations. The linux box definitely sounds more sane for small to mid-size enterprises, I guess (and would allow you to get rid of the macos filesystem). I'd say you really need to make a plan or list on what exactly is needed and how important the data is and for how long it needs to be stored (there might be government regulations on how long to store documents for example), else it's just guesswork.

Maybe it's enough to store the original tapes somewhere safe, if they're already digital (e.g. DV, digibeta) they just need to be read out again, which I guess should be pretty cost effective.

CU,
lamer_de

[Ledgem]

Originally Posted by lamer_de

Maybe it's enough to store the original tapes somewhere safe, if they're already digital (e.g. DV, digibeta) they just need to be read out again, which I guess should be pretty cost effective.

The originals are U-Matics, Betas, and VHS format. We had a pretty nasty time acquiring a UMatic player that could output color and play the tapes without too much noise and other junk. Aside from the fact that I'd imagine that most UMatic players that exist are on their last legs, I'd also guess that for the UMatics that are in poorer shape, they probably can't take more than 10 more plays and rewinds. The Betas and VHS are perfectly fine, of course, but the digitalization effort is partially to get the video and audio off while the tapes are in good condition. It's a moot point either way, because the director wants to toss the tapes once we're done getting importing them.

I checked into the NAS/DAS and while they look neat, it feels like they'd serve a different purpose. For us, we just import the video and then create a DVD using DVD Studio Pro and a rather finnicky DVD printer. These videos are all old language videos that were used for university classes as a resource, and I guess they want to make those videos more easily available. I'd never even heard of a UMatic before I took this job, and Betas were something from my early childhood, so I'm guessing that the library wasn't seeing much use as of the past decade or so The idea to retain the imported digital video was more of an afterthought. I'd imagine that it's keeping with the idea that perhaps in the future they'd want to shift formats to Blu-Ray or HD-DVD, and it'd be nicer to do that with the uncompressed version rather than to rip it off of the DVDs that we made (not that there's any wonderful quality that people would be missing out on - these are all off of tape sources, after all).

In other words, nobody else needs to access these videos on a regular basis. I'd imagine that we'd be storing these drives and perhaps 20 years down the line, if people even remember what's on the drives, they'd dig them out, plug them in, and use them to create discs on the new formats. Possibly it could even be used to just make more DVDs, should the ones that we're making now break/get lost/experience increased demand. In my mind, the DAS/NAS would be good for something like a 3D Rendering company working a render farm, where people need access to large files and need to be able to store large files on a day to day basis. Our video sources are largely one-time only deals, with no need for access to the video source once the DVD is made. Does that change the recommendations any?

[jpwong]

I have no idea what the associated costs of this would be, but have you looked at Tape Backup drives? In terms of physical storage space and labelability, this would probably be better long term inactive storage than trying to keep rows of RAIDed hard disks from getting mixed or dropped.

[Mentar]

Originally Posted by Ledgem

In other words, nobody else needs to access these videos on a regular basis. I'd imagine that we'd be storing these drives and perhaps 20 years down the line, if people even remember what's on the drives, they'd dig them out, plug them in, and use them to create discs on the new formats. Possibly it could even be used to just make more DVDs, should the ones that we're making now break/get lost/experience increased demand. In my mind, the DAS/NAS would be good for something like a 3D Rendering company working a render farm, where people need access to large files and need to be able to store large files on a day to day basis. Our video sources are largely one-time only deals, with no need for access to the video source once the DVD is made. Does that change the recommendations any?

Careful. Keep in mind that the maximum longevity of harddisks is 5-7 years. After 4-5 years, you need to do a migration to either new disks or a different format, or - otherwise - risk losing the content. You'll plug the old disks in, but they won't work anymore or show read errors.

If the task is really to keep the video safe for a longer time, your best bet are still magnetic tapes, who are tagged at 10+ years. Especially when it's the "store away and forget" kind of data.

[grey_moon]

Originally Posted by Mentar

Careful. Keep in mind that the maximum longevity of harddisks is 5-7 years. After 4-5 years, you need to do a migration to either new disks or a different format, or - otherwise - risk losing the content. You'll plug the old disks in, but they won't work anymore or show read errors.

Ah they are the numbers I have always been looking for, but do you have any sources please?

[Ledgem]

If you could cite a source for those HD lifetime numbers I'd appreciate it, as well. I've only heard numbers about hard drive lifetime for a powered-on drive; I've never heard of how long a drive that is off and stored away will last for.

I did some looking into tape drive solutions, and it seems like it may be viable. I was surprised to find that for $3,600 you can get a 4.16 TB (after compression) capacty drive; for $600 more you can bump it up to 8.32 TB (after compression; 4.2 TB uncompressed space). There are other prices and sizes but I think that those may be the best overall value. My thinking is that, rather than buying a ton of new HDs to mirror the RAID and then make a new RAID as well, it may be better to back up the current RAID to tape, wipe the RAID, and continue to use it for video work, backing it up when it becomes full.

I do have a question about these tape drives, though. I've never heard of the media types used, and on top of that, I don't see any drive bays. So these tape drives are more like hard drives rather than CD drives, in that the tape is all internal and can never be ejected? I've never seen a tape drive like that, but I guess everything is different in the corporate world. If anyone can clarify that, I'd appreciate it.

I'll email my director and ask him what his long-term plans are, and let him know about the tape solution. I'd guess that hard drives for a RAID may be cheaper initially, but if tape lasts longer, it'd probably be the better option for our needs.