There are only two kinds of hard drives in the world. Those that fail and those that haven’t failed yet. This mantra has been imprinted in anyone’s head who has had irreplaceable data on a drive that fails and you lose that data forever.
The Business Need
Being a filmmaker, I create tons and tons of content. With most of my projects I will capture 2TB-4TB of data for each job. I use a RAID array with redundancy built in for my editing drive, but once I’ve completed the project I then need to archive it in a way that keeps it safe but also allows me to get to that data. Sometimes clients will call you up a year or two later and need a re-edit of some footage of something.
Ideally I could backup and archive everything to online storage. Costs have come down a lot for online storage, but it’s still not super cheap. Even if it was, upload speeds in Australia are so slow that trying to upload 20TB per month is just not practical.
The other reason I can’t use online backup is that I do a fair bit of corporate video work and so am sometimes working with footage that is under NDA, so I can’t upload that video to an offsite server.
Up until now, this has been my archive “solution”: I would buy 3 x 3TB hard drives; I would then use Hedge to create three identical copies of my data on those 3TB drives. Then one drive would go in a climate-controlled draw in my office, one would be stored in my garage and one would be stored in a fireproof safe at my in-laws house.
This “solution” has a lot of pitfalls. To begin with, those hard drives are not cheap when you’re buying three at a time and doing that around once a month if not more. I should be taking each drive out of storage every six months and so to spin it up to ensure they don’t degrade, I say “I should” as I don’t do this nearly as often as I should.
The final pitfall is that it’s a pain to make three copies of everything every few weeks.
The Solution
I’ve had my eye out for better solutions for some time. I had often thought of looking at a LTO tape solution, but always assumed it was too expensive and too complicated, so I just didn’t look at it. This changed a few months ago when I was watching a session about backup during an online conference. They mentioned a few points about LTO tapes which encouraged me to look at them closer.
The first thing I found was that the LTO drives are not cheap. If you are looking at an internal solution you’re looking at about $2K AUD or more. I am a Mac user and so I didn’t have the option of an internal solution and needed a stand-alone drive. This means I was looking at anywhere from $4K – $7K AUD. Which sounds silly to lay out that much on a drive, but you see that’s that thing: with the LTO solution you will spend a boatload on the drive, but then the tapes are going to save you money in the long run.
With LTO 8 you can save 12TB uncompressed and 30TB compressed on a single tape, and a single tape costs about $108 AUD. Compare this to if I was getting 12TB of hard drives that would cost me $125 per 3TB drive so that’s $500 for the 4 x 3TB drives, but I then have to multiply that by 3 for 3 copies. So about $324 for 3 x 12TB with tapes or $1500 for the 3 x 12TB of hard drives.
Different generations of tape drives will copy different amounts. So for example LTO 7 holds 6TB uncompressed and 15TB compressed. LTO 9 will hold 18TB uncompressed and 45TB compressed. But LTO 9 tapes cost more than LTO 8 and LTO 7 cost less than LTO 8. Also check if the drive you buy will be forward/backward compatible as not all of them are.
I went with the Fujifilm LTO Ultrium 8 data cartridge. The drive that I bought, the OWC Mercury Pro, came with one and I got another 10, which should at least sort me out for moving all the stuff I have on hard drives onto tapes. Or at least get me started.
There is also how long drives last and how long tapes last. Hard drives last anywhere from 3 to 5 years. LTO tapes last about 30 years. Now of course that 30 years is the best case scenario, but even if they only last 20 years that’s pretty darn impressive. You do need to take good care of the tapes, you can’t just leave them on shelf. Ideally, you want to keep them somewhere that is climate controlled. I am going to keep them in an Esky (or cooler, or chilly bin, depending on what you want to call it!) in my office, which doesn’t get too hot or too cold, and I’ll keep those humidity bags in there to keep it dry.
What is little talked about is how to store tapes long term. When the tapes are not in use, they should be stored on end. They should not be allowed to lay flat (reel flanges parallel with the table top) for extended periods of time. There is a vague guideline to migrate your data every decade, at the very least you would want to test the tape for data integrity at intervals.
Despite some (sensible) requirements, long term you are going to save a lot more money going with LTO drives than going with hard drives.
What took me so long?
But what about the other reason I hadn’t looked at LTO tapes before? I thought they’d be too complicated. I’ve found this is slightly true, but not really.
The drive I went with is the OWC Mercury Pro LTO. This is an LTO 8 drive that is backwards compatible with LTO 7. It has Thunderbolt 3 in and out, a display port out and also has an internal bay that you can install a hard drive in. I think the idea behind this is, you put a SSD in there and copy your data onto that to then have it copy faster onto the tape.
And that’s something I didn’t know before. With the different applications you can get for LTO drives these days, the tape can look to your computer just like an external drive. So to get stuff off, you could use it like drag and drop. To get stuff on is a bit different as you need to use one of the Apps you can get for this to copy data onto the tape.
Being able to see what stuff is on the tape as if it’s a hard drive is useful. But remember, it will take time to get data off. This is because the tape drive has to actually move the tape around to find and recover the data from it.
Backup and Restore
I tried two different Apps. Canister from Hedge software is the simpler of the two. It has a fairly straightforward interface and it’s mostly just drag and drop. It also creates an index of what you’re copying onto the tape. This is really helpful as you can then search that index for find which tape you put stuff on without having to actually mount the tape and search the tape. The “challenge” with Canister is that it can be too simple if something goes wrong. I had a few copies finish but with a warning. The warning it gives you is a text files that shows what did copy and what didn’t, but not much info on why.
This happened to me a couple of times and I will say that their tech support is really good. They went above and beyond to work out what was wrong and to fix it.
The other Application I tried out was P5 from Archiware. This is a far more full featured archive option as it’s more targeted towards enterprise users. To put it simply, you get far more buttons. But the challenge with that is P5 can take a bit more work to get your head around. I can’t tell you how the troubleshooting process is with P5 as it worked perfectly. Though it did take me about 30 minutes of fiddling with it to work out how to get going.
Two things I’d like to mention is that you also need to install drivers for your drive before you use either of these software options. This may sound obvious for PC users, but Mac users may not be used to this.
Also you can’t have both installed on the same machine at the same time. Did I mention that LTO stuff can be a bit finicky?
I’d highly recommend you take a look at both theses options and decide which one is going to work best for you as they both have their advantages and challenges.
I’d suggest looking at an option that creates LTFS format, from what I’ve been able to tell, this is the most compatible and so even if you decide to use a different drive or software in the future, you should still be able to read your tapes.
Keep in mind that tapes copy at up to 300MB/s, so its’ going to take some time to copy stuff. I found that it took about 1 hour per TB. So copying 12TB took about 12-14 hours. And another thing to mention is that these drives are not exactly quiet. So maybe do the copies overnight.
That brings us onto copying stuff off the tapes when you need it. I found this process the simplest. You mount the tape, select the stuff you want, where you want it and press go. Simple as that. I actually had to do this for a real job as I was doing these tests and was exceptionally relieved as to how easy this was.
Conclusions
To summarise my experience with LTO, it was not as simple and straight forward as I was hoping, but not really as complicated as I had first thought. Yes it cost me a pretty penny up front, but within less than a year it’s going to pay for itself.
So I would highly recommend anyone who has large amounts of data that they need to keep, but not access regularly have a look at LTO as a possible solution for you.
I’d like to thank Archivware for providing me with a temp licence to try out their software, Fujifilm for providing me with some tapes and Hedge for all their help getting Canister working.
