DAM stuff: the big data reorg of 2015

I’ve spent the most of the past two weeks doing some one DAM thing or another. No, not cussin’, but rather Digital Asset Management – it’s a fancy way of saying I’ve been organizing my photos. Considering I have been doing this photography thing to varying degrees for the past 25+ years, and have been something of an obsessed maniac much of the past 5 or 6, there has been quite a bit of organization and clean up to do.

Harddrives, during the great Data Reorg and Lightroom Import of 2015.
I think there are around 24TB of hard drives here, used for various things. About 8TB is active storage, another 8TB for backups, and the rest is either for temporary use or is, after this recent reorg, empty.

Personal strategies

The reasons and goals for this spring cleaning are many. First, it just needs to be done periodically. I’ve been pretty organized all along, fortunately, but no matter how careful one is, systems change. The naming and filing conventions I use now are different that what I standardized on when I began as a digital photographer, which of course were radically different from a box of prints and negatives before that.

Some of the change is technology evolving, some is my philosophy changing, and some is just a result of learning better practices. Anyhow, from time to time it’s a good idea to revisit files and make sure they are (still) organized.

Lightroom

I’ve also decided to once again try to embrace Lightroom, and so I wanted to have things well organized before I started that process. It has taken me two weeks, pretty much 8-12 hours/day, 7 days a week, to finish.

Moving photos from around eight or nine different drives onto three new ones takes time. I have around 7 TB of photo and video files on those three drives. Then importing those 390,000 photos keeps Lightroom busy for a while, and then there were the many thousands of files that threw errors that I had to deal with.

lightroom_psd_errors
In total there were over 3000 of these psd issues to resolve.
lightroom_not_responding
Lots of this going on. As you can see, I don’t have a ton of apps running. My iMac is a bit old, but still should be ok, and has 8GB of RAM to work with.

Ok, so maybe there was some cussin’, too…

lightroom_helpful_error
This is my favorite error message. So helpful! (Would it kill you, Adobe, to at least include WHICH folder of the 500 I am currently importing caused the problem?!)

My iMac is, admittedly, a few years old, and most of my drives connect via USB 2.0 so neither of those factors help any.

lightroom_frozen
I had several opportunities to watch Lightroom hang while it digested nearly 400,000 photos.

There were a few that Lightroom didn’t like because they were some weird bit depth (I had one that was 32 bits?!) or color space (Indexed RGB anyone?). A couple of files were just corrupt. Most of the errors, though – several thousand – were PSD files that it couldn’t read because they hadn’t been saved with “compatibility turned on” (thanks a lot, Adobe, for not reading your own files).

To fix that I had to find all of the offending files and do what I still think is a horrible hack of a workaround – I created an action to open the file, add a blank layer, “save as” to overwrite the file (now with compatibility), then delete the fake layer, save, and close. Finding the files, in hundreds of different folders, and running that action, even in bulk batches, took several days.

Thinking ahead – digital longevity

Another reason to do this reorganization – one that applies to every photographer – is that hard drives age and technologies change. It’s a good idea to periodically move your files to a newer drive, both in terms of the drive’s wear and tear, and its technology.

There are plenty of dire warnings about how today’s photographs that exist only digitally will not be retrievable as our computers evolve – if you have any photos stored on 3.5 inch floppies…got any way to see them now? – but by periodically moving everything to the latest and greatest storage device this risk is mitigated.

There are currently 12.5 TB of drives in this relatively small box my dad made for me.
There are currently 12.5 TB of drives in this relatively small box my dad made for me.

And I still argue that my relatively little box of hard drives – which I could easily copy and stick in a small safe-deposit box – has better longevity and fidelity – with some TLC – than a room full of boxes of scratched negatives. We aren’t all Jay Maisel with a massive bank vault for a home to store them in (and even he isn’t, any more).

Yes, hard drives fail, yes technology changes. But it’s so easy and cheap to make multiple copies that if one fails, you should have at least two others. I think I can place every single photo I’ve ever made in my entire life on one 8TB hard drive for less than $300. If you had 500,000 negatives and prints (which the non-Winogrands among us probably couldn’t do on film to begin with) how much space would that take up, and how much would buying / renting / maintaining that space cost? Plus if the house is on fire it’s difficult to carry a room full of negs out to safety.

IMG_0941
There are currently 12.5 TB of drives in this relatively small box my dad made for me. The wooden one, not the Altoids tin.

(I am not advocating digital vs film, not talking about esthetic choices, etc., just saying that the whole “digital photos are going to disappear over time it’s the end of photography you must have a negative and a print” argument is a little bogus, at least for a thinking and conscientious (semi-)professional photographer.)

As is clear from the photo at the top, I have chosen to use primarily internal drives and either a dock or an enclosure to read them. The drives are generally cheaper without a case and power supply and such, and they are smaller and easier to store. They are not designed for this type of use, though, so if you go this route use some sense, be careful, and be aware of the risk.