Borkware Photoblog

February 27, 2010

Reducing the size of your Lightroom 2 Catalog

Filed under: hacks, Lightroom, Random, tips — Mark Dalrymple @ 10:33 pm

My Lightroom library has 12000 images. My catalog (Lightroom 2 Catalog.lrcat) was half a gig (556 megs). That meant that every time I played with Lightroom I’d eventually have to ship 500 megs to the Time Capsule during the next backup. 48K of overhead per photo seems excessive.

By deleting history for all photos, I got it down to 161 megs (or about 13K per image). I rarely revisit the history of the edits I do in Lightroom, so I don’t see a compelling reason to keep them around.

How to do it

  • Make sure you have a backup of your catalog, just in case.
  • In the Library, go to “All Photos” in the Catalog
  • Command-A to select all
  • “D” to go into the develop module
  • Develop -> Clear History , choose “Selected Photos”
    Picture 6.png
  • Catalog Settings -> Relaunch and Optimize
  • Bask in the glory of having less catalog to back up

What happened

Danger. Nerdity approaching

To help optimize my backups I started using this über-nifty utility called Backup Loupe which shows you what was actually backed up by time machine. I use a Time Capsule, and get annoyed when backups take a long time. I worked to trim down my backed-up directories and got (usually) pretty small backups.

Except when I used LightRoom. If I opened up LR and did anything, I’d have to ship the 550 meg catalog over to the Time Capsule. Half a gig isn’t that much in today’s scheme of things, but it’s still annoying. It still has to move across the network. It still has to get archived to an external disk for off-site storage (which is a glacial process to start out with). And it’s a fair amount of space. Worst-case is that you change the catalog once an hour, you’d end up with 390 gigs consumed on the Time Capsule (24 hourly backups, 31 daily backups). That’s a third of the space I paid for.

The catalog is stored as Lightroom 2 Catalog.lrcat in your Lightroom user directory. Luckily this file is a sqlite3 database:

% file '/Users/markd/Pictures/Lightroom/Lightroom 2 Catalog.lrcat'
/Users/markd/Pictures/Lightroom/Lightroom 2 Catalog.lrcat: SQLite database (Version 3)

That’s something I can deal with.

There’s a spiffy utility called sqlite3_analyzer, available from the SQLite download page.

Run it and it’ll spew a load of stuff at you.

In particular are the page counts for the tables:

ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP.. 101129      70.9%
ADOBE_ADDITIONALMETADATA.............. 17135       12.0%
ADOBE_IMAGEDEVELOPSETTINGS............ 6118         4.3%
ADOBE_PREVIEWCACHEPYRAMIDLEVELS....... 2809         2.0%

“Library Image Develop History Step” takes the most space, distantly followed by “Additional Metadata”.

Paging down the analyzer output, you find the detailed information about the table and its indices:

*** Table ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP and all its indices ***

Percentage of total database..........  70.9%
Number of entries..................... 131181
Bytes of storage consumed............. 414224384
Bytes of payload...................... 404684584   97.7%
Average payload per entry............. 3084.93

So 131,181 individual history steps, taking up 395 megs of space. An average of 3K per entry. Or about eleven steps per photo.

Just for fun, I wanted to see what the most-used steps were.

sqlite> select count(name), name from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP group by name order by 1 desc limit 20;
6423 | Crop Rectangle
4353 | Add Brush Stroke
2623 | Paste Settings
2485 | Exposure
1851 | Update Exposure Adjustment
1700 | Clarity
1555 | Vibrance
1534 | Fill Light
892 | Brightness
850 | Black Clipping
766 | Highlight Recovery
454 | Add Graduated Filter
402 | Update Brightness Adjustment

Looks like I do a lot of cropping, a lot of adjustment brushing, a lot of settings pasting, and a lot of exposure adjusting. I know I need to compose in-camera better and hold the damn thing straight. Looks like I have concrete evidence for that now.

In a copy of the Lightroom catalog, I picked the top 30 of history step types and created tables of their contents, just to see who the biggest physical consumers were. I figure all editing history steps are not created equal.

Using the power of emacs, I quickly created and executed a lot of commands like:

create table blah_crop as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Crop Rectangle';
create table blah_addbrush as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Add Brush Stroke';
create table blah_pastesettings as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Paste Settings';

And then re-ran the sqlite3_analyzer.

ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP.. 101129      42.1%
BLAH_ADDBRUSH......................... 60780       25.3%
ADOBE_ADDITIONALMETADATA.............. 17135        7.1%
BLAH_UPDATEEXPOSUREADJ................ 10898        4.5%
BLAH_ENABLEBRUSH...................... 6417         2.7%
ADOBE_IMAGEDEVELOPSETTINGS............ 6118         2.5%
BLAH_UPDATEBRIGHTNESS................. 4147         1.7%

The percentage of space that “Develop History Step” consumes has dropped because of the duplicate data I inserted. But you can see that the steps take up 101129 database blocks, and the “Add Brush Stroke” history step takes up well over half of them. 60,780 blocks consumes 237 megs. With 4353 “Add Brush Strokes”, that averages out to 56K per brush stroke.

Looking at one brush step, you can see why:

sqlite3> select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Add Brush Stroke' limit 1;
...
Dabs = { "M 0.319562 0.913815",
"M 0.349541 0.931545",
"M 0.361839 0.944725",
"M 0.329643 0.946453",
"M 0.297456 0.948412",
"M 0.265375 0.952863",
"M 0.233249 0.955063",
"M 0.201048 0.953529",
"M 0.168847 0.951995",
...

That’s a lot of dabbing.

After some milling around, I discovered “Clear History”, verified that it wouldn’t reset my image’s edits, just wipe out the history. After clearing out all the history, and vacuuming the database, it got a lot smaller.

Now, the big spenders in the file are

ADOBE_ADDITIONALMETADATA.............. 17135       41.3%
ADOBE_IMAGEDEVELOPSETTINGS............ 6120        14.8%
ADOBE_PREVIEWCACHEPYRAMIDLEVELS....... 2809         6.8%
AGLIBRARYFILE......................... 2180         5.3%
ADOBE_LIBRARYIMAGEDEVELOPSNAPSHOT..... 2007         4.8%
AGPHOTOPROPERTY....................... 1741         4.2%
ADOBE_IMAGES.......................... 1665         4.0%

Which works out to about 5K of additional metadata per image, which I can live with.

Advertisements

February 7, 2009

Self-Portraiture Tools

Filed under: gallery, Lightroom, Nikon, portraiture, technique — Mark Dalrymple @ 4:25 pm

Img0082.jpgYou can do all the book reading and video watching in the world, but it won’t help you much if you don’t actually practice the stuff. I have a problem with wasting other people’s time, so I ended up doing a fair amount of self-portraiture to experiment with different techniques and get a feel for the tools and gear at my disposal. I’ve done a couple of sessions in different parts of the house, and have come up with some things that worked for me.

One hard part is just triggering the camera. I’ve got a D3, which doesn’t have a built-in wireless shutter release, so I used a cable release for one session. That meant the camera had to be close enough for the cable to reach (it’s not terribly long), which meant using a wider lens (24-70 in this case) I’d be out of luck with a longer lens

A big problem was getting the focus even somewhat accurate. There was a lot of sit down shoot get up chimp tweak focus sit down shoot get up chimp tweak etc etc etc until I found a good spot. Once that’s done, you can leave the camera alone and just mess around with the lights.

For my second session I tried tethering. The first problem was getting a long enough USB cable to connect the camera and the computer. I found an “active usb cable” online which works nicely. It uses power from the usb port to enable a longer distance cable. I could get my 70-200 far enough away by using this cable.

The next step is software. Aperture can do tethering, but unfortunately, it locks out the camera controls. Once you start tethering, you can’t change anything on the camera. Lightroom can’t do tethering, but you can fake it by having tether software drop a file into the file system and tell Lightroom where to look for things. Things to make it go. Tip : go to “loupe” mode, and new photos will appear large on screen. Other library modes will put the new photos at the end and leave the current one selected.

Img0074.jpgCanon folks get remote control software for free with their cameras. Unfortunately (for me) Nikon tries to make their software a profit center. Camera Control Pro 2 is the software to use for remote control of the camera. It’s also $140-$200. Luckily there’s a 30 day trial, and I’m concentrating on this stuff for a month, so I can get by with the trial.

CCP2 is a nice bit of software, letting you change settings on the camera from your laptop, then click a button to fire the shutter. The image comes over the wire and gets dropped in the file system where it gets picked up by Lightroom. LiveView can feed a video stream to the computer, and you can trigger the contrast-detection autofocus. WOOT.

This means I can sit in my comfy chair, point the camera my direction (and hope the lens doesn’t shatter this time), use LiveView autofocus to focus, and trigger the exposure with a click of a button. I used a mouse so I didn’t have to lean over the lappy to fire the camera.

One alternative to tethering is to get a HDMI monitor and connect that to the camera. Set the autochimp mode and see the image appear after you shoot it. If the camera is physically distant you can’t zoom in or trigger other modes.

There’s still a fair amount of up-and-down action that happens, particularly adjusting lighting ratios. I used an SU-800 wireless trigger along with remote strobes. The SU-800 is attached to the camera, so you have to get up and piddle with that. Next time I’m going to try using a hotshoe extender cable so I can keep the commander within arm’s reach.

January 31, 2009

New Galleries: Seattle

Filed under: gallery, Lightroom — Mark Dalrymple @ 10:18 pm

I’ve finally edited and uploaded some photos I took while visiting the Seattle/Kirkland area. Seattle Photoduggery includes shots of a larval blubberbot, plus waking around gray and mossy parks and arboretums.

Seattle Airport is cool stuff I found wandering around SEA-TAC, the Seattle/Tacoma airport. It’s a surprisingly fun airport. I recommend scheduling a couple of extra hours layover to see everything.

Many of these I edited in LightRoom – my first “real” use of the software. I like it. Even though it’s a modal interface compared to Aperture, I did not mind it much. The only pain was when I started falling back to Aperture keyboard shortcuts, which do very different things in LightRoom.

January 28, 2009

Lightroom Shortcut Reference

Filed under: Lightroom, tips — Mark Dalrymple @ 11:26 pm

Here are some shortcut keys I’ve come across in Lightroom 2. This will get updated as I come across more.

General

cmd-/ – show all of the command shortcuts for the current module.

cmd-opt-[1-5] – move between different modules

F5-F8 – Toggle the around-screen panels (T/B/L/R)

tab – Toggle left/right panels

shift-tab – Toggle all panels

L – Cycle through “Lights-out” modes (dim, dark, normal)

F – Cycle through three full-screen modes

cmd-shift-F – Full Screen

cmd-shift-F, T – Full view with minimal distractions

cmd-option-shift-E – Export using the last settings

T – Toggle grid view toolbar

0-5 – apply zero to 5 star rating

6-9 – red, yellow, green, blue label. No shortcut for purple or none.

Library Module

G – Return to (library) grid mode

J – Toggle library grid cells between the three display states

P – Mark as pick (white flag)

X – Mark as reject (black flag)

U – Clear flag state

shift-{P, X, U} – Set flag state and move to next image

N – Go into survey mode.

/ – Deselect active photo (also removes from survey mode)

cmd-D – Deselect everything


C – Compare mode. “Select” image is on the left. Use right arrow to move on. Use X<-Y button to make new shot the Select.



B – Add to Quick Collection

cmd-return – Slide show of the quick collection

\ – Toggle library filter bar

cmd-G – Group as a stack (only in folder panel)

cmd-shift-G – Ungroup stack

S – Toggle stack contents

cmd-option-K – Turn paint tool on or off

option-click-+ in smart collection editor – Add a collection option (with any/all subcriteria)

I – Cycle through different info windows

cmd-J – Configure the info windows (loupe view)

Z – Zoom to 100%

ctrl-,- – Change thumbnail size in grid view

D – Go to Develop module

Develop Module

J – Toggle clipping highlights

option-click – exposure/recovery slider and blacks slider to get clipped region

\ – Toggle master/version

Y – Side-by-side master/version

shift-Y – Split-screen master/version

option-Y – Side-by-side, vertical

cmd-' – Make virtual copy

cmd-click Sync – Auto sync on multiple selected images

cmd-{1,2,3,etc} – Go to successive panels on the right

, . – Jump to next/previous slider in “Basic” editing panel

V – Quick grayscale

cmd-N – Make a snapshot

K – Adjustment brush

M – Gradient filter

' – Invert gradient

H – Toggle adjustment brush pin visibility

A – Toggle adjustment brush automask

1-0 – Control adjustment brush flow (10% -> 100%)

O – Toggle adjustment brush mask

Return – Adds new adjustment brush pin

/ – Toggle between A and B adjustment brushes

[ ] – Change size of adjustment brush. Add shift to change feather.

cmd-shift-H – Toggle rule-of-thirds grid in crop tool

R – Toggle crop tool

R, shift-tab, cmd-shift-H, L, L – Crop in lights-out mode. (go into crop mode, hide panels, hide thirds grid if shown, lights off)

page-down – When zoomed in, will scroll by screenfulls, down and then across.

N – Toggle spot remover tool

January 22, 2009

Just Finished Reading: Adobe Lightroom 2 Book for Digital Photographers

Filed under: Books, Buy It, Lightroom, Photoshop — Mark Dalrymple @ 2:22 am

51UKRSIADvL._SL500_AA240_.jpg

Rating: Buy It

As I mentioned earlier, my current main projects are portraiture and LightRoom. What better way to kick off the Lightroom 30-day trial then with a book that Explains It All. It took me a weekend to work through it all, although I did skim over the printing and slideshow chapters. I’ll probably be using Photoshop to do all my printing, and I never show slideshows to people.

If there is a Kelby book on a topic, I’ll usually reach for that first. They tend not to have a great deal of reference utility for me, but for a survey of features of a product like Lightroom or Photoshop, they’re hard to beat.

They’re also fun. I’m sure some folks find it annoying, but I enjoy humor in my technical books. (Just read some of the reviews of My Latest Tome for corroboration). Each chapter has a mostly-unrelated stream of consciousness introduction, covering topics from flobotnor to Vince Versace’s hallucinogenic cooking. Individual tutorial steps, in between the “select this” or “use X to toggle this mode” instructions, there live little chunklets of humor too.

His LightRoom 2 book is just like his Seven-Step photoshop book. There are chapters on the different modules and the main features in those modules with step-by-step tutorials on using the features. As he goes along, the different features are used together so you can get a feel for a workflow in using the product. Each chapter ends with a couple of pages of quickies discussing advanced features. If grok the preceding chapter, the quickies are all you need.

You can download sample photos to work through some of the examples, which is about the only real complaint I have with the book. Some of the examples don’t have corresponding photos in the download (due to usage rights). Sometimes a photo was actually placed in a later chapter’s download archive, so you thought it was removed on purpose, but it was just mis-filed. And some of the photos were “after” photos. In particular, the dust spot removal sample of the hotel in Dubai had already been corrected. No Fun!

Of course, he covers all of the Lightroom modules (library, develop, slide show, print, web), but he also includes examples of round-trips to photoshop, including double-processing a file. Unfortunately for that one he seems to have copied and pasted a dozen pages from his seven-point book with all sorts of photoshop stuff which I had already seen before. Also included are two “workflows”, one for on-location portrait photographers shooting tethered, and another for folks on vacation with their laptop, showing how to use libraries on a remote machine, and then migrating stuff back to their Main Machine when they get back. There’s a bonus video at Kelby Training with a wedding workflow. You have to type in a Secret Code from the book to get access to this video.

If you’ve never used LightRoom before, and you’ve just started your 30 day free trial, it’s worth it to get this book to get up to speed quickly on the software so you can decide whether to spend $300 on it.

January 18, 2009

This month’s Projects: Portraiture and LightRoom

Filed under: Aperture, Lightroom, projects, technique — Mark Dalrymple @ 5:02 pm

Now that Learn Objective-C has shipped, I actually have some photography time to myself. Just got back from business travel, so my months now run from middle of the month to middle of month.

This month, the projects are:

  • laurel_web.jpg Portraiture. I’m going to be taking some individual and group portraits of my friends in the Aeolian Winds of Pittsburgh. I’ve already shot some individual portraits (as seen here), and some group shots at outdoor concerts, but never a “formal” sitting of the individuals and group. This will also be an excuse to go back and re-read all of my portraiture books.

  • Lightroom. Aperture so far has been my Photographic Axe of choice. I love the vaults and way I can bounce around and do stuff. Unfortunately, there are just a couple of things that have really been getting on my nerves: Nearly all of my portrait-orientation photos come out rotated wrong, so I have to find and rotate them manually, which then destroys the correctness of the “landscape/portrait” metadata. Also, doing bulk edits with plugins (like the Nik filters) causes the edited images to end up in stacks (which is fine), but not as the pick. Cmd-/ (make pick) does not work with multiple selection, so I have to manually set the pick for every stack. When you do this with 2 or 300 shots, well, that sucks. Frequently, when paging through large collections in full-screen mode, Aperture will get stuck at the “loading” phase. Going back and then forward clears it, disrupting the working rhythm. Also, I still can’t make Levels do what I want it to do like I can with Photochop curves. And I really like the Fill Light™ control in ACR. I haven’t been able to duplicate that in Aperture.

So, I’ve picked up the Scott Kelby Lightroom book, and I’m also a Kelby Training subscriber, so this month I’m gonna live in LightRoom and see if things are any more pleasant.

Create a free website or blog at WordPress.com.