Borkware Photoblog

February 27, 2010

Reducing the size of your Lightroom 2 Catalog

Filed under: hacks, Lightroom, Random, tips — Mark Dalrymple @ 10:33 pm

My Lightroom library has 12000 images. My catalog (Lightroom 2 Catalog.lrcat) was half a gig (556 megs). That meant that every time I played with Lightroom I’d eventually have to ship 500 megs to the Time Capsule during the next backup. 48K of overhead per photo seems excessive.

By deleting history for all photos, I got it down to 161 megs (or about 13K per image). I rarely revisit the history of the edits I do in Lightroom, so I don’t see a compelling reason to keep them around.

How to do it

  • Make sure you have a backup of your catalog, just in case.
  • In the Library, go to “All Photos” in the Catalog
  • Command-A to select all
  • “D” to go into the develop module
  • Develop -> Clear History , choose “Selected Photos”
    Picture 6.png
  • Catalog Settings -> Relaunch and Optimize
  • Bask in the glory of having less catalog to back up

What happened

Danger. Nerdity approaching

To help optimize my backups I started using this ├╝ber-nifty utility called Backup Loupe which shows you what was actually backed up by time machine. I use a Time Capsule, and get annoyed when backups take a long time. I worked to trim down my backed-up directories and got (usually) pretty small backups.

Except when I used LightRoom. If I opened up LR and did anything, I’d have to ship the 550 meg catalog over to the Time Capsule. Half a gig isn’t that much in today’s scheme of things, but it’s still annoying. It still has to move across the network. It still has to get archived to an external disk for off-site storage (which is a glacial process to start out with). And it’s a fair amount of space. Worst-case is that you change the catalog once an hour, you’d end up with 390 gigs consumed on the Time Capsule (24 hourly backups, 31 daily backups). That’s a third of the space I paid for.

The catalog is stored as Lightroom 2 Catalog.lrcat in your Lightroom user directory. Luckily this file is a sqlite3 database:

% file '/Users/markd/Pictures/Lightroom/Lightroom 2 Catalog.lrcat'
/Users/markd/Pictures/Lightroom/Lightroom 2 Catalog.lrcat: SQLite database (Version 3)

That’s something I can deal with.

There’s a spiffy utility called sqlite3_analyzer, available from the SQLite download page.

Run it and it’ll spew a load of stuff at you.

In particular are the page counts for the tables:

ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP.. 101129      70.9%
ADOBE_ADDITIONALMETADATA.............. 17135       12.0%
ADOBE_IMAGEDEVELOPSETTINGS............ 6118         4.3%
ADOBE_PREVIEWCACHEPYRAMIDLEVELS....... 2809         2.0%

“Library Image Develop History Step” takes the most space, distantly followed by “Additional Metadata”.

Paging down the analyzer output, you find the detailed information about the table and its indices:

*** Table ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP and all its indices ***

Percentage of total database..........  70.9%
Number of entries..................... 131181
Bytes of storage consumed............. 414224384
Bytes of payload...................... 404684584   97.7%
Average payload per entry............. 3084.93

So 131,181 individual history steps, taking up 395 megs of space. An average of 3K per entry. Or about eleven steps per photo.

Just for fun, I wanted to see what the most-used steps were.

sqlite> select count(name), name from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP group by name order by 1 desc limit 20;
6423 | Crop Rectangle
4353 | Add Brush Stroke
2623 | Paste Settings
2485 | Exposure
1851 | Update Exposure Adjustment
1700 | Clarity
1555 | Vibrance
1534 | Fill Light
892 | Brightness
850 | Black Clipping
766 | Highlight Recovery
454 | Add Graduated Filter
402 | Update Brightness Adjustment

Looks like I do a lot of cropping, a lot of adjustment brushing, a lot of settings pasting, and a lot of exposure adjusting. I know I need to compose in-camera better and hold the damn thing straight. Looks like I have concrete evidence for that now.

In a copy of the Lightroom catalog, I picked the top 30 of history step types and created tables of their contents, just to see who the biggest physical consumers were. I figure all editing history steps are not created equal.

Using the power of emacs, I quickly created and executed a lot of commands like:

create table blah_crop as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Crop Rectangle';
create table blah_addbrush as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Add Brush Stroke';
create table blah_pastesettings as select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Paste Settings';

And then re-ran the sqlite3_analyzer.

ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP.. 101129      42.1%
BLAH_ADDBRUSH......................... 60780       25.3%
ADOBE_ADDITIONALMETADATA.............. 17135        7.1%
BLAH_UPDATEEXPOSUREADJ................ 10898        4.5%
BLAH_ENABLEBRUSH...................... 6417         2.7%
ADOBE_IMAGEDEVELOPSETTINGS............ 6118         2.5%
BLAH_UPDATEBRIGHTNESS................. 4147         1.7%

The percentage of space that “Develop History Step” consumes has dropped because of the duplicate data I inserted. But you can see that the steps take up 101129 database blocks, and the “Add Brush Stroke” history step takes up well over half of them. 60,780 blocks consumes 237 megs. With 4353 “Add Brush Strokes”, that averages out to 56K per brush stroke.

Looking at one brush step, you can see why:

sqlite3> select * from ADOBE_LIBRARYIMAGEDEVELOPHISTORYSTEP where name = 'Add Brush Stroke' limit 1;
...
Dabs = { "M 0.319562 0.913815",
"M 0.349541 0.931545",
"M 0.361839 0.944725",
"M 0.329643 0.946453",
"M 0.297456 0.948412",
"M 0.265375 0.952863",
"M 0.233249 0.955063",
"M 0.201048 0.953529",
"M 0.168847 0.951995",
...

That’s a lot of dabbing.

After some milling around, I discovered “Clear History”, verified that it wouldn’t reset my image’s edits, just wipe out the history. After clearing out all the history, and vacuuming the database, it got a lot smaller.

Now, the big spenders in the file are

ADOBE_ADDITIONALMETADATA.............. 17135       41.3%
ADOBE_IMAGEDEVELOPSETTINGS............ 6120        14.8%
ADOBE_PREVIEWCACHEPYRAMIDLEVELS....... 2809         6.8%
AGLIBRARYFILE......................... 2180         5.3%
ADOBE_LIBRARYIMAGEDEVELOPSNAPSHOT..... 2007         4.8%
AGPHOTOPROPERTY....................... 1741         4.2%
ADOBE_IMAGES.......................... 1665         4.0%

Which works out to about 5K of additional metadata per image, which I can live with.

April 10, 2008

Tilt/Shift Movements

Filed under: Random — Mark Dalrymple @ 7:32 pm
Ken Rockwell has an introduction to Camera movements : why tilt and shift, as it applies to 35mm PC (T/S) lenses.

DIY Bokeh

Filed under: Random, tips — Mark Dalrymple @ 7:19 pm
“Bokeh” is a term that describes the out-of-focus highlights in an image. Different lenses have different kinds of bokeh, from smooth and creamy to seriously ugly. DIY Photography and Studio Lighting show a way to make your own bokeh shapes. Now to see if I can make a happy-face bokeh.

Create a free website or blog at WordPress.com.