Bug #617

Optimize TIFF writing

Added by Andreas Huggel 378 days ago. Updated 74 days ago.

Status:Closed Start:28 Feb 2009
Priority:Normal Due date:
Assigned to:Andreas Huggel % Done:

100%

Category:TIFF parser
Target version:0.19

Description

Writing to TIFF uses too much memory.

Reading
ahuggel@mowgli> ls -la P1010003.TIF 
-rw------- 1 ahuggel ahuggel 9451593 01-Mar-09 P1010003.TIF
valgrind exiv2-0.18 P1010003.TIF 
malloc/free: 2,121 allocs, 2,121 frees, 103,862 bytes allocated.
Non-intrusive writing
valgrind exiv2-0.18 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,608 allocs, 6,608 frees, 10,086,737 bytes allocated.
Intrusive writing
valgrind exiv2-0.18 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 5,940 allocs, 5,940 frees, 19,548,553 bytes allocated.

Related issues

related to Exiv2 - Bug #671 Writing to read-only TIFF-like file fails Resolved 06 Jan 2010

Associated revisions

Revision 1755
Added by Andreas Huggel 378 days ago

#617: For TIFF images, use memory mapping for non-intrusive writing instead of reading image into memory.

Revision 1918
Added by Andreas Huggel 127 days ago

#617: Changed TiffParser::encode to use an instance of BasicIo instead of a Blob for intrusive writing (API change).

History

Updated by Andreas Huggel 378 days ago

With r1755 the image is no longer loaded into memory for non-intrusive writing. The amount of memory allocated is therefore reduced by about the size of the image (for both write modes, since non-intrusive writing is always attempted first):

Non-intrusive writing
valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,746 allocs, 6,746 frees, 650,106 bytes allocated.
Intrusive writing
valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 6,102 allocs, 6,102 frees, 10,114,031 bytes allocated.

Updated by Andreas Huggel 378 days ago

The backdraw of the change in r1755 is that in case Exiv2 crashes somewhere in the non-intrusive writing logic, this is now likely to result in modified and possibly corrupted image.

Updated by Andreas Huggel 298 days ago

The next step is to replace the use of a Blob in the intrusive writing methods with a BasicIo. This will require an API change in the *Parser::encode methods.

Updated by Andreas Huggel 298 days ago

  • % Done changed from 0 to 30

Updated by Andreas Huggel 298 days ago

  • Status changed from New to Assigned
  • Assigned to set to Andreas Huggel

Updated by Andreas Huggel 127 days ago

r1918 finally implements the second step: In intrusive writing, the image is no longer written to a Blob but directly to a BasicIo instance, i.e., straight to a file instead of a memory buffer. With this change, the complete image is never loaded into memory anymore.

Non-intrusive writing:

valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,322 allocs, 6,322 frees, 644,462 bytes allocated.

Intrusive writing:

valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 5,627 allocs, 5,627 frees, 526,459 bytes allocated.

Updated by Andreas Huggel 127 days ago

These changes also have an effect on performance when dealing with large TIFF images. For non-intrusive writing, the performance gain is huge, since the image is now modified in-place, without ever loading it. For intrusive writing, the time is now reasonably close to that of a simple copy operation.

The following tests were done with a 120MB TIFF image, trunk is roughly 0.18.2, unstable is r1918.

big.tif 126,172,596 bytes
time cp big.tif big1.tif
real    0m2.254s
user    0m0.001s
sys     0m0.205s

Non-intrusive writing

exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday' big.tif

      unstable            trunk
real  0.023s              3.101s
user  0.013s              0.117s
sys   0.011s              0.888s
mem   8,374,303 bytes     134,551,658 bytes

Intrusive writing

exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday noon exactly now' big.tif

      unstable            trunk
real  2.654s              3.790s
user  0.044s              0.802s
sys   0.744s              0.995s
mem   6,579,811 bytes     259,330,485 bytes

Updated by Andreas Huggel 127 days ago

  • Status changed from Assigned to Resolved
  • Target version set to 0.19
  • % Done changed from 30 to 100

Updated by Andreas Huggel 74 days ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF