Project

General

Profile

Bug #617

Optimize TIFF writing

Added by Andreas Huggel over 12 years ago. Updated almost 12 years ago.

Status:
Closed
Priority:
Normal
Category:
tiff parser
Target version:
Start date:
28 Feb 2009
Due date:
% Done:

100%

Estimated time:

Description

Writing to TIFF uses too much memory.

Reading

ahuggel@mowgli> ls -la P1010003.TIF 
-rw------- 1 ahuggel ahuggel 9451593 01-Mar-09 P1010003.TIF
valgrind exiv2-0.18 P1010003.TIF 
malloc/free: 2,121 allocs, 2,121 frees, 103,862 bytes allocated.

Non-intrusive writing

valgrind exiv2-0.18 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,608 allocs, 6,608 frees, 10,086,737 bytes allocated.

Intrusive writing

valgrind exiv2-0.18 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 5,940 allocs, 5,940 frees, 19,548,553 bytes allocated.


Related issues

Related to Exiv2 - Bug #671: Writing to read-only TIFF-like file failsClosed06 Jan 2010

Actions

Associated revisions

Revision 1755 (diff)
Added by Andreas Huggel over 12 years ago

#617: For TIFF images, use memory mapping for non-intrusive writing instead of reading image into memory.

Revision 1918 (diff)
Added by Andreas Huggel about 12 years ago

#617: Changed TiffParser::encode to use an instance of BasicIo instead of a Blob for intrusive writing (API change).

History

#1

Updated by Andreas Huggel over 12 years ago

With r1755 the image is no longer loaded into memory for non-intrusive writing. The amount of memory allocated is therefore reduced by about the size of the image (for both write modes, since non-intrusive writing is always attempted first):

Non-intrusive writing

valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,746 allocs, 6,746 frees, 650,106 bytes allocated.

Intrusive writing

valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 6,102 allocs, 6,102 frees, 10,114,031 bytes allocated.

#2

Updated by Andreas Huggel over 12 years ago

The backdraw of the change in r1755 is that in case Exiv2 crashes somewhere in the non-intrusive writing logic, this is now likely to result in modified and possibly corrupted image.

#3

Updated by Andreas Huggel over 12 years ago

The next step is to replace the use of a Blob in the intrusive writing methods with a BasicIo. This will require an API change in the *Parser::encode methods.

#4

Updated by Andreas Huggel over 12 years ago

  • % Done changed from 0 to 30
#5

Updated by Andreas Huggel over 12 years ago

  • Status changed from New to Assigned
  • Assignee set to Andreas Huggel
#6

Updated by Andreas Huggel about 12 years ago

r1918 finally implements the second step: In intrusive writing, the image is no longer written to a Blob but directly to a BasicIo instance, i.e., straight to a file instead of a memory buffer. With this change, the complete image is never loaded into memory anymore.

Non-intrusive writing:

valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF
malloc/free: 6,322 allocs, 6,322 frees, 644,462 bytes allocated.

Intrusive writing:

valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF
malloc/free: 5,627 allocs, 5,627 frees, 526,459 bytes allocated.
#7

Updated by Andreas Huggel about 12 years ago

These changes also have an effect on performance when dealing with large TIFF images. For non-intrusive writing, the performance gain is huge, since the image is now modified in-place, without ever loading it. For intrusive writing, the time is now reasonably close to that of a simple copy operation.

The following tests were done with a 120MB TIFF image, trunk is roughly 0.18.2, unstable is r1918.

big.tif 126,172,596 bytes
time cp big.tif big1.tif
real    0m2.254s
user    0m0.001s
sys     0m0.205s

Non-intrusive writing

exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday' big.tif

      unstable            trunk
real  0.023s              3.101s
user  0.013s              0.117s
sys   0.011s              0.888s
mem   8,374,303 bytes     134,551,658 bytes

Intrusive writing

exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday noon exactly now' big.tif

      unstable            trunk
real  2.654s              3.790s
user  0.044s              0.802s
sys   0.744s              0.995s
mem   6,579,811 bytes     259,330,485 bytes
#8

Updated by Andreas Huggel about 12 years ago

  • Status changed from Assigned to Resolved
  • Target version set to 0.19
  • % Done changed from 30 to 100
#9

Updated by Andreas Huggel almost 12 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF