Optimize TIFF writing
|Status:||Closed||Start date:||28 Feb 2009|
|Assignee:||Andreas Huggel||% Done:|
Writing to TIFF uses too much memory.
ahuggel@mowgli> ls -la P1010003.TIF -rw------- 1 ahuggel ahuggel 9451593 01-Mar-09 P1010003.TIF valgrind exiv2-0.18 P1010003.TIF malloc/free: 2,121 allocs, 2,121 frees, 103,862 bytes allocated.
valgrind exiv2-0.18 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,608 allocs, 6,608 frees, 10,086,737 bytes allocated.
valgrind exiv2-0.18 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 5,940 allocs, 5,940 frees, 19,548,553 bytes allocated.
#617: For TIFF images, use memory mapping for non-intrusive writing instead of reading image into memory.
#1 Updated by Andreas Huggel almost 9 years ago
With r1755 the image is no longer loaded into memory for non-intrusive writing. The amount of memory allocated is therefore reduced by about the size of the image (for both write modes, since non-intrusive writing is always attempted first):
valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,746 allocs, 6,746 frees, 650,106 bytes allocated.
valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 6,102 allocs, 6,102 frees, 10,114,031 bytes allocated.
#6 Updated by Andreas Huggel about 8 years ago
r1918 finally implements the second step: In intrusive writing, the image is no longer written to a Blob but directly to a BasicIo instance, i.e., straight to a file instead of a memory buffer. With this change, the complete image is never loaded into memory anymore.
valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,322 allocs, 6,322 frees, 644,462 bytes allocated.
valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 5,627 allocs, 5,627 frees, 526,459 bytes allocated.
#7 Updated by Andreas Huggel about 8 years ago
These changes also have an effect on performance when dealing with large TIFF images. For non-intrusive writing, the performance gain is huge, since the image is now modified in-place, without ever loading it. For intrusive writing, the time is now reasonably close to that of a simple copy operation.
The following tests were done with a 120MB TIFF image, trunk is roughly 0.18.2, unstable is r1918.
big.tif 126,172,596 bytes time cp big.tif big1.tif real 0m2.254s user 0m0.001s sys 0m0.205s
exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday' big.tif unstable trunk real 0.023s 3.101s user 0.013s 0.117s sys 0.011s 0.888s mem 8,374,303 bytes 134,551,658 bytes
exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday noon exactly now' big.tif unstable trunk real 2.654s 3.790s user 0.044s 0.802s sys 0.744s 0.995s mem 6,579,811 bytes 259,330,485 bytes