Bug #617
Optimize TIFF writing
| Status: | Closed | Start: | 28 Feb 2009 | |
| Priority: | Normal | Due date: | ||
| Assigned to: | Andreas Huggel | % Done: | 100% |
|
| Category: | TIFF parser | |||
| Target version: | 0.19 | |||
Description
Writing to TIFF uses too much memory.
Readingahuggel@mowgli> ls -la P1010003.TIF -rw------- 1 ahuggel ahuggel 9451593 01-Mar-09 P1010003.TIF valgrind exiv2-0.18 P1010003.TIF malloc/free: 2,121 allocs, 2,121 frees, 103,862 bytes allocated.Non-intrusive writing
valgrind exiv2-0.18 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,608 allocs, 6,608 frees, 10,086,737 bytes allocated.Intrusive writing
valgrind exiv2-0.18 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 5,940 allocs, 5,940 frees, 19,548,553 bytes allocated.
Related issues
| related to Exiv2 - Bug #671 | Writing to read-only TIFF-like file fails | Resolved | 06 Jan 2010 |
Associated revisions
#617: For TIFF images, use memory mapping for non-intrusive writing instead of reading image into memory.
#617: Changed TiffParser::encode to use an instance of BasicIo instead of a Blob for intrusive writing (API change).
History
Updated by Andreas Huggel 378 days ago
With r1755 the image is no longer loaded into memory for non-intrusive writing. The amount of memory allocated is therefore reduced by about the size of the image (for both write modes, since non-intrusive writing is always attempted first):
Non-intrusive writingvalgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,746 allocs, 6,746 frees, 650,106 bytes allocated.Intrusive writing
valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 6,102 allocs, 6,102 frees, 10,114,031 bytes allocated.
Updated by Andreas Huggel 378 days ago
The backdraw of the change in r1755 is that in case Exiv2 crashes somewhere in the non-intrusive writing logic, this is now likely to result in modified and possibly corrupted image.
Updated by Andreas Huggel 298 days ago
The next step is to replace the use of a Blob in the intrusive writing methods with a BasicIo. This will require an API change in the *Parser::encode methods.
Updated by Andreas Huggel 298 days ago
- % Done changed from 0 to 30
Updated by Andreas Huggel 298 days ago
- Status changed from New to Assigned
- Assigned to set to Andreas Huggel
Updated by Andreas Huggel 127 days ago
r1918 finally implements the second step: In intrusive writing, the image is no longer written to a Blob but directly to a BasicIo instance, i.e., straight to a file instead of a memory buffer. With this change, the complete image is never loaded into memory anymore.
Non-intrusive writing:
valgrind ./exiv2 -M'set Exif.Image.Software Exiv2' P1010003.TIF malloc/free: 6,322 allocs, 6,322 frees, 644,462 bytes allocated.
Intrusive writing:
valgrind ./exiv2 -M'set Exif.Image.Software The Exiv2 utility' P1010003.TIF malloc/free: 5,627 allocs, 5,627 frees, 526,459 bytes allocated.
Updated by Andreas Huggel 127 days ago
These changes also have an effect on performance when dealing with large TIFF images. For non-intrusive writing, the performance gain is huge, since the image is now modified in-place, without ever loading it. For intrusive writing, the time is now reasonably close to that of a simple copy operation.
The following tests were done with a 120MB TIFF image, trunk is roughly 0.18.2, unstable is r1918.
big.tif 126,172,596 bytes time cp big.tif big1.tif real 0m2.254s user 0m0.001s sys 0m0.205s
Non-intrusive writing
exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday' big.tif
unstable trunk
real 0.023s 3.101s
user 0.013s 0.117s
sys 0.011s 0.888s
mem 8,374,303 bytes 134,551,658 bytes
Intrusive writing
exiv2 -M'set Exif.Photo.DateTimeOriginal Yesterday noon exactly now' big.tif
unstable trunk
real 2.654s 3.790s
user 0.044s 0.802s
sys 0.744s 0.995s
mem 6,579,811 bytes 259,330,485 bytes
Updated by Andreas Huggel 127 days ago
- Status changed from Assigned to Resolved
- Target version set to 0.19
- % Done changed from 30 to 100
Updated by Andreas Huggel 74 days ago
- Status changed from Resolved to Closed