Exiv2

I could be wrong about this being Windows only. I just converted 3900x2616 Nikon .NEF file to 16 bit TIF and transferred the EXIF data over (this was in my program, not using the standalone exiv2 program) and the resulting image was corrupted too. Could it be related to image size? Smaller images seemed to have worked better.

Paul Miller wrote:

I could be wrong about this being Windows only. I just converted 3900x2616 Nikon .NEF file to 16 bit TIF and transferred the EXIF data over on my Mac (this was in my program, not using the standalone exiv2 program) and the resulting image was corrupted too. Could it be related to image size? Smaller images seemed to have worked better.

Paul

Are you using the Windows binary from the web site, or did you build this yourself? If you built it yourself, how did you build it? msvc, msvc64, MinGW, Cygwin, or some other way?

Are you describing two different issues here?
1) filename is sometimes corrupted
2) file contents are sometimes corrupted

And can you post an example of a bad (and a good) file somewhere, so we can download and reproduce the issue.

Thanks.

I built the source myself from the latest svn repo, using Visual Studio 2008 (RTM, not SP1). Also built from source on Mac using gcc 4.2 for i386 and x86_64 arches. Linking statically.

The two issues seem to be related - is it possible it's writing a temp file (the name with numbers at the end) and then gives up/crashes -but only some of the time? Though my app isn't crashing.

I'm on my Mac right now and can't repro it reliably there but I'll get some images posted over the weekend.

Thanks as usual for being very responsive!

Paul

We aim to please (and it's Friday afternoon and I'm bored at work).

It's quite possible that the two issues are caused by single memory/corruption. Thanks for confirming that there are two artifacts (the corrupted file-name, and the corrupted image). I have an iMac at home (and Windows and Linux) and if you have test files which you'd like me to investigate, I'll look at them on Sunday.

Robin

Is there anything unusual about your setup, e.g., are you writing to an external disk, a different partition etc.?

The files that end in a number are temporary files as you suspected. They are renamed eventually. Temporary files are only used if the image is larger than 1MB otherwise exiv2 uses a temporary memory buffer instead.

I can't see any recent change in that area (basicio.cpp), we'll definitely need samples of these corrupted TIFFs to have only a remote chance to track this down. More likely we'll need a reproducer. The detailed steps how you could create a corrupted TIFF may be useful even if it doesn't happen always.

If you want to check for memory corruption issues yourself, run your program in Valgrind (available for Mac OSX but not Windows).

Andreas

Ok while working on a test case I realized that the corruption only seems to happen when using LZW compression. If I use NONE or ZIP TIFF compression the file seems to be tagged properly. Can anyone else confirm?

I'll continue to work on my simple test case to see if I can replicate the same problem as in my full-blown application.

Sorry that was a red-herring. I have my test program working now and it doesn't matter which compression option is used.

So I have a reproducible case now. I have a source Nikon image I'm reading the EXIF data from, then I'm generating a largish 16-bit TIFF image (just a checkerboard grid) and attempting to apply the EXIF data to that image. Sometimes, it throws an exception when trying to rename the temp file to the final .tif file. Other times it completes but the resulting image cannot be opened in Photoshop - it's corrupted.

I've uploaded my test kit here: http://www.fxtech.com/files/exiftest.zip

This contains my large Nikon image that I'm extracting the EXIF data from, the test program and Visual Studio project (which will need to be modified to build on another machine, since it pulls in LIBTIFF, ZLIB, and the EXIV2 libraries, which are not included), and a sample out.tif which is in the corrupted form. Also is the Windows binary for you to try like this:

exiftest.exe

This will load the _DSC0733.NEF EXIF data and write out.tif, with EXIF data (this results in the corrupted image).

exiftest.exe 1

This will skip the EXIF writing stage, and just generate the out.tif, which CAN be opened in Photoshop.

I suspect it's one or more of the tags in this image that is causing the problem, and possibly the resulting size of the image. I hope this is useful to someone. Let me know if you have any problems reproducing this!

Paul

Thanks for doing the digging on this and posting those files (they've arrived safely!).

504 /Users/rmills/Downloads/exiftest $ dir
-rwxr-xr-x@ 1 rmills staff 1.9M Apr 9 2011 exiftest.exe
-rwxr-xr-x@ 1 rmills staff 15K Apr 9 2011 exiftest.vcproj
-rwxr-xr-x@ 1 rmills staff 2.4M Apr 9 2011 out.tif
-rwxr-xr-x 1 rmills staff 3.9K Apr 9 2011 exiftest.cpp
-rwxr-xr-x@ 1 rmills staff 98K Jul 19 2010 zlib1.dll
-rwxr-xr-x@ 1 rmills staff 20M Dec 14 2006 _DSC0733.NEF
505 /Users/rmills/Downloads/exiftest $

I'll have a look in this on Sunday (or Monday evening). Is this a Windows only or a platform independent issue?

Robin

Definitely on Windows. I'll try to get my test program up and running on the Mac tomorrow morning and see if I can repro there.

Paul,

These appear to be two different issues:
1) Copying metadata from a NEF into a TIFF image results in a corrupted TIFF image
2) "Sometimes, it throws an exception when trying to rename the temp file to the final .tif file."

The first one is easier to deal with, I believe all that is required is to prune the Exif data before setting it in the TIFF image. In particular, remove Exif.Image.SubIFDs and all Exif.SubImage*.* tags. Exiv2 does this if you copy Exif data to a JPEG, see ExifParser::encode in src/exif.cpp for sample code. However if the target is a TIFF, it doesn't do this currently (because the same parser is used to write TIFFs and NEFs).

For the second issue, what is the exception thrown?

Andreas

I added the internal code to strip those tags out and now it's writing valid TIFF files - great!

The first time I ran the updated code it failed on the rename of the temp file though, but now I can't reproduce it. I'll keep an eye out for that one.

I'm still getting random crashes when saving EXIF data to somewhat large (over 100MB) TIFF images. It doesn't happen all the time, but when it does I get this exception:

FILENAME.tifXXXX: Failed to rename file to FILENAME.tif: File exists

Well, yeah FILENAME.tif exists - that's the file I'm writing to. I suspect there is some kind of race condition with the deletion of the .tif and renaming of the temp file to it. Maybe the memory unmap operation is asynchronous? This is way down in the guts of the exif write code so I still don't know what is going on.

Anyone else ever see this?

AHA - I found the problem in my code. Referencing deleted memory - oh my! Explains everything.