Bug #701

Nikon capture nx won't save a raw file which was metadata manipulated from svn exiv2

Added by Frank Hommes almost 7 years ago. Updated almost 7 years ago.

Status:ClosedStart date:03 May 2010
Priority:NormalDue date:
Assignee:-% Done:

100%

Category:tiff parser
Target version:0.20

Description

I can't save my pictures using nikon capture nx when I manipulate the metadata with exiv2. I don't know where the problem is, if it's a capture nx problem or an exiv2 problem.

I'm using the newest exiv2 with digikam 1.3

metadata_exiftool.mie (17.6 KB) Frank Hommes, 04 May 2010 02:42

metadata_exiv2.mie (14.9 KB) Frank Hommes, 04 May 2010 02:42

metadata.txt Magnifier (13.7 KB) Frank Hommes, 19 May 2010 07:29

metadata_broken_file_with_applied_fix.txt Magnifier (13.7 KB) Frank Hommes, 19 May 2010 10:14


Related issues

Blocks Exiv2 - Feature #668: "TIFF-safe" setExifData variant Closed 03 Jan 2010

Associated revisions

Revision 2219
Added by Andreas Huggel almost 7 years ago

#701: Sort Sub-IFDs by group before writing, as the order may not be correct after copying image tags.

Revision 2225
Added by Andreas Huggel almost 7 years ago

Branch for a hacked version of exiv2 to repair Nikon D90 images which were corrupted due to bug #701. Use at your own risk!

Revision 2226
Added by Andreas Huggel almost 7 years ago

#701: Hack to change the order of subdirs. This can be used to fix corrupted images.

History

#1 Updated by Andreas Huggel almost 7 years ago

What metadata manipulations have you done with exiv2?
Can you provide an example and step-by-step explanation (as simple as possible)?
Do you get any error or warning message from capture nx when you work with exiv2-manipulated images?

#2 Updated by Frank Hommes almost 7 years ago

I just did some tests:

I took an older nef image from my camera which is not effected by this bug. This image can be changed within capture nx and then saved as a nef. Capture nx only saves my changes without destrying any data. Saving works without a problem.

Second I took a picture and changed the rating with digkam and saved it to the raw file. I can load this file in capture nx but when I try to save it, it will give me an error (no specific one... Just Error 7...).
Third when I add a keyword the same happens. I can load it but saving as a nef doesn't work anymore.

Anyway, I can still save the picture from capture nx with all changes to a jpg or tiff. But just not nef, which is most important I guess.

I hope this explanation is better.

#3 Updated by Andreas Huggel almost 7 years ago

Can you take an image that works, then add the Rating tag using only the exiv2 command-line tool, then try again with Capture NX?
If Capture NX can't write to the image anymore, can you repeat this test with exiftool instead of exiv2?
If that also results in an image which capture nx doesn't like to write to, then repeat the test with exiv2 again but instead of adding the Rating tag, modify an existing ASCII text tag and make it longer than it is. E.g., a description or software tag, but not make or model.

Thanks in advance.
Andreas

#4 Updated by Frank Hommes almost 7 years ago

Ok, I also did additional experiments.
When I change a picture with capture nx then exiv2 gives me the following error when I try to change the rating with the command tool:
freanki@tristan:~/win/pics/temp$ exiv2 -M'set Exif.Image.Rating 5' 20100115-122401_changed_by_nikon_exiv2_rating_does_not_work.nef
Error: Directory Unknown with 32944 entries considered invalid; not read.
Error: Directory Unknown with 32944 entries considered invalid; not read.

From now on I took the original picture which came from the camera as nef:
- original pic changed with exiv2 command tool: can't save afterwards
- original pic changed with exiftool command tool: WORKS!

So now I tried to extract the metadata from the file changed with exiftool and write it to the file changed with exiv2 tools. I got the following error:
freanki@tristan:~/win/pics/temp$ exiftool -tagsFromFile metadata_exiv2.mie 20100115-122401_exiv2_rating_changed_new_metadata.nef
Error: [minor] SubIFD entry 15 has zero count - 20100115-122401_exiv2_rating_changed_new_metadata.nef
0 image files updated
1 files weren't updated due to errors

I attached the different metadata extracted from exiftools. Maybe it helps to find the problem.

#5 Updated by Andreas Huggel almost 7 years ago

Frank, can you please do a quick test with the latest SVN version. I'd be very happy if this problem also disappeared...

#6 Updated by Frank Hommes almost 7 years ago

I already tried it but waited for your suggestion here since it didn't help.

I took the original picture, changed the rating with the new exiv but it still could not be saved...

#7 Updated by Frank Hommes almost 7 years ago

oh, and of course it worked with the old version of exiv...

I tried to read out the metadata from the original and the changed file but there is no error I see... Just 8 more SubIFD1 entries.

#8 Updated by Frank Hommes almost 7 years ago

Oh and the latest patch solved the following error:

freanki@tristan:~/win/pics/temp$ exiftool -tagsFromFile metadata_exiv2.mie 20100115-122401_exiv2_rating_changed_new_metadata.nef
Error: [minor] SubIFD entry 15 has zero count - 20100115-122401_exiv2_rating_changed_new_metadata.nef
0 image files updated
1 files weren't updated due to errors

But I still haven't found a way to get my pictures to work with capture nx 2 :(
It's a pity because I used to do it this way and wasn't expecting such an error... yeah, my mistake I know...

All my affords to fix the exif data with exiftool, like copy them from original to the altered one, don't work...

I guess it has something to do with the end of the file because Nikon writes all changes to the nef (I'm doing with programs like capture nx) to the end of it. I can still read the pictures and save them as jpgs...

#9 Updated by Frank Hommes almost 7 years ago

Oh, I just noticed another thing in capture nx.

When I work on a raw file that is corrupted because of svn-exiv then there is no panel which should show the settings I choose with the camera like white-balance and stuff like that!
I read on several pages that modifying some areas of the exifdata like the model would destroy the camerainformation needed for some rawconverters to develop the image (but I guess you already know that).
Shall I send you an example image?

#10 Updated by Andreas Huggel almost 7 years ago

Shall I send you an example image?

Yes, can you send me one original file that, when edited with exiv2, shows the reported behaviour?
I can modify it myself.

Thanks,
Andreas

#11 Updated by Andreas Huggel almost 7 years ago

Exiv2 trunk reverses the order of the sub-images! That must be another side-effect of #668...

#12 Updated by Frank Hommes almost 7 years ago

Can it be fixed? And if it gets fixed, will it be possible to repair my files?

#13 Updated by Andreas Huggel almost 7 years ago

Can you please try if it works r2219? I hope this fixes the reported problem. But it won't help with files which are already corrupted. Do you have many of these? I'll have to come up with a special repair tool for that if really necessary.

#14 Updated by Frank Hommes almost 7 years ago

I tried r2219 but it's still not working. Seems to be another bug :(

Actually I have quite some pictures where I would be happy if they could be rescued. Next time I will only work on copys with svn-versions, sorry I forgot that...

#15 Updated by Andreas Huggel almost 7 years ago

Here, the result of

$ exiv2-0.19 -M'set Exif.Image.Rating 5' original.nef

and
$ exiv2-r2219 -M'set Exif.Image.Rating 5' original.nef

is exactly the same image. Byte for byte. So if exiv2-0.19 works then the current trunk must work too.
Unless something is wrong with my tests. I'll redo them. Can you also redo yours?

#16 Updated by Frank Hommes almost 7 years ago

Seems to be a mistake on my side :)
Somehow the original picture must have gotten corrupted...

I can confirm now that it works. I will do additional checks later since I can edit the picture but the rating is not shown but it could be the other metadata capture nx is using... We will see :)

#17 Updated by Andreas Huggel almost 7 years ago

Good :) I repeated my tests too, the result is still the same.

#18 Updated by Frank Hommes almost 7 years ago

Would you do that repair tool?

I will take more care in the future with svn-versions...

#19 Updated by Andreas Huggel almost 7 years ago

Yes, I'll give it a try. Just need to find some time.

#20 Updated by Gilles Caulier almost 7 years ago

I'm using the newest exiv2 with digikam 1.3

Just to be sure, go to "Help/Components Info" from digiKam menu to see which version of Exiv2 and libkexiv2 you use with digiKam & co...

Gilles Caulier

#21 Updated by Andreas Huggel almost 7 years ago

Frank,

You can try the following steps to fix your broken images. But I strongly recommend you make a backup of the corrupted images first. (Yes, the corrupted ones :) Essentially, the "repair-tool" is a hacked version of exiv2 which changes the order of the first two sub-directories of an image during an "intrusive" write operation. For that you need to delete or add a tag or modify one in such a way that the new value requires more space than the old one (e.g., by making an Ascii value longer). If you run this on a good image, it will get corrupted. (But you should be able to reverse the effect by running it again.) The tool only works on images with exactly two sub-directories, like the one you provided as a sample. It will throw an error if the image has more than two. In this case, I'll have to study the image. Hopefully there are no such cases.

These are the steps:

  • Get the hacked version of exiv2: In the exiv2 top-directory, issue the command svn switch svn://dev.exiv2.org/svn/branches/repairbug701, then build as usual. The resulting exiv2 binary in src/ is the "repair-tool". Change to the src/ directory. Do not install this exiv2 version or use it with digiKam or any other application...
  • Run something like ./exiv2 -M'del Exif.Image.Rating' corrupted-image.nef. (This is just an example, you can add or delete or modify any tag, just make sure that if you modify a tag, the new value is larger than the old one.) Do this with one or two images first and test, before you run it on all.
  • Once the images are fixed, you should switch back to the trunk by running svn switch svn://dev.exiv2.org/svn/trunk in the top-directory of your exiv2 tree.

Good luck and let me know how it goes...
Andreas

#22 Updated by Frank Hommes almost 7 years ago

Andreas,

thanks very much for taking some time to help me with my corrupted pictures.

I tried your steps but:
freanki@tristan:~/progs/src/trunk/build/src$ ./exiv2 -M"set Exif.Photo.UserComment charset=Ascii This might repair the picture" ~/win/pics/temp/20100426-102416.nef

resulted in a corrupted images which can't be loaded anymore by capture nx. Running it over again, restored the image the way it was before.

I attached the output of: exiv2 -pa 20100426-102416.nef > metadata.txt on this report since I don't know how many directories it has and I hope you can see it there.

Frank

#23 Updated by Andreas Huggel almost 7 years ago

metadata.txt shows that the image is broken - Exif.SubImage1.NewSubfileType says 'Primary image' and that for SubImage2 indicates a thumbnail in the second sub-directory. It should be the other way around.

Could it be that something went wrong in the first step?

Can you try this instead: Check-out the branch somewhere and build the repair tool from scratch, something like this:

$ svn co svn://dev.exiv2.org/svn/branches/repairbug701
$ cd repairbug701
$ make config; ./configure --disable-shared; make -j3
$ cd src
$ ./exiv2 -M....

Andreas

#24 Updated by Frank Hommes almost 7 years ago

Oh thats total right.
The metadata is of the file which was broken in the first place without the fix. I posted the metadata because you said: "The tool only works on images with exactly two sub-directories, like the one you provided as a sample. It will throw an error if the image has more than two. In this case, I'll have to study the image."
So I thought the original might have had two such directories. I attached the metadata of the broken file with the applied fix which won't start in capture nx.

I will try your other idea tomorrow and see if the outcome is the same because I'm in a hurry right now.

Frank

#25 Updated by Frank Hommes almost 7 years ago

yes, it's the same error.

#26 Updated by Andreas Huggel almost 7 years ago

yes, it's the same error.

That's not good and not enough info to say any more. Can you send me a picture which doesn't work and I'll try with that here. Maybe it's different from the image I tested with.

I'll be away this weekend and will only get to look into this next week.

Andreas

#27 Updated by Andreas Huggel almost 7 years ago

  • Category set to tiff parser
  • Status changed from New to Resolved
  • Target version set to 0.20
  • % Done changed from 0 to 100

The "repair tool" is still not working. Will have another look at that after the 0.20 release.

#28 Updated by Andreas Huggel almost 7 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux