Exiv2

I just try to copy exif data from original file to unopenable one with exiftool with this command line :

exiftool -exif:all= -tagsfromfile nb-photos-20090715-142043.nef -exif:all nb-photos-20090715-142043K.nef

And it appears that unopenable NEF became openable again with CaptureNX and apparently keep other metadatas.

So it seems it's may be an exif matters, I hope it's help you.

Nicolas

It seems the solution above doesn't work for NEF which was already modified by CaptureNX and after metadata edited by exiv2.

So I hope a solution could be found.

Thanks

Nicolas

The corrupt picture has an incomplete makernote. In particular, there is no whitebalance info in the makernote anymore. The image data seems to be intact. Exiv2 typically doesn't discard this information without being instructed to do so.

I'm also not clear about what exactly was done to the sample images attached. Even the first one, nb-photos-20090715-142043.nef, looks like it has been edited: It contains IPTC data, a user comment and GPS coordinates. The second one looks like it was edited not only with digiKam but also with ViewNX: it has a Exif.Image.Software tag that contains "ViewNX 1.5 W" and a Exif.Image.ProcessingSoftware tag with "digiKam-1.0.0-beta5".

Can you provide a sample right before editing it with digiKam and the edited image right after saving it with digiKam. We need to be sure that all the differences in these pictures are caused by digiKam.

Then we would at least have narrowed the possible culprits to exiv2 and digiKam+Co. To prove that this is really an exiv2 bug, it is best if the corruption can be reproduced with just exiv2 alone, e.g., like in the description of bug #636.

Andreas

Yes,

Sylvain Crouzillat notice the same thing yesterday on the digikam userlist. I'm apologize for that (I should verify before uploading finally), may be for this picture I made too many test and lost makernote (I don't know how), but I try the solution for #636 with other files unsuccesly.

I'm uploading on my web filestorage three files, the RAW from the memory card, one with five star rating applying by Digikam, one other five star rating applying by ViewNx.

I have installed evix2 from SVN, and it seems to be take without rebuilding Digikam as XMP core version is 4.4.0 and was before 4.1.0, but the digikam version still unopenable for me with capture NX.

I'm a little puzzled because Sylvain Crouzillat use the mostly same version of digikam and exiv2 0.18.2 on Mandriva 10 and he doesn't have this matter.

Here are the 3 files URL :

http://photonoxx.free.fr/digikamNX2/DSC_2854.NEF
http://photonoxx.free.fr/digikamNX2/DSC_2854digikam.NEF
http://photonoxx.free.fr/digikamNX2/DSC_2854viewnx.NEF

I hope this time it was better for you, and sorry again for the previous file. In fact it come from an already edited picture edited with digikam on Ubuntu 9.04 (openable with captureNX), and more recently geotagged under 9.10 (and became unopenable), and I effectively try to tag it under viewNX thinking may be it'll overwrite metadata change (but no).

Thanks

Nicolas

Nicolas Boulesteix wrote:

Yes,

Sylvain Crouzillat notice the same thing yesterday on the digikam userlist. I'm apologize for that (I should verify before uploading finally), may be for this picture I made too many test and lost makernote (I don't know how), but I try the solution for #636 with other files unsuccesly.

I'm uploading on my web filestorage three files, the RAW from the memory card, one with five star rating applying by Digikam, one other five star rating applying by ViewNx.

I have installed evix2 from SVN, and it seems to be take without rebuilding Digikam as XMP core version is 4.4.0 and was before 4.1.0, but the digikam version still unopenable for me with capture NX.

I'm a little puzzled because Sylvain Crouzillat use the mostly same version of digikam and exiv2 0.18.2 on Mandriva 10 and he doesn't have this matter.

Here are the 3 files URL :

http://photonoxx.free.fr/digikamNX2/DSC_2854.NEF
http://photonoxx.free.fr/digikamNX2/DSC_2854digikam.NEF
http://photonoxx.free.fr/digikamNX2/DSC_2854viewnx.NEF

I hope this time it was better for you, and sorry again for the previous file. In fact it come from an already edited picture edited with digikam on Ubuntu 9.04 (openable with captureNX), and more recently geotagged under 9.10 (and became unopenable), and I effectively try to tag it under viewNX thinking may be it'll overwrite metadata change (but no).

Thanks

Nicolas

An important pointer there, is core of libkexiv2 interface for digiKam, used to update metadata using Exiv2. Look code from this method :

http://lxr.kde.org/source/KDE/kdegraphics/libs/libkexiv2/libkexiv2/kexiv2.cpp#397

... perhaps something is wrong there...

Note : I recommend to try with libkexiv2 from trunk (1.0.0). digiKam & co need to be recompiled in this case...

Gilles Caulier

Hi,

I have may be something new !

I have build and instal Exiv2, Kdegraphics libs, Digikam and Kipiplugins from SVN. The first test give the same matter, but continuing tests, I notice the MakerNotes:colorhue is missing from the edited NEF.

In the original file, this tag is set with the value "MODE3" (case sensitive), and if I set with exiftool an other value like "mode3" or anything else than MODE, the file can't be opened by CaptureNX, if i set back MODE value, CaptureNX succed to open it !!

I try to add this tag to a digikam edited Nef file, but exiftool seems to don't wan't to create this particular, but if I'm copying it from the original file, I succeed to open it again in CaptureNX.

Now, I have no idea from where come this matter, and I have to verify on other file if it's working or if if it works only for this particular file.

Kind regards

Nicolas

Ok, I ignore the original sample images and concentrate on the new ones.

Nicolas, you're right, the Exif.Nikon3.ColorMode tag is missing from the makernote of the digiKam-modified image, and this is the only relevant difference in the two makernotes.

I try to add this tag to a digikam edited Nef file, but exiftool seems to don't wan't to create this particular

Try this:

exiv2 -M'set Exif.Nikon3.ColorMode MODE3' DSC_2854digikam.NEF

If I understand correctly, you're saying with this tag added it works again? I can't verify this here because I don't have the Nikon software that you use and ufraw is able to display the image with and without that tag.

Gilles, looking at the pointer to libkexiv2 source, this is not the place where this tag goes missing AFAICS. It rather seems that d->exifMetadata() doesn't contain this tag anymore in this function.

Andreas

Correction: The original image, DSC_2854.NEF, has two Exif.Nikon3.ColorMode tags, the one edited by digiKam has only one such tag left:

$ exiv2 -pa DSC_2854.NEF | grep ColorMode
Exif.Nikon3.ColorMode                        Ascii       6  COLOR
Exif.Nikon3.ColorMode                        Ascii       9  MODE3   
$ exiv2 -pa DSC_2854digikam.NEF | grep ColorMode
Exif.Nikon3.ColorMode                        Ascii       6  COLOR

So Nicolas, instead of the command I gave earlier, use this to test if you can fix the edited images:

exiv2 -M'add Exif.Nikon3.ColorMode MODE3' DSC_2854digikam.NEF

Ok, Thanks I'n going to test that but something is slightly strange.

With the tag ouptput of exiftool, I don't have two tags named ColorMode on a right file but only one with the value "COLOR" and another tag named ColorHue with the value "MODE3".

here an example output for the makernotes group for an openable picture

[MakerNotes] MakerNoteVersion: 2.10
[MakerNotes] ColorMode: Color
[MakerNotes] Quality: Raw
[MakerNotes] WhiteBalance: Auto
[MakerNotes] FocusMode: AF-C
[MakerNotes] FlashSetting: Normal
[MakerNotes] FlashType: 
[MakerNotes] WhiteBalanceFineTune: 0
[MakerNotes] WB_RBLevels: 1.5078125 1.6015625 1 1
[MakerNotes] ProgramShift: 0
[MakerNotes] ExposureDifference: 0
[MakerNotes] PreviewImageStart: 13130
[MakerNotes] PreviewImageLength: 110790
[MakerNotes] FlashExposureComp: 0
[MakerNotes] ISOSetting: 100
[MakerNotes] FlashExposureBracketValue: 0.0
[MakerNotes] ExposureBracketValue: 0
[MakerNotes] CropHiSpeed: Off (3904x2616 cropped to 3904x2616 at pixel 0,0)
[MakerNotes] SerialNumber: 
[MakerNotes] ColorSpace: sRGB
[MakerNotes] ToneComp: Auto
[MakerNotes] LensType: G
[MakerNotes] Lens: 17-70mm f/2.8-4.5
[MakerNotes] FlashMode: Did Not Fire
[MakerNotes] AFAreaMode: Dynamic Area
[MakerNotes] AFPoint: Mid-right
[MakerNotes] AFPointsInFocus: Mid-right
[MakerNotes] ShootingMode: Single-Frame
[MakerNotes] AutoBracketRelease: Manual Release
[MakerNotes] ContrastCurve: (Binary data 4160 bytes, use -b option to extract)
[MakerNotes] ColorHue: Mode1
[MakerNotes] LightSource: Natural
[MakerNotes] ShotInfoVersion: 0207
[MakerNotes] VibrationReduction: Off
[MakerNotes] HueAdjustment: 0
[MakerNotes] NEFCompression: Lossy (type 1)
[MakerNotes] NoiseReduction: Off
[MakerNotes] LinearizationTable: (Binary data 1412 bytes, use -b option to extract)
[MakerNotes] WB_RGGBLevels: 386 256 256 410
[MakerNotes] LensDataVersion: 0201
[MakerNotes] ExitPupilPosition: 85.3 mm
[MakerNotes] AFAperture: 4.0
[MakerNotes] FocusPosition: 0x11
[MakerNotes] FocusDistance: 0.63 m
[MakerNotes] LensIDNumber: 127
[MakerNotes] LensFStops: 6.00
[MakerNotes] MinFocalLength: 17.3 mm
[MakerNotes] MaxFocalLength: 71.3 mm
[MakerNotes] MaxApertureAtMinFocal: 2.8
[MakerNotes] MaxApertureAtMaxFocal: 4.5
[MakerNotes] MCUVersion: 28
[MakerNotes] EffectiveMaxAperture: 4.0
[MakerNotes] RawImageCenter: 1952 1308
[MakerNotes] SensorPixelSize: 6.05 x 6.05 um
[MakerNotes] ImageCount: 100
[MakerNotes] DeletedImageCount: 0
[MakerNotes] ShutterCount: 100
[MakerNotes] FlashInfoVersion: 0101
[MakerNotes] ExternalFlashFirmware: n/a
[MakerNotes] ExternalFlashFlags: (none)
[MakerNotes] FlashCommanderMode: Off
[MakerNotes] FlashControlMode: Off
[MakerNotes] FlashGroupAControlMode: Off
[MakerNotes] FlashGroupBControlMode: Off
[MakerNotes] FlashGroupAExposureComp: 0
[MakerNotes] FlashGroupBExposureComp: 0
[MakerNotes] ImageOptimization: Normal
[MakerNotes] MultiExposureVersion: 0100
[MakerNotes] MultiExposureMode: Off
[MakerNotes] MultiExposureShots: 0
[MakerNotes] MultiExposureAutoGain: Off
[MakerNotes] HighISONoiseReduction: Off

I test your command but it's don't work. In fact, Exiftool output only one value for colorMode tag and exiv2 output the two values, but exiftool don't output the missing ColorHue tag (on exiftool web site it's visible in Nikon tag and it tag_id is 0x008d and the colorMode tag have 0x0003 ID.

I try to change your command and replace ColorMode by ColorHue but it's not recognized.

Nicolas

Andreas Huggel wrote:

Correction: The original image, DSC_2854.NEF, has two Exif.Nikon3.ColorMode tags, the one edited by digiKam has only one such tag left:

[...]

So Nicolas, instead of the command I gave earlier, use this to test if you can fix the edited images:
[...]
(just add instead of set the tag)

Looks like my second correction made it just 2min earlier :) Have a look at my post above your last one...

Andreas

Andreas Huggel wrote:

Looks like my second correction made it just 2min earlier :) Have a look at my post above your last one...

Andreas

:)))

Yes indeed, It mays work, but actually I'm at office with no linux avaible :/

On the othetr hand, if you upload here a modified version of DSC_2854digikam.nef, I can test if it's openable with CaptureNX (which I have installed on my workstation).

I'll test all the rest this evening as quickly I'll can do.

Have I just to apply the patch or have I to rebuild Digikam too ?

For additional informations, Sylvain Crouzillat give me a method to rescue a part of my corrupted files, but after testing some case are not resolved (pictures which have been edited by CaptureNX with new work version inside and edited after by Digikam for metadata).

The Sylvain's method consist of adding a modification which set a new ColorHue tag, but it's not visible on the makernotes (it's made in an up-level, I don't know if you know, but CaptureNX embed any modification made on a picture on the same file, and you can have many different version of picture treatment in the file, and in this case it's the same thing, CaptureNX create a new version with MODE3 setting and it's why it's don't work with already "versioned" files)... So I'm hopeful of this patch.

Thanks

Amicably, Nicolas

Andreas Huggel wrote:

On the othetr hand, if you upload here a modified version of DSC_2854digikam.nef, I can test if it's openable with CaptureNX (which I have installed on my workstation).

File is attached. It was created like this:

[...]

Can you open it with CaptureNX?

Have I just to apply the patch or have I to rebuild Digikam too ?

Just apply the patch and install (replacing the existing shared library). Of course that only works if libkexiv2 is dynamically linked with exiv2.

Good luck :)
Andreas

Thanks for the file, but finally I have downloaded the Windows version of exiv2... I don't think about it before oops :>/

I just replace >'< by >"< in the command line (probably due to windows format ?) and I try to open the resulting file in CaptureNX and.... it's work !!

I'll make more test this evening, particularly for kind of pictures which didn't work with the Sylvain's method.

For the patch, do I just to make it executable and launch it ?

Thanks again,

Hopefully, Nicolas

Have I just to apply the patch or have I to rebuild Digikam too ?

Just apply the patch and install (replacing the existing shared library). Of course that only works if libkexiv2 is dynamically linked with exiv2.

For the patch, do I just to make it executable and launch it ?

You need to apply the patch (or change that one line of code manually), re-compile the exiv2 library and re-install it.
Then digiKam, or rather libkexiv2, should load the new version when it starts.

Now that this looks like a way to fix the broken images, I'm very interested if the patch fixes the problem in digiKam, i.e., prevents digiKam from deleting the ColorHue tag in the first place.

Andreas

Andreas Huggel wrote:

Have I just to apply the patch or have I to rebuild Digikam too ?

Just apply the patch and install (replacing the existing shared library). Of course that only works if libkexiv2 is dynamically linked with exiv2.

For the patch, do I just to make it executable and launch it ?

You need to apply the patch (or change that one line of code manually), re-compile the exiv2 library and re-install it.
Then digiKam, or rather libkexiv2, should load the new version when it starts.

Now that this looks like a way to fix the broken images, I'm very interested if the patch fixes the problem in digiKam, i.e., prevents digiKam from deleting the ColorHue tag in the first place.

Andreas

May be you'll begin to think I lost my neurons but where have I to apply the patch ? Whole system or in svn path of exiv2 ? Excuse me, but I never do this before, so I will appreciate a simple "walthrough" lol.

yes this looks like a way to fix broken images :)))

I'm nearly in the mood for being sick just to go back home and test this (but there's non hope for that ;))

Nicolas

No problem :)

The patch needs to be applied to the Exiv2 source code. Since you said you compiled Exiv2 from SVN and digiKam loaded that version, just use that again.

Copy the patch file from the comment above to the Exiv2 src/ directory and change to the src/ directory. Then run these commands (or something similar, if I screwed up):

$ patch -p0 < bug657.patch
$ cd ..
$ make; make install

That's all. If you can see a ColorHue tag somewhere in digiKam now, it is using the modified version. If the patch command fails for some reason, just look at the bug657.patch file, open nikonmn.cpp in an editor and make the small change manually.

Andreas

I've checked-in the change with r1931. So instead of using the patch command above, you can simply run svn update.

The first test are all good, Digikam show the ColorHue tag info if it exists, and I try to edit metadata with digikam on an uncorrupted pictures and CaptureNX open it after being edited.

I take a basic photo (not previously modified by CaptureNX before corrupted) and I apply the command line to add the ColorHue tag, and I open it with CaptureNX.

The dark thing in the good news, is, apparently, the photo already modified by CaptureNX before being corrupted don't benefit of the command line, with a little nuance.

If the modify involves Camera parameters, rescue seems workinh.
If the modify involves more advanced features, like BW conversion, U-point, etc... for the first files I try to patch, restoration seems to fail.

The good news is, I test to modify a NEF file with CaptureNX advanced features and after edit metadata with digikam, and the file still openable.

So the last matter which stay open for me (except what I don't test for the moment) is to rescue this kind of RAW.

I continue the tests

Thanks for this patch

Nicolas

May be i can precise a little... I don't make expansive test, but the difficulty to restore a photo can affect only CaptureNX 1 modified pictures... or may be the matter come from earlier issue I didn't yet notice.

Good. So this little patch fixes the original issue and we have a way to fix images corrupted by digiKam due to this issue.

If I understand correctly, the remaining problem is that you haven't found a way yet to fix other corrupted images. But it is not clear how these images got corrupted in the first place. As such this is not an Exiv2 bug at this stage.

In order to keep things simple, I'd like to close this bug now. For the remaining problem, please provide a corrupted image and try to find out the steps to corrupt it. I'll have a look at the image to see if it can be fixed. Please open another bug for this discussion or post to the forum (and once we understand where the corruption came from, we can still open a new bug if necessary).

Let me know if you're ok with that, or if I misunderstood something.

Andreas

Yes, i'm perfectly ok with that thanks !

Just a last answer, now to fix pictures I think the best command line is the one with "set" argument and not "add", but is there an option to prevent overwrite in case where the tag already exists ? And can I use now "ColorHue" instead of 0x008d in the command ?

Thanks

Nicolas

Nicolas Boulesteix wrote:

May be i can precise a little... I don't make expansive test, but the difficulty to restore a photo can affect only CaptureNX 1 modified pictures... or may be the matter come from earlier issue I didn't yet notice.

To conclude, I'm fairly sure the still corrupted files was in fact corrupted by #636 issues and escaped to notice up to now.

Applying both command line (ColorHue and ByteOrder) seems to fix the files.

Nicolas