White balance issues being corrupted or removed
I've written a little C++ program to add lens information to my (predominantly Sony) raw files when I shoot with classic/manual lenses. The issue I am having is that, when I write the modified EXIF data back to the raw file, darktable complains that it is unable to read the white balance information from the image. An example of what I am doing is below:
Image::AutoPtr image=ImageFactory::open(filePath.toStdString()); if(image.get()==0) return false; image->readMetadata(); ExifData exifData=image->exifData(); exifData["Exif.Photo.LensMake"]="Helios"; image->setExifData(exifData); image->writeMetadata();
I'm writing a lot more tags than than that, but I was wondering if my basic approach is flawed. Am I missing or doing something wrong?
All help gratefully appreciated,
Further to my last:
I should say that if I set these tags and then read them back on the command line, with exiv2 or exiftool, they report fine, its just in Darktable and it only complains about not being able to read the white balance information.
I can set the following tags without Darktable complaining:
I cannot set any of the following, either individually or together:
The two that really matter to me are LensSerialNumber and LensSpecification.
Does anyone have any idea as to why I might be able to set some and not others? Should I be able to set these or am I doing something stupid? I seem to be able to set all these values via exiftool command line without any issues.
Any help or suggestions would be gratefully appreciated,
Can you attach a file from your camera and I will investigate. I'm currently on vacation and will look at this next week.
There are several matters to consider:
1) There are tag which cannot be changed in the metadata. Most of them relate the high-resolution image such as Resolution. When the function image->IsImage() returns true, exiv2 will (silently) ignore modification to the tag. The tags you have listed (Exif.Image.ApertureValue may be restricted.
2) darktable may impose additional restrictions about which exiv2 knows nothing.
3) I strongly encourage to use the command-line utility exiv2(.exe) which you can download or build from source to eliminate darktable from the mix. I see you're using your own code, and that's "more-or-less" what's done in exiv2(.exe)
4) The function image->IsImage() is implemented for each of the common file formats (JPG, PNG, TIFF etc). As darktable is a raw image processor, I expect you're using RAW images. You'll have to look at the image processor for your raw image as it could be CRW, CR2 or another format.
I hope that helps you for the moment. Please send sample images and I will investigate further when I get home. Our Redmine service restricts images to 20mb and I if you're image are larger, please use a cloud sharing service such as DropBox to send me images.
Firstly, sorry to disturb your vacation.
I strongly suspect that I am doing something wrong in the code as I am using RAW files directly from a Sony A7ii, the metadata from which can be successfully modified with exiftool without upsetting either Darktable or Lightroom.
1) This may be true for some of them, but I can seem to edit them via exiftool CLI.
2) Possibly, but Lightroom also doesn't seem to like the resulting files.
3) I am integrating libexiv2 in to another project and would rather not pipe to the command line.
4) I am indeed using RAW image (Sony ARW, which I beleive are a TIFF variant?). I have tried using the TiffImage class directly, as follows, with no better results:
TiffImage::AutoPtr image=ImageFactory::open(filePath.toStdString()); image->readMetadata(); ExifData exifData=image->exifData(); exifData["Exif.Photo.LensMake"]="Helios"; image->setExifData(exifData); image->writeMetadata();
I assume I am still doing something wrong.
Thanks very much for your help,
I like helping folks. It's often fun! Together, we'll get to the bottom of this. I've used one of our test ARW files and can set Exif.Photo.LensMake with a fairly recent build of the exiv2 application:
549 rmills@rmillsmbp:~/temp $ exiv2 -vV | grep -ie date -ie time date=Apr 29 2018 time=11:47:38 have_gmtime_r=1 have_timegm=1 xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/ 550 rmills@rmillsmbp:~/temp $ curl -O http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9376k 100 9376k 0 0 12.6M 0 --:--:-- --:--:-- --:--:-- 12.7M 551 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep lens/i RAW_SONY_A330.ARW Exif.Sony2.LensID Long 1 Sony AF DT 55-200mm F4-5.6 SAM (SAL55200-2) 552 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW 553 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep lens/i RAW_SONY_A330.ARW Exif.Sony2.LensID Long 1 Sony AF DT 55-200mm F4-5.6 SAM (SAL55200-2) Exif.Photo.LensMake Ascii 7 rmills 554 rmills@rmillsmbp:~/temp $Let's try to focus on libexiv2, your code and files. We can discuss LightRoom, darktable and exiftool later. I have great respect for LightRoom, darktable and exiftool. However, like Pooh Bear, I am a bear of exceeding small brain! Too many simultaneous topics confuse me.
Thank you for your code. Can you provide a small test file, please. Keep the lens cap on and set the camera to its minimum image size. My Nikon D5300 default raw is 6000x4000 (24mb) and also supports a "medium" 4496x3000 (15mb) and "small" 2992x2000 (6mb). Our redmine installation limits uploads to 20mb.
$ exiv2 -vV |grep -ie date -ie time date=Jun 20 2018 time=18:53:17 have_gmtime_r=1 have_timegm=1 xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/
I too can set and read back the tags, either with the command line application as you demonstrate or with my own code; both methods cause Darktable to complain the white balance information can't be read (the were readable before editing) and Lightroom just won't open them. I tried this with the test image you listed above with the same result, so I don't think its just my code. I can only assume (lib)exiv2 is corrupting the headers in some way when certain tags are written?
Thanks very much for your help,
Are you saying:
1) Download http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW
2) Lightroom and Darktable are happy
3) exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW
4) Lightroom and Darktable are now unhappy
Have I understood your issue?
There are file analysis tools built into exiv2. For example, to dump the structure of the file:
$ exiv2 -pS imageTo recursively dump image structures embedded in the file:
$ exiv2 -pR imageI've used those tools as follows:
714 rmills@rmillsmbp:~/temp $ curl -O http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW 715 rmills@rmillsmbp:~/temp $ exiv2 -pR RAW_SONY_A330.ARW > before.txt 716 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW 717 rmills@rmillsmbp:~/temp $ exiv2 -pR RAW_SONY_A330.ARW > after.txt 718 rmills@rmillsmbp:~/temp $ diff before.txt after.txt
Most of the data in the file has relocated within the file (written at a different offset within the file). However I haven't spotted anything that has been removed or corrupted.
I've performed the same process using the option
$ exiv2 -pa image which dumps all metadata. There are differences, however nothing that seems significant. Or, to put it another way, I believe I can explain all the differences. The metadata does not seem to have been disturbed.
I need to know why Lightroom and Darktable are unhappy.
Yes, that is exactly the issue. Darktable consistently complains that it is unable to read the white balance data. As I said in one of previous posts, some tags are fine and some cause the issue. It might, therefore, be worth changing a tag that works fine and one that causes issues and seeing what moves in the file? For example: set 'Exif.Photo.LensModel' (darktable will be happy) and set 'Exif.Photo.LensMake' (darktable will be unhappy) and compare the difference in structure?
Thanks very much for help,
I've done this and the differences are tiny (and seem correct).
732 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensModel rmills' RAW_SONY_A330.ARW 736 rmills@rmillsmbp:~/temp $ exiv2 -pa RAW_SONY_A330.ARW > after2.txt 737 rmills@rmillsmbp:~/temp $ diff after.txt after2.txt 738 rmills@rmillsmbp:~/temp $Have you spoken to the darktable people about this? I'm hunting for something I cannot reproduce without darktable. I'm not familiar with darktable, even if I pull down darktable and reproduce this, I have to understand what is causing darktable to be unhappy.
I will contact the darktable community and see if they have any ideas.
The other thing I will do is set the same tag with exiftool and exiv2 and see what the differences are (if any).
Thanks for you ongoing help.
So, I used your example file 'RAW_SONY_A330.ARW' and ran the following commands:
$ cp RAW_SONY_A330.ARW RAW_SONY_A330_Exiv2.ARW $ cp RAW_SONY_A330.ARW RAW_SONY_A330_ExifTool.ARW $ exiv2 -pa RAW_SONY_A330.ARW > before.txt $ exiv2 -M'set Exif.Photo.LensMake Helios' RAW_SONY_A330_Exiv2.ARW $ exiv2 -pa RAW_SONY_A330_Exiv2.ARW > exiv2.txt $ exiftool -LensMake=Helios RAW_SONY_A330_ExifTool.ARW $ exiv2 -pa RAW_SONY_A330_ExifTool.ARW > exiftool.txt
Both the original and the exiftool modified version open fine in Darktable, the exiv2 version does not. BOth modified version report 'Helios' as the lens make as expected when using either exiv2 CLI or exiftool. There are differences when I run exiv2 -pa on the two modified versions. I attach all three output txt files to see if the differences can help narrow down the problem.
Here are the four differences between exiftool.txt and exiv2.txt
exiftool.txt exiv2.txt Exif.SubImage1.StripOffsets Long 1 606984 565768 Exif.Image.JPEGInterchangeFormat Long 1 76614 35398 Exif.Image.DNGPrivateData Byte 4 122 123 0 0 16 142 0 0 Exif.Thumbnail.JPEGInterchangeFormat Long 1 73016 31800
StripOffsets which has to do with the storage of the image pixels. It's an offset in the file to the start of the first row (or something like that). For sure the image looks ok on the Mac Preview application.
I think this is an offset to private data (written by the manufacturer)
It's a long (4 bytes). It has something to with the defining how the image has been stored/compressed. https://www.awaresystems.be/imaging/tiff/tifftags/jpeginterchangeformat.html
I feel we have to ask somebody at darktable to look/debug RAW_SONY_A330_Exiv2.ARW and explain why it does not retrieve the WhiteBalance. Perhaps darktable has a log file with this information.