Exiv2

Further to my last:

I should say that if I set these tags and then read them back on the command line, with exiv2 or exiftool, they report fine, its just in Darktable and it only complains about not being able to read the white balance information.

I can set the following tags without Darktable complaining:

Exif.Photo.LensModel
Exif.Photo.FocalLength
Exif.Photo.FNumber
Exif.Photo.MaxApertureValue
Exif.Image.Artist
Exif.Image.Copyright

I cannot set any of the following, either individually or together:

Exif.Photo.LensSpecification
Exif.Photo.LensMake
Exif.Photo.LensSerialNumber
Exif.Image.FocalLength
Exif.Image.FNumber
Exif.Image.ApertureValue
Exif.Photo.ApertureValue
Exif.Image.MaxApertureValue

The two that really matter to me are LensSerialNumber and LensSpecification.

Does anyone have any idea as to why I might be able to set some and not others? Should I be able to set these or am I doing something stupid? I seem to be able to set all these values via exiftool command line without any issues.

Any help or suggestions would be gratefully appreciated,

Rob

Rob

Can you attach a file from your camera and I will investigate. I'm currently on vacation and will look at this next week.

There are several matters to consider:
1) There are tag which cannot be changed in the metadata. Most of them relate the high-resolution image such as Resolution. When the function image->IsImage() returns true, exiv2 will (silently) ignore modification to the tag. The tags you have listed (Exif.Image.ApertureValue may be restricted.

2) darktable may impose additional restrictions about which exiv2 knows nothing.

3) I strongly encourage to use the command-line utility exiv2(.exe) which you can download or build from source to eliminate darktable from the mix. I see you're using your own code, and that's "more-or-less" what's done in exiv2(.exe)

4) The function image->IsImage() is implemented for each of the common file formats (JPG, PNG, TIFF etc). As darktable is a raw image processor, I expect you're using RAW images. You'll have to look at the image processor for your raw image as it could be CRW, CR2 or another format.

I hope that helps you for the moment. Please send sample images and I will investigate further when I get home. Our Redmine service restricts images to 20mb and I if you're image are larger, please use a cloud sharing service such as DropBox to send me images.

Robin

Hi Robin,

Firstly, sorry to disturb your vacation.

I strongly suspect that I am doing something wrong in the code as I am using RAW files directly from a Sony A7ii, the metadata from which can be successfully modified with exiftool without upsetting either Darktable or Lightroom.

1) This may be true for some of them, but I can seem to edit them via exiftool CLI.
2) Possibly, but Lightroom also doesn't seem to like the resulting files.
3) I am integrating libexiv2 in to another project and would rather not pipe to the command line.
4) I am indeed using RAW image (Sony ARW, which I beleive are a TIFF variant?). I have tried using the TiffImage class directly, as follows, with no better results:

TiffImage::AutoPtr image=ImageFactory::open(filePath.toStdString());
image->readMetadata();
ExifData exifData=image->exifData();

exifData["Exif.Photo.LensMake"]="Helios";

image->setExifData(exifData);
image->writeMetadata();

I assume I am still doing something wrong.

Thanks very much for your help,

Rob

Rob

I like helping folks. It's often fun! Together, we'll get to the bottom of this. I've used one of our test ARW files and can set Exif.Photo.LensMake with a fairly recent build of the exiv2 application:

549 rmills@rmillsmbp:~/temp $ exiv2 -vV | grep -ie date -ie time
date=Apr 29 2018
time=11:47:38
have_gmtime_r=1
have_timegm=1
xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/
550 rmills@rmillsmbp:~/temp $ curl -O http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9376k  100 9376k    0     0  12.6M      0 --:--:-- --:--:-- --:--:-- 12.7M
551 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep lens/i RAW_SONY_A330.ARW 
Exif.Sony2.LensID                            Long        1  Sony AF DT 55-200mm F4-5.6 SAM (SAL55200-2)
552 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW 
553 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep lens/i RAW_SONY_A330.ARW 
Exif.Sony2.LensID                            Long        1  Sony AF DT 55-200mm F4-5.6 SAM (SAL55200-2)
Exif.Photo.LensMake                          Ascii       7  rmills
554 rmills@rmillsmbp:~/temp $ 

Let's try to focus on libexiv2, your code and files. We can discuss LightRoom, darktable and exiftool later. I have great respect for LightRoom, darktable and exiftool. However, like Pooh Bear, I am a bear of exceeding small brain! Too many simultaneous topics confuse me.

Thank you for your code. Can you provide a small test file, please. Keep the lens cap on and set the camera to its minimum image size. My Nikon D5300 default raw is 6000x4000 (24mb) and also supports a "medium" 4496x3000 (15mb) and "small" 2992x2000 (6mb). Our redmine installation limits uploads to 20mb.

Hi Robin,

$ exiv2 -vV |grep -ie date -ie time
date=Jun 20 2018
time=18:53:17
have_gmtime_r=1
have_timegm=1
xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/

I too can set and read back the tags, either with the command line application as you demonstrate or with my own code; both methods cause Darktable to complain the white balance information can't be read (the were readable before editing) and Lightroom just won't open them. I tried this with the test image you listed above with the same result, so I don't think its just my code. I can only assume (lib)exiv2 is corrupting the headers in some way when certain tags are written?

Thanks very much for your help,

Rob

Are you saying:

1) Download http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW
2) Lightroom and Darktable are happy
3) exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW
4) Lightroom and Darktable are now unhappy

Have I understood your issue?

There are file analysis tools built into exiv2. For example, to dump the structure of the file:

$ exiv2 -pS image

To recursively dump image structures embedded in the file:

$ exiv2 -pR image

I've used those tools as follows:

714 rmills@rmillsmbp:~/temp $ curl -O http://exiv2.dyndns.org:8080/userContent/testfiles/raw/sony/RAW_SONY_A330.ARW
715 rmills@rmillsmbp:~/temp $ exiv2 -pR RAW_SONY_A330.ARW > before.txt
716 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensMake rmills' RAW_SONY_A330.ARW
717 rmills@rmillsmbp:~/temp $ exiv2 -pR RAW_SONY_A330.ARW > after.txt
718 rmills@rmillsmbp:~/temp $ diff before.txt after.txt 

Most of the data in the file has relocated within the file (written at a different offset within the file). However I haven't spotted anything that has been removed or corrupted.

I've performed the same process using the option $ exiv2 -pa image which dumps all metadata. There are differences, however nothing that seems significant. Or, to put it another way, I believe I can explain all the differences. The metadata does not seem to have been disturbed.

I need to know why Lightroom and Darktable are unhappy.

Robin

Hi Robin,

Yes, that is exactly the issue. Darktable consistently complains that it is unable to read the white balance data. As I said in one of previous posts, some tags are fine and some cause the issue. It might, therefore, be worth changing a tag that works fine and one that causes issues and seeing what moves in the file? For example: set 'Exif.Photo.LensModel' (darktable will be happy) and set 'Exif.Photo.LensMake' (darktable will be unhappy) and compare the difference in structure?

Thanks very much for help,

Rob

I've done this and the differences are tiny (and seem correct).

732 rmills@rmillsmbp:~/temp $ exiv2 -M'set Exif.Photo.LensModel rmills' RAW_SONY_A330.ARW
736 rmills@rmillsmbp:~/temp $ exiv2 -pa RAW_SONY_A330.ARW > after2.txt 
737 rmills@rmillsmbp:~/temp $ diff after.txt  after2.txt 
738 rmills@rmillsmbp:~/temp $ 

Have you spoken to the darktable people about this? I'm hunting for something I cannot reproduce without darktable. I'm not familiar with darktable, even if I pull down darktable and reproduce this, I have to understand what is causing darktable to be unhappy.

I will contact the darktable community and see if they have any ideas.

The other thing I will do is set the same tag with exiftool and exiv2 and see what the differences are (if any).

Thanks for you ongoing help.

So, I used your example file 'RAW_SONY_A330.ARW' and ran the following commands:

$ cp RAW_SONY_A330.ARW RAW_SONY_A330_Exiv2.ARW
$ cp RAW_SONY_A330.ARW RAW_SONY_A330_ExifTool.ARW
$ exiv2 -pa RAW_SONY_A330.ARW > before.txt
$ exiv2 -M'set Exif.Photo.LensMake Helios' RAW_SONY_A330_Exiv2.ARW
$ exiv2 -pa RAW_SONY_A330_Exiv2.ARW > exiv2.txt
$ exiftool -LensMake=Helios RAW_SONY_A330_ExifTool.ARW
$ exiv2 -pa RAW_SONY_A330_ExifTool.ARW > exiftool.txt

Both the original and the exiftool modified version open fine in Darktable, the exiv2 version does not. BOth modified version report 'Helios' as the lens make as expected when using either exiv2 CLI or exiftool. There are differences when I run exiv2 -pa on the two modified versions. I attach all three output txt files to see if the differences can help narrow down the problem.

Here are the four differences between exiftool.txt and exiv2.txt

                                                            exiftool.txt   exiv2.txt
Exif.SubImage1.StripOffsets                  Long        1  606984         565768
Exif.Image.JPEGInterchangeFormat             Long        1  76614          35398
Exif.Image.DNGPrivateData                    Byte        4  122 123 0 0    16 142 0 0
Exif.Thumbnail.JPEGInterchangeFormat         Long        1  73016          31800

Exif.SubImage1.StripOffsets
StripOffsets which has to do with the storage of the image pixels. It's an offset in the file to the start of the first row (or something like that). For sure the image looks ok on the Mac Preview application.

Exif.Image.DNGPrivateData
I think this is an offset to private data (written by the manufacturer)
https://www.awaresystems.be/imaging/tiff/tifftags/dngprivatedata.html

JPEGInterchangeFormat.
It's a long (4 bytes). It has something to with the defining how the image has been stored/compressed. https://www.awaresystems.be/imaging/tiff/tifftags/jpeginterchangeformat.html

I feel we have to ask somebody at darktable to look/debug RAW_SONY_A330_Exiv2.ARW and explain why it does not retrieve the WhiteBalance. Perhaps darktable has a log file with this information.