Bug #1064

exiv2 -iX <file.jpg> deletes Makernote

Added by Alan Pater about 2 years ago. Updated 11 months ago.

Status:ClosedStart date:24 Apr 2015
Priority:NormalDue date:
Assignee:Robin Mills% Done:

100%

Category:xmpEstimated time:5.00 hours
Target version:0.26

Description

exiv2 -iX <file.jpg> is used to import metadata from an XMP sidecar file which was typically created with the command: exiv2 -eX <file.jpg>.

If the jpg file contains a makernote, -iX deletes that makernote. This was tested on images with both Nikon makernotes and on images with Olympus makernotes.

The XMP sidecar file created by -eX does not contain any makernote information. The XMP specification does not say anything about storing makernotes in XMP sidecar files, so that behaviour is normal and correct.

Importing metadata from XMP sidecar file should not delete makernotes.

Associated revisions

Revision 4220
Added by Robin Mills over 1 year ago

#1057, #1064, #922, #1148. Work in progress. This is a composite patch of several matters in development. None are totally complete at this time.

Revision 4427
Added by Robin Mills 11 months ago

#1064 Fix -iX to merge (not replace) metadata from source to destination.

History

#1 Updated by Robin Mills about 2 years ago

  • Status changed from New to Assigned
  • Target version set to 0.26

I'm going to push this one to v0.26. If this is a bug, it's been there a long time and nobody has mentioned it. If this a feature request to preserve the Makernote in the image being edited, I want to defer it to 0.26.

I feel that exiv2 -eX should export the Makernote as the attribute exif.Makernote="..base64.." and exiv2 -iX should recreate the Makernote. There are at least three reasons:

  1. It's symmetrical
  2. It's consistent with exiv2 -e{tgt} and exiv2 -i{tgt} which create and read .exv "sidekick files"
  3. It's clean. Preserving the Exif.Makernote when updating other Exif takes is ugly. It might even be difficult to implement.

#2 Updated by Alan Pater about 2 years ago

Lot's of people will be happy if we can figure out a reliable way to store makernotes in XMP. But also, some say no, not possible, don't go there. So not everyone will be happy ...

Phil Harvey, exiftool: "... the MakerNote tag existed in the early Adobe XMP specification, they never defined an encoding method for the data, so the tag is essentially useless .." http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=1878.0

Leonard Rosenthol, Adobe Systems: "I've discussed this the XMP team and they do NOT recommend the inclusion of a "binary blob" of data inside of XMP, and ESPECIALLY NOT the MakerNote data." http://lists.freedesktop.org/archives/create/2007-June/000835.html

Perhaps a -eMX option to export the makernote to a separate "binary blob" file alongside the xmp sidecar file? Then -iMX would re-import them both.

#3 Updated by Robin Mills about 2 years ago

Thanks for digging into this, Alan. We mustn't put the makernote into the XMP. Adobe, undoubtably for very good reasons, have said "No. Don't go there!". We don't need to add an M flag to say "Do a Microsoft and ignore the standard" We already have the .exv file that preserves the makernote. We don't need a sidecar to the sidecar!

So that brings us back to the original topic which is to retain the image's makernote when reading in a sidecar. What happens if the side car contradicts the makernote. We don't have an code to rewrite/update the makernote.

We could compromise and retain the makernote if it doesn't contradict the sidecar. However when there's conflict, we remove the makernote. That's a mess. Somebody will report a bug saying, sometimes the makernote is deleted and sometimes it isn't.

So, how about an m import tgt to say "retain the makernote". We can put out a warning if we think it's conflicted.

We could have import tgt m or M:
m = retain makernote when not conflicted
M = retain makernote unconditionally

#4 Updated by Alan Pater about 2 years ago

Robin Mills wrote:

So that brings us back to the original topic which is to retain the image's makernote when reading in a sidecar. What happens if the side car contradicts the makernote. We don't have an code to rewrite/update the makernote.

I don't think there are any conflicts between information in the
makernote and in XMP, IPTC or Exif. I thought makernotes are completely
separate tags in a completly different location inside the file.

For instance, all the Nikon makernote tags are under Exif.Nikon...

http://www.exiv2.org/tags-nikon.html

All the Exif tags are under Exif.Image or Exif.Photo or Exif.GPSInfo.
There is no overlap.

#5 Updated by Robin Mills about 2 years ago

Oh, the makernote tag is safely stored in the file. If information for the XMP is derived from the MakerNotes, it could be edited in XMP. When we read the XMP back, it could conflict. Example:

725 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa -g eed ~/R.jpg 
Exif.Photo.ISOSpeedRatings                   Short       1  400
Exif.Nikon3.ISOSpeed                         Short       2  400
Exif.Nikon3.CropHiSpeed                      Short       7  0 6016 4016 6016 4016 0 0
726 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 
The XMP looks like this:
727 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -eX -f ~/R.jpg
728 rmills@rmillsmbp:~/gnu/exiv2/trunk $ grep -i -B 2 -A 2 speed ~/R.xmp 
    </rdf:Alt>
   </exif:UserComment>
   <exif:ISOSpeedRatings>
    <rdf:Seq>
     <rdf:li>400</rdf:li>
--
--
     <rdf:li>400</rdf:li>
    </rdf:Seq>
   </exif:ISOSpeedRatings>
   <exif:Flash
    exif:Fired="False" 
731 rmills@rmillsmbp:~/gnu/exiv2/trunk $
So if we modify the 400 to 200 and read it back into the image, we have 2 values for the ISOSpeedRatings. I think a warning is all that required.

#6 Updated by Alan Pater about 2 years ago

Agreed. ISO speed is not something anyone should be modifying after the fact.

#7 Updated by Alan Pater about 2 years ago

Besides, we probably will not be able to edit Xmp.exif.ISOSpeedRatings.

"Editing existing XMP properties in XMP sidecars does not work"
http://dev.exiv2.org/issues/639

#8 Updated by Robin Mills about 2 years ago

  • Assignee deleted (Robin Mills)

#9 Updated by Robin Mills almost 2 years ago

  • Assignee set to Robin Mills

#10 Updated by Robin Mills over 1 year ago

  • % Done changed from 0 to 10
  • Estimated time set to 10.00

I'm adding a SWAG to my outstanding issues for v0.26 to get a more accurate assessment of the status of v0.26. I can't remember anything about this - so I'll guess about a day's work.

#11 Updated by Robin Mills 11 months ago

  • % Done changed from 10 to 30

Here's my view of this issue. -ie replaces the Makernote, -iX does not. It feels like a bug in -iX. However -iX reads from foo.xmp (not foo.exv). The Sidecar (foo.xmp) does not have the makernote. I think the fix for -iX to update the Exif data from the Sidecar and not to replace it. Here's my analysis:

1 Setup the test data:

810 rmills@rmillsmbp:~/gnu/exiv2/trunk $ curl -O --silent http://clanmills.com/Stonehenge.jpg
811 rmills@rmillsmbp:~/gnu/exiv2/trunk $ mv Stonehenge.jpg  S.jpg
812 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pR S.jpg | grep -i makernot
         416 | 0x927c MakerNote                 | UNDEFINED |     3152 |       914 | Nikon.....II*.....9.........0211 ... 
813 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa --grep MakerNote S.jpg                   
Exif.Photo.MakerNote                         Undefined 3152  (Binary value suppressed)
Exif.MakerNote.Offset                        Long        1  914
Exif.MakerNote.ByteOrder                     Ascii       3  II
814 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -ea --verbose --force S.jpg                   
File 1/1: S.jpg
Writing Exif data from S.jpg to ./S.exv
Writing IPTC data from S.jpg to ./S.exv
Writing XMP data from S.jpg to ./S.exv
815 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pR S.jpg | grep -i makernot
         416 | 0x927c MakerNote                 | UNDEFINED |     3152 |       914 | Nikon.....II*.....9.........0211 ...
816 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa --grep MakerNote S.exv 
Exif.Photo.MakerNote                         Undefined 3152  (Binary value suppressed)
Exif.MakerNote.Offset                        Long        1  820
Exif.MakerNote.ByteOrder                     Ascii       3  II
817 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 
2 -ie works
820 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pR S.jpg | grep -i makernote
         416 | 0x927c MakerNote                 | UNDEFINED |     3152 |       914 | Nikon.....II*.....9.........0211 ...
821 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa S.jpg | grep -i makernote
Exif.Photo.MakerNote                         Undefined 3152  (Binary value suppressed)
Exif.MakerNote.Offset                        Long        1  914
Exif.MakerNote.ByteOrder                     Ascii       3  II
822 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 
3 -iX requires S.xmp
829 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -iX --verbose --force S.jpg
File 1/1: S.jpg
./S.xmp: Failed to open the file
830 rmills@rmillsmbp:~/gnu/exiv2/trunk $
4 The man page states
       -i tgt Insert target(s) for the 'insert' action. Possible  targets  are
          the same as those for the -d option, plus an optional modifier:

          X  :  Insert  metadata  from an XMP sidecar file <file>.xmp. The
          remaining insert targets determine what metadata to insert  from
          the  sidecar  file.  Possible  are  Exif,  IPTC  and XMP and the
          default is all of these. Note that the inserted  XMP  properties
          include those converted to Exif and IPTC.
5 Create S.xmp with -eX

This is xmp and does not contain the makernote (as explained above).

831 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -eX --verbose --force S.jpg
File 1/1: S.jpg
Writing Exif data from S.jpg to ./S.xmp
Writing XMP data from S.jpg to ./S.xmp
832 rmills@rmillsmbp:~/gnu/exiv2/trunk $ grep -i maker S.xmp
833 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 
6 -iX destroys the makernote
833 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -iX --verbose --force S.jpg
File 1/1: S.jpg
Writing Exif data from ./S.xmp to S.jpg
Writing IPTC data from ./S.xmp to S.jpg
Writing XMP data from ./S.xmp to S.jpg
834 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa S.jpg | grep -i makernote
835 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pR S.jpg | grep -i makernote
835 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 
7 Conclusion

I think -iX should "merge" the metadata from S.xmp into S.jpg. It currently replaces it. I'm going to investigate how to achieve that.

If a user wishes to retain the existing behaviour, it can be easily achieved by deleting the Exif metadata with -de before using -iX. For example:

$ exiv2 -de -iX S.jpg

#12 Updated by Robin Mills 11 months ago

  • Status changed from Assigned to Closed
  • % Done changed from 30 to 100
  • Estimated time changed from 10.00 to 5.00

Fix submitted: r4427

Brilliant analysis, Robin. The code in src/actions.cpp+1993 did not handle the preserve flag correctly.

    int metacopy(const std::string& source,
                 const std::string& tgt,
                 int targetType,
                 bool preserve);

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux