Bug #774

"exiv2 -eX" followed by "exiv2 -iX" produces invalid XMP metadata packet

Added by Volker Grabsch over 6 years ago. Updated over 2 years ago.

Status:ClosedStart date:30 May 2011
Priority:NormalDue date:
Assignee:Alan Pater% Done:

100%

Category:xmp
Target version:0.25

Description

For any image "test.jpg" that contains XMP metadata, the following sequence of commands will corrupt the XMP:

exiv2 -eX test.jpg
exiv2 -iX test.jpg

This happens with all image formats in exiv2-trunk as well as exiv2-0.21.1, and older versions. The problem is that these commands make different assumptions about the exact file format of the XMP sidecar file "test.xmp".

The first command "exiv2 -eX" reads the XMP metadata and modifies it before writing to "test.xmp". That is, it removes the beginning and trailing processing instructions, and prepends an processing instruction instead. (It also adds metadata from IPTC to XMP, but this seems to be irrelevant for this issue.)

The second command "exiv2 -iX", however, embeds the "test.xmp" file as is, without restoring the processing instructions. Thus, the "test.jpg" will contain invalid XMP metadata.

In summary, "exiv2 -iX" expects "test.xmp" to be identical to the XMP block in the image file:

<?xpacket begin=...?>
<x:xmpmeta ...>
  ...
</x:xmpmeta>
<?xpacket end="w"?>

while "exiv2 -eX" produces a slightly modified format:

<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta ...>
  ...
</x:xmpmeta>

Shouldn't both commands agree on one file format? This would avoid unpleasant surprises.

Note that the corrupted file is usually still readable with Exiv2, but technically incorrect because of the missing processing instructions. This may lead to problems with other tools, as well as problems with the upcoming EPS support, which will depend on valid processing instructions as specified in the XMP spec.

Currently, the only workaround is to embed the XMP metadata via "exiv2 -ix" instead of "exiv2 -iX". However, it would be much better if the produced XMP sidecar file would be directly usable by "exiv2 -iX".

xmpsidecar.changes (1.04 KB) Alan Pater, 07 Apr 2015 03:00

774.exiv2.eX.xmp (455 Bytes) Alan Pater, 07 Apr 2015 03:00

774.exiv2.pX.xmp (2.42 KB) Alan Pater, 07 Apr 2015 03:00

Associated revisions

Revision 3662
Added by Robin Mills over 2 years ago

#774. Thanks to Alan for resolving this matter and providing the patch.

Revision 3670
Added by Robin Mills over 2 years ago

#774. Fix MSVC compiler warning.

History

#1 Updated by Alan Pater over 2 years ago

Confirmed using exiv2 trunk r3657. The new -pX method of generating an XMP file produces a standard format.

$ exiv2 -eX

<?xml version="1.0" encoding="UTF-8"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">

$ exiv2 -pX
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> 
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">

#2 Updated by Robin Mills over 2 years ago

  • Status changed from New to Assigned
  • Assignee set to Alan Pater
  • Target version set to 0.26

Alan

Volker is a very good engineer and contributed the Exiv2 EPS support. A tough project on which he worked very hard. I think Volker's identified the problem accurately - we are not being consistent on -eX (export XMP) and -iX (import XMP). Clearly this isn't correct. And you and I have been discussing a "roundtrip" option to extract/import.

I'm going to assign this to you for v0.26. It's been broken for 4 years without anybody else mentioning this. So waiting a while longer can't hurt any more! I'm hoping to find somebody to help you with the C++ for the XML stuff, and otherwise, I'll fix the code. You can remind me when I have more time and you have time to verify my fix.

A word of clarification about the -pX option. It doesn't "generate" xmp. It extracts the raw xmp data from the image file.

Robin

#3 Updated by Alan Pater over 2 years ago

Thanks Robin. I stand corrected on -pX not generating the XMP. It was, however, generated by exiv2. Rather then an existing image taken "in the wild", so to speak, I created a clean test image using imagemagick and then wrote a single xmp key to that image using exiv2 -M. So it was writing from that exiv2 command which generated the valid xmp header:

 <?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> 
I found Volker's work on EPS support and see that he is very explicit and clear on valid XMP headers. It looks to me that the invalid xmp header comes from xmpsidecar.cpp. I will test locally with Volker's valid xmp header and see if I can generate a valid xmp sidecar.

I have looked at xmp headers from a variety of images written from a variety applications and they all show variations of Volker's valid xmp headers.

#4 Updated by Alan Pater over 2 years ago

  • File xmpsidecar.changes added
  • File 774.exiv2.xmp added

I changed the xmlHeader in xmpsidecar.cpp to a valid header and my simple test was successful. exiv2 -eX wrote the valid xmp header. Can someone suggest further tests? I would like to be sure that something was not missed and that this change won't have any undesirable effects.

I'll submit a real patch once we are sure that I haven't broken anything.

#5 Updated by Alan Pater over 2 years ago

  • File deleted (xmpsidecar.changes)

#6 Updated by Alan Pater over 2 years ago

  • File deleted (774.exiv2.xmp)

#7 Updated by Alan Pater over 2 years ago

Forget the footer: string.

Apart from whitespace, both xmp files are now the same.

#8 Updated by Robin Mills over 2 years ago

  • Assignee changed from Alan Pater to Robin Mills
  • Target version changed from 0.26 to 0.25

Submitted r3662. Thanks to Alan for resolving this matter and providing the patch.

I'm going to assign this to myself to add a regression detector to test/bugfixes.sh.

#9 Updated by Volker Grabsch over 2 years ago

It's nice to see that old issue finally fixed!

#10 Updated by Robin Mills over 2 years ago

Volker

How nice to hear from you. I hope you are well. I'm very pleased to say that your code is working perfectly. I can't recall anybody reporting any issue with it.

I retired in April 2014 and Alison and I have returned to our 'old' home in England after 14+ years in Silicon Valley. Very happy to be home.

Exiv2 is very active. Too much to do and not enough hands! I've been joined by Alan and he's a very enthusiastic XMP kind of guy. He doesn't do C++, however he most certainly knows things that I don't. He's working through old issues and dealing with stuff that's been ignored. I've still got your extended attributes to fix and although I feel guilty - there's always something that seems more important!

Right. I hope everything's good with you. Regards to Michael.

Robin

#11 Updated by Robin Mills over 2 years ago

  • Status changed from Assigned to Resolved
  • Assignee changed from Robin Mills to Alan Pater

Fix submitted: r3669. I'm going to mark this "resolved" as there is a regression detector already in place. I'm going to assign this to Alan to be sure he gets the credit in the v0.25 release notes.

#12 Updated by Alan Pater over 2 years ago

  • % Done changed from 0 to 100

#13 Updated by Andreas Huggel over 2 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux