Exiv2

I updated to master, moved the xmp toolkit files to their original exiv2 place and adapted the cmake files. Please checkout branch xmpsdk_next.

Jens

There are lots of 'XMP Toolkit error 8: Unimplemented method XMPMeta::DeleteNamespace' from the XMP related tests now.

The first XMP-SDK version that we used, version 4.1.1, didn't implement this function. It raised an exception instead. XMP-SDK version 4.4.0 provided an implementation and with r2374 I (partially) updated Exiv2 to actually use it (in XmpParser::registerNs in xmp.cpp but apparently forgot about XmpParser::unregisterNs). In XMP-SDK version 5.1.2, the implementation disappeared again and was replaced with the earlier exception.

I suggest to use the implementation from version 4.4.0 and see what happens. If we remove the call from Exiv2, the issue about the 'migration of namespaces' will probably reappear (see the latest test case added to test/bugfixes-test.sh a few days ago).

Andreas

I disagree here, I suggest to remove SXMPMeta::DeleteNamespace() call.

In short: This will fix Bug #751 as this bug is the result of 'migration of namespaces'-patch and reintroduces forum topic 687 - but this is not a bug.

Long version:

Bug #751 is caused by removing/reinserting namespaces for 'XmpParser::registerNs()' function. For the changed prefix 'xap' to xmp' this results in cyclic delete/register namespace calls which are not thread safe in xmpsdk 4.4.2. If we remove the SXMPMeta::DeleteNamespace() call we did not loose anything. Files using the old 'xap' prefix are correctly parsed and every save does use the new prefix 'xmp'.

Forum topic 687 complains about changed prefixes like 'whatevertag_1_'. This is caused only if the suggested prefix 'whatevertag' is already used by some other namespace. XMPProgrammersGuide.pdf, section "Working with namespaces" makes it clear: "It is very important to understand that prefixes are local and scoped in the XML. The prefix is only a means to look up the URI; the prefix itself is not considered in name comparison. Software MUST NEVER depend on the use of a specific prefix in stored XML." So the prefix does not matter, it is only a synonym for the namespace. If you want your humable good looking prefix in the xmp, you have to register the namespace before parsing files with a 'SXMPMeta::RegisterNamespace()' functioncall.

Regards, Jens

The XMP specification is less restrictive than the XMP-SDK. Essentially all it says about prefixes is: "The namespace prefix used in written XML—and, as a consequence, in XMP—serves only as a key to look up the appropriate URI."

In Exiv2, the user should be in control of prefixes and I want to ensure that the prefix seen after reading an XMP property from a file is the same that was used when the property was written. Since prefixes are only "keys to look up URIs" and are written to the serialized XMP packet, that really shouldn't be an issue.

The root cause for the restrictions in the XMP-SDK is that it uses a global registry (or two) for prefixes and namespace URIs. Instead, it should use a registry that is local to the parsed object, so that if two images are parsed at the same time in two threads, their prefix/URI pairs remain independent.

When Adobe implemented the XMPMeta::DeleteNamespace method in version 4.4.0 they were aware that it doesn't solve the real issue as well as the proper solution, as a comment above the function was also added that says:

// *** We would be better off not having this. Instead, have local namespaces from parsing be
// *** restricted to the object that introduced them.

In version 5.1.2, it seems that instead of fixing the problem accordingly they removed the implementation of that function again and decided to simply describe the design restriction better.
At least the comment is still there.

So still, for Exiv2 to move forward, I suggest to use the old implementation of XMPMeta::DeleteNamespace to see if the tests pass with that. Then we'd have upgraded XMP-SDK to 5.1.2 and be ready to decide how to fix the remaining issues, which seem to be pretty much the same as before in this area.

Andreas

• You made the 'migration of namespace'-patch to work arround forum topic 687 - I probably would have done well that time.
• You introduced Bug #751 with the changes of the 'migration of namespace'-patch
• Adobe xmpsdk only has a global namespace registry - this could be a issue
• Adobe decided to implement a SXMPMeta::DeleteNamespace() to the global registry in version 4.4, knowing that this would be a hack as the comment says
• Adobe decided to remove the SXMPMeta::DeleteNamespace() function in version 5.1.2 again, because this hack introduces further bugs, at least:
  • You are able to redefine the xml, xmp and rdf prefixes, which is forbidden
  • You encounter theadings issues like Bug #751
• In version 5.1.2 Adobe moved the global namespace registry to a private class without delete modifier to prevent you from patching with the implementation of version 4.4 - shure one can work also against that
• Namespaces and prefixes in general are not new to the xmp-specification or xmpsdk, they are part of xml - http://www.w3.org/TR/2006/REC-xml-names-20060816/#NT-PrefixedName says "Note that the prefix functions only as a placeholder for a namespace name. Applications SHOULD use the namespace name, not the prefix, in constructing names whose scope extends beyond the containing document."
• It makes no sense to take control of the prefixes in the xmp packet, as these are only placeholders for the namespace and there are other standard conform applications whoose may use other prefixes in case of conflicts

So you could work arround forum topic 687 by providing a delete namespace function, but you introduce further bugs with that. On the other hand you could revert the 'migration of namespace'-patch and fix Bug #751 with that.

You are the maintainer of the exiv2-project and I do not force you about things to do. Is it your decision how to handle things like file hirarchy or making implementations. I do not want to be incooperative, I just want a working solution. Perhaps there is some plausible reason to do the first, but until that I'm not willing to spend my spare time on implementing NEW issues, support them and making work arrounds to them later.

Regards, Jens

Jens, I'm away this weekend and will only be able to provide a proper reply on Tuesday.

Andreas

• You introduced Bug #751 with the changes of the 'migration of namespace'-patch

Possibly, I hadn't realized that this bug may be a side-effect of the patch.

• In version 5.1.2 Adobe moved the global namespace registry to a private class without delete modifier to prevent you from patching with the implementation of version 4.4 - shure one can work also against that

That's good! It hopefully makes it easier to move the registry into the (T)XMPMeta class.

• It makes no sense to take control of the prefixes in the xmp packet, as these are only placeholders for the namespace and there are other standard conform applications whoose may use other prefixes in case of conflicts

So you could work arround forum topic 687 by providing a delete namespace function, but you introduce further bugs with that. On the other hand you could revert the 'migration of namespace'-patch and fix Bug #751 with that.

Take a step back from XMP-SDK to Exiv2. Exiv2 addresses XMP properties with a triplet of the form 'Xmp.Prefix.Property' in which the Prefix part is the preferred prefix of a namespace. In this context, the issues described in forum topic 687 are clearly Exiv2 bugs: From an Exiv2 user standpoint it cannot be that I set a property 'Xmp.Prefix.Property' and next time I read the metadata, I get 'Xmp.Prefix_1_.Property' (and all the other strange behaviour discussed in the thread). The infamous patch fixes this bug, but issue #751 is still open and may even have been introduced in the process.

Now the way forward is to eventually fix that issue too. My suggestion was to first establish the same baseline that we had prior to the upgrade to 5.1.2, so that we can test the upgrade itself. Maybe another path is preferable, that's fine with me, as long as the end result is that at least the forum issue and preferably also bug #751 are fixed.

Perhaps there is some plausible reason to do the first, but until that I'm not willing to spend my spare time on implementing NEW issues, support them and making work arrounds to them later.

We cannot simply re-introduce the previous behavior and say it is not a bug (anymore), even though we found out in the meantime that the use of prefixes in the XMP-SDK conflicts with the way Exiv2 wants to use them.

I still believe that the way Exiv2 wants to use XMP prefixes does not violate the XMP or underlying XML standards. It just conflicts with the XMP-SDK design. As far as I can see, if we have a local prefix/namespace registry in XMP-SDK as well as Exiv2, all related problems are solved (although this will require API changes in Exiv2).

In my opinion, this is a superior solution, not a workaround. It resolves thread-safety issues in this area naturally, it allows the XMP-SDK to support both register and delete namespace operations and give users stronger guarantees with regards to prefixes.

Andreas

Andreas,

please keep in mind that, whenever you want to be able to build against external xmpsdk, you can not do any functional changes or depend on Adobes own hacks. You can fix warnings, compile errors or perhaps implement some optimization. But you'll always get in trouble if you make functional changes, be that on building on external xmpsdk or be that on simple update of sdk version.

• Now let's talk about the prefix in the xmp packet:
  Remember, the prefix is only a object local placeholder for the namespace. You also introduced another bug to the described use case in the forum topic. Previously there was registered property (http://ns.adobe.com/imageapp/1.0/):uuid. After the given command
  exiv2 -M "reg imageapp http://ns.imagapp.pro/imageapp/v1/" -M "set Xmp.imageapp.uuid ab75b096-83d3-11df-91a5-000c2953179a" 001.jpg
  in the forum thread you insert a new property (http://ns.imagapp.pro/imageapp/v1/):uuid. Here is the mistake: You removed the first property by this in favor of setting the second one. Both properties are independend of each other, they are in a different namespace. And you can not solve this, as two different namespaces can not share the same prefix. So again, you can not force a prefix in the xmp-packet!

• For exiv2 library you have some option to think about:
  As the whole exiv2 bundle uses two namespace registries, one in exiv2 library itself and one in xmpsdk, you can allow namespace overrides in the exiv2 library part. You can do a implementation that something like this will work (only meta syntax):
  exiv2 -M "reg imageapp http://ns.imagapp.pro/imageapp/v1/" -M "set Xmp.imageapp.uuid ab75b096-83d3-11df-91a5-000c2953179a" 001.jpg
  exiv2 -M "reg imageapp http://ns.imagapp.pro/imageapp/v1/" -M "get Xmp.imageapp.uuid" 001.jpg
  But again, you can not force the prefix in the xmp packet. In the above sample you need to reregister the namespace again before a second usage (which looks synchronous to the first command).
  As a second part, you can also think about some xmpsdk modification (in the included sdk version) to make the prefix in the xmp packet more friendly to the suggested one by using a object local registry - but you can not depend on that.

Hope this makes things clearer.

Regards, Jens

I've posted in the Adobe XMP SDK forum to get the viewpoint of the XMP-SDK developers on this.