Exiv2

Susanne

Thanks for getting in touch. And thank you for using Exiv2. I've been working with Exiv2 for more than 10 years and I think this could be the first time that anybody has discussed side-car files. And I think you have uncovered a bug that's been there forever!

You have also revealed a confuser error. Any tag that begins with "Exif" is a member of the Exif family of tags and shouldn't be in XMP. The library has convertors to synchronise Exif, IPTC and XMP. So messing with Exif.Image.Artist in a side-car isn't a good idea, however it should work.

I'll investigate next week and give you an update.

You may be interested to hear that we reached Exiv2 v0.27 RC3 yesterday and I'm confident that we will achieve Exiv2 v0.27 GM on 28 December 2018. http://exiv2.dyndns.org

Robin

686 rmills@rmillsmbp:~/temp $ exiv2 -pX http://clanmills.com/Stonehenge.jpg > S.xmp
687 rmills@rmillsmbp:~/temp $ xmllint --format --pretty 2 S.xmp 
<?xml version="1.0"?>
<?xpacket
begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta
    xmlns:x="adobe:ns:meta/" 
    x:xmptk="XMP Core 4.4.0-Exiv2" 
  ><rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
    ><rdf:Description
        xmlns:xmp="http://ns.adobe.com/xap/1.0/" 
        xmlns:dc="http://purl.org/dc/elements/1.1/" 
        rdf:about="" 
        xmp:Rating="0" 
        xmp:ModifyDate="2015-07-16T20:25:28+01:00" 
      ><dc:description
        ><rdf:Alt
          ><rdf:li
              xml:lang="x-default" 
            >Classic View</rdf:li
          ></rdf:Alt
        ></dc:description
      ></rdf:Description
    ></rdf:RDF
  ></x:xmpmeta
>
<?xpacket
end="w"?>
688 rmills@rmillsmbp:~/temp $ for i in {1..5}; do exiv2 -M"set Exif.Image.Artist $i" S.xmp; done
689 rmills@rmillsmbp:~/temp $ xmllint --format --pretty 2 S.xmp 
<?xml version="1.0"?>
<?xpacket
begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta
    xmlns:x="adobe:ns:meta/" 
    x:xmptk="XMP Core 4.4.0-Exiv2" 
  ><rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
    ><rdf:Description
        xmlns:xmp="http://ns.adobe.com/xap/1.0/" 
        xmlns:dc="http://purl.org/dc/elements/1.1/" 
        rdf:about="" 
        xmp:ModifyDate="2015-07-16T20:25:28+01:00" 
        xmp:Rating="0" 
      ><dc:creator
        ><rdf:Seq
          ><rdf:li
            >1</rdf:li  <---- This was only updated once!
          ></rdf:Seq
        ></dc:creator
      ><dc:description
        ><rdf:Alt
          ><rdf:li
              xml:lang="x-default" 
            >Classic View</rdf:li
          ></rdf:Alt
        ></dc:description
      ></rdf:Description
    ></rdf:RDF
  ></x:xmpmeta
>
<?xpacket
end="w"?>
690 rmills@rmillsmbp:~/temp $ 

I've logged a bug against v0.27 for this matter and hope it will be fixed. https://github.com/Exiv2/exiv2/issues/589

I'm very surprised by this. I almost can't imagine how that could happen as the code reads the XMP into an internal memory structure which is then modified by the -M"set..." command. Then the XMP is rewritten to file, if it has been modified. Clearly the XMP doesn't know it has been modified. Ah, yes, there's the explanation. It has something to do with the convertors who are busy restoring the data. Crap! The convertors are a bad idea. We should only read/write exactly what is in the file. No magic. No confusion. We should never use the convertors on a side-car. I'm confident that I'll find this next week. I expect it'll be a "low risk" fix although we are now in "code freeze" for v0.27. I'll update you next week.

Susanne:

I've investigated this and reported my fix in https://github.com/Exiv2/exiv2/issues/589

As we are now in "code freeze", I'm unwilling to add this to Exiv2 v0.27. If you built Exiv2 from source, you can update your code and rebuild. If you don't want to build from source, I'm happy to complete the fix so that the test harness passes and give you a private build on v0.27.0.4 with this included. Let me know which platform you require.

Robin

Wow! Thanks a lot! That's fast! :-D

I saw, that libexiv2 could handle xmp-sidecar files, though why should i use another lib to handle them ;-)

From my point of view, I would be happy, if the converters (I think you mean e.g.: write Exif.Image.Artist to Xmp.dc.creator) would function as before, it's quite handy. And - from my point of view - would be nice, if the sidecar files and the image-files has the same logic from the view of the developer who uses libexiv2. ... just my 2ct. (If I'm allowed).

I'm currently use the packaged libexiv2 from Linux-Mint and Debian which is currently 0.25. I think I will need to compile it by myself for the future, to have the newest version available.

Thanks a lot Robin!

Susanne

Don't use another library. Exiv2 is best of breed (by far).

When I published v0.27 RC3 on Friday, I hoped it would be the final RC. Regrettably not. Last night, a user reported some security issues which I will fix and push to RC4 later this week.

I will push "your fix" into RC4. There is almost no risk to libexiv2 and no test suite changes. It's simply a fix for a bug that's been there for about 10 years!

Perhaps you could download and install Exiv2 v0.27 RC4 Linux bundle and let me know how it works for you. You don't need to build anything and I don't expect it to disturb your machine. It'll be available tomorrow on http://exiv2.dyndns.org:8080/userContent/builds/Version/0.27.0.4/

Robin

Robin, great to hear!

I'll try it tomorrow to build it and give it a try! Thanks a lot! :-)

Thanks a lot, again! :-)

If you want to build it, you'll have to pull down the RC4 code:

$ git clone https://github.com/exiv2/exiv2 --depth 50 --branch 0.27-RC4
$ cd exiv2
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make test
$ sudo make install

Or you can pull down the bundle from exiv2.dyndns.org. The build server runs Ubuntu. I don't know if those builds can be used on Mint.

Robin,

Don't know if it's already the version which I should test... (downloaded from github 0.27-RC4) or should I wait for http://exiv2.dyndns.org:8080 ...?

I encountered the following:

In XMP-sidecars:
changing Exif.Image.Artist works only the first time. (changes Exif.Image.Artist and Xmp.dc.creator)
changing Xmp.dc.creator works like an array (like Xmp.dc.subject), it adds every entry to a list and shows this list comma-separated. And the list is displayed as Exif.Image.Artist.

In JPG-files:
changing Exif.Image.Artist changes only Exif.Image.Artist and I could change it as often as I like.
changing Xmp.dc.creator works like an array (like Xmp.dc.subject), it adds every entry to a list and shows this list comma-separated. It DOESN't change Exif.Image.Artist.

Sorry if I missunderstand something!

The build is now available http://exiv2.dyndns.org:8080/userContent/builds/all/exiv2-0.27.0.4-Linux64-2018%3A12%3A10_18%3A56%3A49.tar.gz

I'm not sure what you're looking at on Github. Is it the issue report? https://github.com/Exiv2/exiv2/issues/589 I've pushed the fix for #589 into branch 0.27-RC4, so if you've built from source, it should be working.

Robin

I updated my git-clone of 0.27-RC4 this morning to the latest changes and did a recompile.

I have the following code to write my metadata to objects (e.g. xmp-files):

my code to write metadata is something like¶

void ImageBrowserModel::write(QString xmpfile, QString exifKey, QString text) {
        if (currentDirectory.exists(xmpFile)) {
            Exiv2::Image::AutoPtr xmpImage = Exiv2::ImageFactory::open(xmpFile.toLatin1().data());
            xmpImage->readMetadata();
            if (exifKey.startsWith("Exif")) {
                Exiv2::ExifData exifData = xmpImage->exifData();
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                exifData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setExifData(exifData);
            }  else if (exifKey.startsWith("Xmp")) {
                Exiv2::XmpData xmpData = xmpImage->xmpData();
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                xmpData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setXmpData(xmpData);
            }  else if (exifKey.startsWith("Iptc")) {
                Exiv2::IptcData iptcData = xmpImage->iptcData();
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                iptcData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setIptcData(iptcData);
            }
            xmpImage->writeMetadata();
        } else {
            Exiv2::Image::AutoPtr xmpImage = Exiv2::ImageFactory::create(Exiv2::ImageType::xmp, xmpFile.toLatin1().data());
            if (exifKey.startsWith("Exif")) {
                Exiv2::ExifData exifData;
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                exifData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setExifData(exifData);
            }  else if (exifKey.startsWith("Xmp")) {
                Exiv2::XmpData xmpData;
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                xmpData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setXmpData(xmpData);
            }  else if (exifKey.startsWith("Iptc")) {
                Exiv2::IptcData iptcData;
                Exiv2::XmpProperties::registerNs("http://ns.adobe.com/tiff/1.0/", "tiff");
                iptcData[exifKey.toLatin1().data()] = text.toLatin1().data();
                xmpImage->setIptcData(iptcData);
            }
            xmpImage->writeMetadata();
        }
    }

my tests are¶

1.) if the file doesn't exists and I write Exif.Image.Artist = This is a test, it results in: (output of exiv2 -p a <filename>)

Exif.Image.Artist                            Ascii      15  This is a test
Iptc.Application2.Byline                     String     14  This is a test
Iptc.Envelope.CharacterSet                   String      3  
Xmp.dc.creator                               XmpSeq      1  This is a test

2.) if the file exists and I try to write Exif.Image.Artist = This a test2, nothing changed.

3.) even with exiv2 -M"set Exif.Image.Artist Ascii This is a test2" RAW_OLYMPUS_E-M10.xmp same result, nothing changed.

4.) if the file exists and I try to write Xmp.dc.creator = This is a test2 it results in:

Exif.Image.Artist                            Ascii      32  This is a test, This is a test2
Iptc.Application2.Byline                     String     14  This is a test
Iptc.Application2.Byline                     String     15  This is a test2
Iptc.Envelope.CharacterSet                   String      3  
Xmp.dc.creator                               XmpSeq      2  This is a test, This is a test2

5.) if the file exists and I execute exiv2 -M"set Xmp.dc.creator XmpSeq This is a test3" RAW_OLYMPUS_E-M10.xmp, it results in:

Exif.Image.Artist                            Ascii      49  This is a test, This is a test2, This is a test3
Iptc.Application2.Byline                     String     14  This is a test
Iptc.Application2.Byline                     String     15  This is a test2
Iptc.Application2.Byline                     String     15  This is a test3
Iptc.Envelope.CharacterSet                   String      3  
Xmp.dc.creator                               XmpSeq      3  This is a test, This is a test2, This is a test3

6.) and 7.)
Is it correct, that with
exiv2 -M"del Exif.Image.Artist Ascii" RAW_OLYMPUS_E-M10.xmp and
exiv2 -M"del Xmp.dc.creator" RAW_OLYMPUS_E-M10.xmp
nothing changed and the output is still:

susanne@snoopy ~/abc/OlyRawTest2 $ exiv2 -p a RAW_OLYMPUS_E-M10.xmp

Exif.Image.Artist                            Ascii      49  This is a test, This is a test2, This is a test3
Iptc.Application2.Byline                     String     14  This is a test
Iptc.Application2.Byline                     String     15  This is a test2
Iptc.Application2.Byline                     String     15  This is a test3
Iptc.Envelope.CharacterSet                   String      3  
Xmp.dc.creator                               XmpSeq      3  This is a test, This is a test2, This is a test3

my conclusion ...¶

I'm happy, that my code results in the same as the command-line utility! (uff! yeah! :-) )

Would you think, that this behaviour is the expected one? (Makes me a bit confused... )

Initial writings could be done through the Exif-Tags, changes only via the Xmp-tags?

For me, it would be great to also have the possibility, to change the content through exif-tags also ... (But I know it's your project and decision!) :-)

Susanne

Would you like to talk one-to-one on Skype? I'm in England. My Skype name is "clanmills". Or we can talk on the phone, if you prefer. I'm 100% confident of success and that you will be happy. A little more work will get us to your destination.

I didn't write Exiv2, so I'm not too sure what's expected in some places. And for sure, I don't know what you expect! I don't think the convertors are a good idea. I like deterministic systems. The convertors introduce some "random" behaviour. However the convertors have been there forever, so I can't just "yang them out".

Comments:
1) You don't need to register the tiff namespace. It's "preregistered" for you. You can see the pregistered namespace with the command $ exiv2 --verbose --version. And you can filter that with --grep:

522 rmills@rmillsmbp:~/gnu/exiv2/team/drawings $ exiv2 -vVg xmlns | grep tiff
xmlns=tiff:http://ns.adobe.com/tiff/1.0/
523 rmills@rmillsmbp:~/gnu/exiv2/team/drawings $ 

2) The "this is a test, this is a test" is because Xmp.dc.creator is an XmpSeq (an array). So every time you define another value, it's added to the XmpSeq. You can delete the XmpSeq with -M"del Xmp.dc.creator foo.xmp" and then define it as an XmpText with -M"set Xmp.dc.creator XmpText Robin Mills" foo.xmp .

3) I think it's very useful to inspect the XML in the image: $ exiv2 -pX foo.xmp | xmllint --format -- as this makes it easier to see what's happing. This is very useful when you are operating with XMPSeq and XMPBag (a structure/object entity). There's also a utility exiv2json which will format the objects in JSON which you can browse.

4) I don't think you need ASCII in the code -M"del .... ASCII" image.xmp

5) Is the only difference in the if ( exists ) {...} is calling ImageFactory create or open. If that's the case we can condense this to::

void ImageBrowserModel::write(QString xmpfile, QString exifKey, QString text)
{
    Exiv2::Image::AutoPtr xmpImage = currentDirectory.exists(xmpFile) 
                                   ? Exiv2::ImageFactory::open(xmpFile.toLatin1().data())
                                   : Exiv2::ImageFactory::create(Exiv2::ImageType::xmp, xmpFile.toLatin1().data())
                                   ;
    xmpImage->readMetadata();
    if (exifKey.startsWith("Exif")) {
        Exiv2::ExifData exifData = xmpImage->exifData();
        exifData[exifKey.toLatin1().data()] = text.toLatin1().data();
        xmpImage->setExifData(exifData);
    }  else if (exifKey.startsWith("Xmp")) {
        Exiv2::XmpData xmpData = xmpImage->xmpData();
        xmpData[exifKey.toLatin1().data()] = text.toLatin1().data();
        xmpImage->setXmpData(xmpData);
    }  else if (exifKey.startsWith("Iptc")) {
        Exiv2::IptcData iptcData = xmpImage->iptcData();
        iptcData[exifKey.toLatin1().data()] = text.toLatin1().data();
        xmpImage->setIptcData(iptcData);
    }
    xmpImage->writeMetadata();
}

And then I can refactor into two functions (to remove the Qt stuff):

void writeMetadata(const char* xmpFile, const char* exifKey, const char* text)
{
    bool  bExists = false;
    FILE* f       = fopen(xmpFile,"r");
    if  ( f ) {
        fclose(f);
        bExists = true ;
    }
    Exiv2::Image::AutoPtr xmpImage = bExists
                                   ? Exiv2::ImageFactory::open(xmpFile)
                                   : Exiv2::ImageFactory::create(Exiv2::ImageType::xmp, xmpFile)
                                   ;
    xmpImage->readMetadata();
    if (exifKey.startsWith("Exif")) {
        Exiv2::ExifData exifData = xmpImage->exifData();
        exifData[exifKey.toLatin1().data()] = ;
        xmpImage->setExifData(exifData);
    }  else if (exifKey.startsWith("Xmp")) {
        Exiv2::XmpData xmpData = xmpImage->xmpData();
        xmpData[exifKey.toLatin1().data()] = text.toLatin1().data();
        xmpImage->setXmpData(xmpData);
    }  else if (exifKey.startsWith("Iptc")) {
        Exiv2::IptcData iptcData = xmpImage->iptcData();
        iptcData[exifKey.toLatin1().data()] = text.toLatin1().data();
        xmpImage->setIptcData(iptcData);
    }
    xmpImage->writeMetadata();
}

void ImageBrowserModel::write(QString xmpfile, QString exifKey, QString text)
{
    writeMetadata(xmpfile.text.toLatin1().data(), exifKey.text.toLatin1().data(),text.text.toLatin1().data());
}

So now we have a function writeMetadata() which you can call from Qt and I can use from the command line.

6) I'm going to get some coffee and walk each of your commands using the dentist's method (you pull them out, one at a time).

Robin,

please don't invest so much time in it. It's my problem, I don't want to hold you back from your day to day work!

I've so little knowledge about c++ and qt (I'm more in groovy/grails/java the last years) but I picked c++ and qt for a little private project of mine (https://github.com/susannej/interlace but it's really dirty code!!! Need lots of cleanup!)
Currently I display a lot of exif/xmp data for each photo, I should condense it to the most important and do a programmatic switch for updating (if the user selects "copyright" I should update Xmp.dc.rights for instance if it's an xmp and Exif.Image.Copyright and Xmp.dc.rights if it's an jpeg etc.) But that's nothing which belongs to exiv2 and I wouldn't like to spam here with my own program problems!

Many, many thanks!!! :-)

Susanne,

If you're doing a lot of metadata handling, and working with both Exif and XMP, then it might be worth having a look at the Metadata Working Group (MWG) guidelines (if you haven't already). They have good advice on how to handle conflicts between different representations of the same information. The specs should be at http://www.metadataworkinggroup.org/specs/ but the site's not currently working for me.

(My little metadata project might amuse you: https://github.com/jim-easterbrook/Photini)

Jim

Jim:
Nice to hear from you again. It's been a while. More than a year, perhaps?

You're right about MWG. I know lots about manipulating metadata in C++. What does the data mean and how should it be used to ensure smooth work-flows across products. I believe that's the business of MWG.

Susanne:
I'm retired. That's why I have so much time to spend on Exiv2! So, if I can help, please let me know. I admire your courage in taking on Qt/C++ on your own. It's a big/bad/amazing/robust/fragile SOB!

Both:
When we release Exiv2 v0.27 on December 28, 2018, I will stop contributing C++. In 2019/2020, I'll deal with users, releases, documentation and the web-site. I plan less time for Exiv2 to pursue other items on my bucket list. https://clanmills.com/BucketList.shtml

Good Luck to you both with your projects and thank you for using Exiv2.

Robin

Sorry for the delay, I catched a bad cold ...

Jim:
Thanks for the tip! Thanks to the wayback-machine, I could get a copy of it!
I'll have a look into your project, looks promising! I hope I could get a few ideas from it. :-) Nothing to amuse, that's something I had in my mind at first, but I would like to extend it a bit (with a database behind etc.) Thanks for the link!

Robin:
Thanks a lot! It's nice to know to have someone I could ask! :-)

Your bucket-list is something, I should have to work out for me, too. My plan - for the moment - is, to spend next year more time for photographing and guitar-playing (and get to a point with my little program, where I could tag and find my images easier than now...) (for photographing -> travelling to different places at the german north-sea coast and/or Scotland etc.)

Thanks a lot to you!