Exiv2

Jerome

Thanks for bringing up this subject. I think your question is, "Can Exiv2 show IPTC history?", to which you've given the answer "No". Our IPTC code was added by Brad and perhaps he can comment on this matter.

Two questions:
1) Can you please attach the test image to this issue report?
2) Which tool produced the output you have given?

I'm not sure which website is "our website". Is this something essential to understanding this issue?

Robin

Hi,
Thanks for your answer. You will find the file in attachment.
The tool used is exiftool

exiftool -X -iptc:all ETH0138028.jpg>ETH0138028.jpg.xml

My issue is not to get history, but to be sure that when i write a tag, all history is erased.

When i write a tag, i always use a file containing all tasks at once.
First deletion of each tag i want to write using del.
And add to add the tag and it's values.

exiv2 -m taskfile shotfile

Regards
Jerome longet

Thanks very much for the image and the clarification of the issue. exiftool is great.

I do not expect Brad to step up on this one as I know he's very busy. If Brad doesn't comment on this, I will hunt through the code.

For sure it sounds as though you could script this to do del on every IPTC tag before doing the add. So another couple of questions, and then I'll get on with the work.

1) Can you script a work-around?
2) Is this urgent?

If this isn't urgent, I'd like to defer this until mid-October. It's OK to ping me next month to remind me about this. Is that OK?

Robin

Hi,
if i delete entries and add the same entrie, history is incremented. The only way i founded is to clear all tags using exiftool and replay my task file using exiv. But the issue is i rewrite all tags, EXIF, IPTC, XMP etc..... with issues about encoding and so .....

This is urgent, because our pictures are used and manipulated on our site http://www.divergence-images.com/ without any issue (we only use exiv2), but exported to another photographer platform like pixpalace who use other tools that had highlighted this issue.

Do as your best to tell me how to manage that with exiv2.
Jerome longet

Jerome

Right. Thank you very much for expressing this clearly. I will take a look at it in the next few days (probably at the weekend). You're welcome to email directly if you wish at robin@clanmills.com

Robin

Jerome

I've spent about 6 hours today looking at this. The file you provided as two IPTC IRBs (Image Resource Blocks) and I think that may be a problem. When seems to be "history" is data in the the other resource block.

I tried adding hundreds of IPTC keywords to a JPG and exiv2 never added a second IRB. So, I'm wondering if PhotoShop (which appears several times in the data) is formatting IPTC into multiple IRBs. I will investigate the free/legal download of PhotoShop CS2 for Mac.

Without a straightforward work-flow that "corrupts" the IPTC data, I don't think I can solve this matter.

Robin

Hi many thanks for this investigation !
I also made lot of tests from my side. We manage about 700 000 picture from photographers, and we have issues with this one only and with all his shots.

The only solution i found is to erase all IPTC tags at once using exiftool. It clean's the IRBs and just after import the latest IPTC tags backed up with exiv2.

I didn't found a way to delete IPTC IRBs in the exiv doc.... Is it possible ?

If can help, i will ask to the concerned photographer to explain me the way he manage his shots tags !

Regards
Jerome longet

Well three heads are better than one! Sure, please ask your user and we might say 'Ah-ha' and solve this.

I tried to install PS/CS2 on my incredible 13"MBP/R running Mountain Lion. 'Installation failed'.

I think I can create a sample app to remove all IPTC/IRBs. Let me see if I can get that to work on Saturday.

Robin

Jerome

Apologies. I didn't have time at the weekend to investigate this. However, I've looked at it today.

Good News¶

There is an option -da to delete all IPTC data in a file:

exiv2 -di R.jpg

       -d tgt Delete target(s) for the 'delete' action. Possible targets are:
              a : all supported metadata (the default)
              e : Exif section
              t : Exif thumbnail only
              i : IPTC data
              x : XMP packet
              c : JPEG comment

About my investigation¶

We have a new feature in development for 0.25 (in branches/gsoc13). We added option --struc to samples/exifprint to print the structure of a JPEG. On your file:

692 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028.jpg 
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  18072   Photoshop 3.0.8BIM............
   30658   0xed APP13  18064   Photoshop 3.0.8BIM............
   48724   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   49302   0xee APP14     14   Adobe.d@....
   49318   0xdb DQT      132   
   49452   0xc0 SOF0      17   
   49471   0xdd DRI        4   
   49477   0xc4 DHT      418   
   49897   0xda SOS       12   
-----------------
693 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 -di ETH0138028.jpg 
694 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028.jpg 
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  14060   Photoshop 3.0.8BIM.%........j.
   26646   0xed APP13  18064   Photoshop 3.0.8BIM............
   44712   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   45290   0xee APP14     14   Adobe.d@....
   45306   0xdb DQT      132   
   45440   0xc0 SOF0      17   
   45459   0xdd DRI        4   
   45465   0xc4 DHT      418   
   45885   0xda SOS       12   
-----------------
695 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ 

I've done little tests to create and delete Iptc.Application2.Keywords in a file. Multiple APP13 blocks appear and disappear on demand (the max length of an APP13 block is 65535 byte). The script to do that is:

699 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ for i in {0..5000}; do exiv2 "-Madd Iptc.Application2.Keywords xxxxx${i}yyyyyyyyyy" R.jpg ; done
700 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc R.jpg
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe1 APP1    6264   Exif..II*.....................
    6270   0xe1 APP1    2607   http://ns.adobe.com/xap/1.0/.<
    8879   0xed APP13  65535   Photoshop 3.0.8BIM............
   74416   0xed APP13  53503   Photoshop 3.0.yyyyyyyyyy.....x
  127921   0xdb DQT      132   
  128055   0xc0 SOF0      17   
  128074   0xc4 DHT      418   
  128494   0xda SOS       12   
-----------------
701 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 "-Mdel Iptc.Application2.Keywords" R.jpg
702 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc R.jpg
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe1 APP1    6264   Exif..II*.....................
    6270   0xe1 APP1    2607   http://ns.adobe.com/xap/1.0/.<
    8879   0xed APP13    108   Photoshop 3.0.8BIM.......3....
    8989   0xdb DQT      132   
    9123   0xc0 SOF0      17   
    9142   0xc4 DHT      418   
    9562   0xda SOS       12   
-----------------
703 rmills@rmills-mbp:~ 

You can see that the file added a second APP13 block and successfully reverted to 1 block when I deleted the Keywords.

Moving forward¶

To really make progress with this we need the work-flow that produced the files with the two APP13 blocks. Can you ask your user about this please?

Robin

Hi Robin,
sorry, i was not available past days. I got the worflow :

• Scan the picture and save to Tiff.
• Edit it in "fotostation version 5.1" http://www.fotoware.com/en/Products/FotoStation/
• That's it....

• Edit it in "fotostation"
• That's it....

Here is in attachment a brand new and clean version of ETH0138028.jpg annotated few days ago from the Tiff version in fotostation. We can see everything is clean !

As it's an old picture (2008), i think the photographer annotated it in an old photoshop version and updated it after with fotostation, but she don't remember ;-).

If i understand well, removing all IPTC using "-di" don't remove Blocks, so the good way is to remove IPTC by keys using actions "-Mdel Iptc.Application2.IPCTKEY" until, i got only 1 block ?

Many thanks for your support !

Regards
Jerome

Jerome

Thanks for the update. Well that's simple. I never like to say "looks like a bug in FotoStation" - however that's what I am thinking.

538 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028_actuel.jpg 
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5392   Exif..MM.*....................
    5416   0xed APP13  15830   Photoshop 3.0.8BIM............
   21248   0xe1 APP1   22518   http://ns.adobe.com/xap/1.0/.<
   43768   0xe2 APP2    3160   ICC_PROFILE......HLino....mntr
   46930   0xee APP14     14   Adobe.d@....
   46946   0xdb DQT      132   
   47080   0xc0 SOF0      17   
   47099   0xdd DRI        4   
   47105   0xc4 DHT      418   
   47525   0xda SOS       12   
-----------------
556 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ cp ~/temp/iptc/ETH0138028.jpg .   # nice new copy of "rogue" file
557 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028.jpg # it has two APP13 blocks
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  18072   Photoshop 3.0.8BIM............
   30658   0xed APP13  18064   Photoshop 3.0.8BIM............
   48724   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   49302   0xee APP14     14   Adobe.d@....
   49318   0xdb DQT      132   
   49452   0xc0 SOF0      17   
   49471   0xdd DRI        4   
   49477   0xc4 DHT      418   
   49897   0xda SOS       12   
-----------------
558 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 -pi ETH0138028.jpg | wc # and 142 iptc tages
    142     628   10201
559 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 "-Mdel Iptc.Application2.Keywords" ETH0138028.jpg # remove the Keywords
560 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 -pi ETH0138028.jpg | wc # 14 iptc tags left
     14      91    1192
561 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ for i in $(exiv2 -pi  ETH0138028.jpg | cut -f 1 -d' '); do exiv2 "-Mdel $i" ETH0138028.jpg ; done # remove them one at a time
562 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 -pi ETH0138028.jpg | wc # no more iptc tags
      0       0       0
563 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028.jpg # still got two APP13 blocks
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  15824   Photoshop 3.0.8BIM............
   28410   0xed APP13  18064   Photoshop 3.0.8BIM............
   46476   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   47054   0xee APP14     14   Adobe.d@....
   47070   0xdb DQT      132   
   47204   0xc0 SOF0      17   
   47223   0xdd DRI        4   
   47229   0xc4 DHT      418   
   47649   0xda SOS       12   
-----------------
564 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ exiv2 -di ETH0138028.jpg # use the DI command
565 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint --struc ETH0138028.jpg # file is smaller, but still has 2 APP13 blocks
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  14060   Photoshop 3.0.8BIM.%........j.
   26646   0xed APP13  18064   Photoshop 3.0.8BIM............
   44712   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   45290   0xee APP14     14   Adobe.d@....
   45306   0xdb DQT      132   
   45440   0xc0 SOF0      17   
   45459   0xdd DRI        4   
   45465   0xc4 DHT      418   
   45885   0xda SOS       12   
-----------------
566 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ 

I will email FotoWare tomorrow and ask them for a free copy of their product for Engineering purposes. They're Germans and they'll be friendly and cooperative.

I'll let you know.

Robin

Waouh ! Blocks are really persistants !

Jerome

It appears to me that FotoStation has put something nasty into the file. You've discovered how to remove the APP13 blocks with exiftool and that helps. However I think the problem is the two APP13 blocks introduced by FotoStation. Of course, I could be wrong and I am quite unfairly blaming FotoStation. (And we all know that Germans make Great Software).

I have the email written and ready to send to sales@fotoware.com. Can you email me and I can copy you in the correspondence with them. I am robin@clanmills.com. Thanks.

Robin

Jérôme

I didn't receive a reply from FotoWare, however thank you for providing this link for an eval of FotoStation:
http://www.fotoware.com/en/Downloads/

I've discovered that you can easily remove the APP13 sections from the JPG as follows:

707 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ bin/exifprint  --struc ETH0138028.jpg 
STRUCTURE OF FILE:
  offset | marker     | size | signature
       2   0xd8 SOI          
       4   0xe0 APP0      16   JFIF.....,.,..
      22   0xe1 APP1    5372   Exif..MM.*....................
    5396   0xe1 APP1    7186   http://ns.adobe.com/xap/1.0/.<
   12584   0xed APP13  18072   Photoshop 3.0.8BIM............
   30658   0xed APP13  18064   Photoshop 3.0.8BIM............
   48724   0xe2 APP2     576   ICC_PROFILE......0ADBE....mntr
   49302   0xee APP14     14   Adobe.d@....
   49318   0xdb DQT      132   
   49452   0xc0 SOF0      17   
   49471   0xdd DRI        4   
   49477   0xc4 DHT      418   
   49897   0xda SOS       12   
-----------------
708 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ (dd bs=$((12584-1)) count=1 if=ETH0138028.jpg ; dd bs=$((48724-1)) skip=1 if=ETH0138028.jpg) > foo.jpg 2>/dev/null 
710 rmills@rmills-mbp:~/gnu/exiv2/gsoc13 $ 

Let's discuss the best way to automate this for your system. Possibilities are: bash script, add option -dI to exiv2, modify or add a new sample application?

Robin

I've received a reply from FotoWare and a permanent license for FotoStation. Thank You to CEO Christopher Frenning. Here is my reply:

Christopher.

Thank you for your response. Very much appreciated.

My user discovered the eval download last week and I installed that on my MacBook Pro (my preferred platform). When I installed FotoStation, I discovered that you use Exiv2 (statically linked, I see). Things can't get better.

Thank You for the offer of Permanent License. It will only be used for Exiv2 support and if I discover that I love using FotoStation, I'll purchase a license for personal use.

We've made good progress with this. I suspect the trail to the work-flow that resulted in the corrupted files can never be re-discovered. And of course I don't know which version/platform of FotoStation or libexiv2 is in use.
I've proposed a solution to my user which involves adding new options to exiv2(.exe) 0.25 (expected late 2013)

1) -pS to print JPG structure.

2) -dI to remove APP13 blocks from a JPG.

These options only enable my user to detect and repair the damage. The root cause remains hidden. When this re-appears (and it will), I will be tooled up with the permanent licence to investigate and fix this.

Thank you for helping. And thank you for using Exiv2.

I'm going to proceed to add those options to Exiv2 (on the gsoc13 branch).

Robin

HI robin,
Thanks for this huge investigation & happy to be able to use those options directly on a futur version. How can i be pinged when it will be integrated on a stable release ?

Regards
Jerome

Jerome

You're welcome. I've discovered that FotoStation uses Exiv2 - so I'm doubly motived to get to the bottom of this. Here's what I said to FotoWare:

Christopher and Andrers

Thanks very much. I mistakenly thought your business was in Germany. Norway - even better. If Anders wants to look at the files, they are available on http://dev.exiv2.org/boards/3/topics/1608 along with a lengthy discussion and analysis.

I have edited the metadata in the original image with FotoStation. I added some IPTC, Xmp and Exif tags. Everything seems OK to me.

FotoStation is a very nice application. I'm going to mark this issue as "resolved" as I don't believe we can make much more progress without more information from the end user.

You'll see that I haven't reproduced this in Foto Station with your user's files. For sure, I'll need the platform/version of Foto Station involved. And probably screen shots of the work flow (tags and values added). I realise the time and effort required by you and your user to document this. If you have different ideas, I will of course listen.

I have added feature request 922 for the options we have discussed adding to exiv2(.exe). I have added to as a watcher and you will receive email as we progress the issue: http://dev.exiv2.org/issues/922

Robin