Exiv2

Adobe published a new Edition of the XMPsdk on 30 July 2016. The latest version of this document XMP-Toolkit-SDK-CC201607/docs/XMPSpecificationPart3.pdf contains this statement on page 13. This statement was not in the 2010 Edition of that document.

The JPEG standard does not prescribe ordering among APPn segments, but some related standards do. For example, Exif requires that Exif APP1 segment be immediately after the SOI. Also, some applications improperly assume that the segments are in a particular order. For compatibility, it is best to put the Exif APP1 first, the XMP APP1 next, the PSIR APP13 next, followed by all other marker segments.

NOTE If the size of the Exif APP1 marker or the PSIR APP13 marker exceeds 64KB, the marker is split into multiple blocks of 64KB size each.

If this is true, we can indeed store huge blobs of Exif metadata. If the metadata exceeds 64k, we can simply divide it into multiple segments. However, I would want to see a reliable example of such a file. By "reliable", I mean it has a blue-chip pedigree (such as PhotoShop). And I'd like to see a specification which is focused on this matter. The statement in this spec doesn't feel definitive and doesn't address APP1 continuation block signatures.

A typical JPEG structure is as follow:

696 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pS test/data/Reagan.jpg 
STRUCTURE OF JPEG FILE: test/data/Reagan.jpg
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |    5718 | Exif..MM.*......................     <---- This is the Exif data (Tiff Encoded)
    5722 | 0xffed APP13 |    3038 | Photoshop 3.0.8BIM..........Z...
    8762 | 0xffe1 APP1  |    5329 | http://ns.adobe.com/xap/1.0/.<?x
   14093 | 0xffe2 APP2  |     576 | ICC_PROFILE......0ADBE....mntrRG chunk 1/1
   14671 | 0xffee APP14 |      14 | Adobe.d@......
   14687 | 0xffdb DQT   |     132 
   14821 | 0xffc0 SOF0  |      17 
   14840 | 0xffdd DRI   |       4 
   14846 | 0xffc4 DHT   |     418 
   15266 | 0xffda SOS  
697 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 

I think Adobe are saying that if you follow an APP1 segment with one or more APP1 segments of exactly 64k bytes you should consider it to be a "segmented" continuation of its predecessor. That seems reasonable to me. However it's quite different from the "chunk" mechanism used to allow ICC profiles to exceed a single APP2 segment. Additionally, APP1 segments begin with a signature such as "Exif\0\0" to identify them to readers.

I've hunted through Phil (of ExifTool fames) repository of 7000 JPG files. Not one has Exif data encoded in this way.

AGFA have files with multiple segments of length 65535. However they are not APP1 segment. Here's the evidence:

702 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pS /mmHD/Users/rmills/Jenkins/testfiles/Phils/Agfa/AgfaOPTIMA1338mT.jpg 
STRUCTURE OF JPEG FILE: /mmHD/Users/rmills/Jenkins/testfiles/Phils/Agfa/AgfaOPTIMA1338mT.jpg
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   46459 | Exif..II*......................
   46463 | 0xffe3 APP3  |   65535 | ................................   <--- what is this?
  112000 | 0xffe4 APP4  |   65535 | ..Hc..w .8<...z..M.77.h...{.....   <--- or this?
  177537 | 0xffe5 APP5  |    7243 | .U......K..u=).pl.W.F...B.$.3...
  184782 | 0xffdb DQT   |     132 
  184916 | 0xffc0 SOF0  |      17 
  184935 | 0xffc4 DHT   |      75 
  185012 | 0xffda SOS  
703 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 

I'm willing to implement "Exif data exceeds 64k" feature in the future provided we have a more substantial specification and good pedigree test files.

This looks like straightforward to implement and we'll add this early in v0.28. I've copied your test file

794 rmills@rmillsmbp:~/temp $ exiv2 -pS http://exiv2.dyndns.org:8080/userContent/testfiles/1232/multi-segment_exif.jpg 
STRUCTURE OF JPEG FILE: http://exiv2.dyndns.org:8080/userContent/testfiles/1232/multi-segment_exif.jpg
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   65535 | Exif..II*......................
   65539 | 0xffe1 APP1  |    5603 | Exif............................
   71144 | 0xffdb DQT   |     132 
   71278 | 0xffc4 DHT   |     418 
   71698 | 0xffc0 SOF0  |      17 
   71717 | 0xffda SOS  
795 rmills@rmillsmbp:~/temp $ 

I think I've understood what you've said about 2^16-1 to ensure that a tag doesn't straddle the jpeg chunks.

Hi Robin,

You requested a "reliable example" of multi-segment Exif. The "multi-segment_exif.jpg" I gave you was written by ExifTool. I have also attached a "cs4_extended_exif.jpg" sample written by Photoshop CS4. To generate this sample I filled the Exif ImageDescription with a very long bit of text (the exiftool source code). Unfortunately, as you can see, the Adobe software is buggy and writes an incorrect count for this tag, but at least it shows you how they intended to split the Exif into multiple segments.

The most difficult thing about adding support for this in ExifTool was in patching the -htmlDump feature to allow for gaps in the middle of the Exif due to the JPEG segment headers. Other than this, it was a fairly trivial addition.