Bug #1242

jp2 metadata: Unrecognized UUID EXIF box

Added by Herbert Kauer 9 months ago. Updated 9 months ago.

Status:ClosedStart date:11 Oct 2016
Priority:NormalDue date:
Assignee:Robin Mills% Done:

100%

Category:exifEstimated time:6.00 hours
Target version:0.26

Description

Hello,

I work with digiKam and ShowFoto 5.2.0 on a win 7 system.
This application uses Exiv2 0.25 for working with metadata.

I opened a *.jpg (that contains only Exif metadata) image with ShowFoto and stored it as *.jp2.

Now Exiftool cannot read the Exif metadata written by Exiv2 (via ShowFoto).
It reports : Unrecognized UUID EXIF box.
Also XnViewMP - which uses an Adobe toolkit for metadata - does not read the metadata of *.jp2 file.

On the other hand ShowFoto/Exiv2 does not read the Exif metadata (in a*.jp2 file) stored by Exiftool.

I asked Phil Harvey, the developer of Exifttol, and he stated:
"ShowFoto has apparently written a JPEG EXIF header in front of the TIFF header (six extra bytes: "Exif\0\0"),
so ExifTool gives the warning and can't process the EXIF.

From interoperability point of view this a very bad situation.
Because Exiftool and XnViewXP write a "common format" of the uuid tag, I guess the problem could be on Exiv2 side.

Please help to solve the big problem, because of interoperability.
Thanks in advance
Best regards
Herb

Test.zip (12.8 MB) Herbert Kauer, 11 Oct 2016 13:46

jp2test.txt Magnifier (3.42 KB) Herbert Kauer, 11 Oct 2016 13:46

dmpf.cpp Magnifier (1.48 KB) Robin Mills, 11 Oct 2016 20:35

0528.jp2 (8.97 MB) Herbert Kauer, 12 Oct 2016 09:15

CB_TM432.jp2 (341 KB) Herbert Kauer, 12 Oct 2016 09:15

relax.jp2 (15.9 KB) Herbert Kauer, 12 Oct 2016 12:48


Related issues

Related to Exiv2 - Feature #1243: Improved JPEG 2000 Support Closed 12 Oct 2016

Associated revisions

Revision 4619
Added by Robin Mills 9 months ago

#1242 Work in progress. Fixing src/jp2image.cpp. Added test file. Test suite to be updated to use Reagan.jp2 (and hopefully additional test files)

Revision 4622
Added by Robin Mills 9 months ago

#1242 Test harness update to use test/data/Reagan.jp2

Revision 4630
Added by Robin Mills 9 months ago

#1242 Issue warning when we encounter erroneous Exif\0\0 in the Exif UUID box.

History

#1 Updated by Robin Mills 9 months ago

  • Priority changed from High to Normal
  • Target version changed from 0.26 to 0.27

Please provide a sample image. It will be investigated and fixed in v0.27. I'm at the end of the v0.26 development cycle and very reluctant to take on anything new at this time.

I understand that you consider this is to be a high priority matter. I disagree. The jp2image.cpp code mostly dates from 2008 and, to my knowledge, this interoperability issue has never been reported. For sure, I have neither sufficient time nor energy to delay v0.26 to investigate this matter.

#2 Updated by Herbert Kauer 9 months ago

Hello,

thanks for your quick reply.
Attached please find such a test.jp2 image.

Phil Harvey stated also that after removing the 6 byte header the remaining data are a proper Exif block.

And attached find also the verbose output of Exiftool when it tries to read metadata.
Maybe it helps a little bit.

Best regards
Herb

#3 Updated by Robin Mills 9 months ago

  • File dmpf.cppMagnifier added
  • Status changed from New to Assigned
  • Target version changed from 0.27 to 0.26
  • % Done changed from 0 to 40
  • Estimated time set to 10.00

Herb

Thank you for providing a test file. I will submit a fix for this tomorrow.

I have never studied the JP2 image format and I'm very surprised to see that we don't have such an image in our test suite. When I dump your file (with the enclosed dmpf.cpp code), I see:

  0x50     80: .uuidJpgTiffExif  ->  1c  u  u  i  d  J  p  g  T  i  f  f  E  x  i  f
  0x60     96: ->JP2Exif..II*..  ->   -  >  J  P  2  E  x  i  f 00 00  I  I  * 00 08
I have manufactured a small test file as follows:
667 rmills@rmillsmbp:~/gnu/exiv2/trunk $ cp test/data/Reagan.jpg .
668 rmills@rmillsmbp:~/gnu/exiv2/trunk $ convert Reagan.jpg Reagan.jp2
669 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiftool -tagsfromfile Reagan.jpg -exif:all Reagan.jp2
    1 image files updated
670 rmills@rmillsmbp:~/gnu/exiv2/trunk $ dmpf Reagan.jp2 | grep -B 3 -A 3 uuid
0x7fb0  32688: ...........(.c..  ->  e7 8b 03 0c 2e c7 c5 b7 f7 80 05  ( ad  c 9e a4
0x7fc0  32704: .t..!e.....0...   ->  0f  t a4 a2  !  e 09 d9 12 ff   00  0 0c d5 d7
0x7fd0  32720: .i....@..d.D....  ->  09  i ed 02 f3 f4  @ ca 1f  d cc  D 14 ff d9 00
0x7fe0  32736: ..>uuidJpgTiffEx  ->  00 05  >  u  u  i  d  J  p  g  T  i  f  f  E  x
0x7ff0  32752: if->JP2MM.*.....  ->   i  f  -  >  J  P  2  M  M 00  * 00 00 00 08 00
0x8000  32768: ................  ->  0d 01 0e 00 02 00 00 01 93 00 00 00 aa 01 0f 00
0x8010  32784: ........>.......  ->  02 00 00 00 12 00 00 02  > 01 10 00 02 00 00 00
671 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 

I believe the 6 bytes Exif\0\0 should not be there.

However, I've copied the metadata from Reagan.jpg to Reagan.jp2 using ExifTool and based on this discussion: http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5696.0

I would really like a "honest" JPEG2000 file that has never been touched by Exiv2 or ExifTool. Preview on the Mac can export JPEG2000 however it remove the metadata. My Adobe Tools (PS Elements 12 and LR5) do not support JPEG 2000. Can you provide a small JP2 file that I can add to the test suite?

My implementation of the fix understands and silently forgives the erroneous Exif\0\0 bytes when it reads such a file. When it writes a JP2 does not write Exif\0\0. So the fix is forgives sins past and fixes files when they are rewritten.

For exiv2 v0.25, I added the option -pS to print the structure of an image. For exiv2 v0.26, I added the option -pR to recursively print the structure of an image. By recursive, I mean that the print will dump image structures embedded in other structures. This is particularly useful in this case as Exif metadata is usually encoded as a TIFF structure. Both options -pS and -pR are implemented by the API Image::printStructure().

For exiv2 v0.26, I have added support for ICC profiles. I'd like to support printStructure() and ICC profiles in src/jp2image.cpp. So, although I have the fix for your issue ready, I'd like to add these additional features. More importantly, I'd like to add test files to our test suite to ensure these fixes are monitored.

This issue will be fixed and submitted to the trunk tomorrow (2016-10-12). I would really appreciate a couple of small (50k or so) JPEG 2000 files. I'd be great for those test files to have embedded Exif/IPTC/XMP/ICC metadata. I hope you have a copy of PhotoShop Creative Suite with which to create those test files. Alternatively, you could ask our good friend Phil to help us out with test data.

#4 Updated by Robin Mills 9 months ago

  • Assignee set to Robin Mills

r4619. Work in progress. Hopefully Herb (or Phil) can provide more test files.

#5 Updated by Herbert Kauer 9 months ago

Hello,

thanks for all your investigations.

You asked for test images. But I regret, I do not have many *.jp2 images.
Background: I work on a small GUI that will display and/or modify metadata via Exiftool. I found ShowFoto and hoped to be able to create e.g. some *.jp2 images for testing with my GUI. So I came to this problem.

But nevertheless I have 2 *.jp2 images, which I found somewhere in the www... I do not remember where I found it. I hope it helps

Thanks again and Best regards
Herb

#6 Updated by Robin Mills 9 months ago

  • Status changed from Assigned to Closed
  • % Done changed from 40 to 100
  • Estimated time changed from 10.00 to 6.00

Herb

Thanks for the files. I don't think there is Exif/IPTC/XML/ICC data embedded in those files. I had an unsuccessful Google for suitable files yesterday. I have not been able to find the specification for JPEG2000. Curiously exiftool does not write ICC profiles in JP2 files. However I submitted the code update because I respect and trust Phil!

Here's what I've done about this:

1) I submitted the code and the test file Reagan.jp2 r4619
2) I have updated our test harness to examine that file on every commit and every nightly build on every platform. r4622
3) I hope to recruit a student in 2017 to work on our raw image support and test (#992).
4) I've opened a new issue (#1243) for v0.27 to implement jp2image::printStructure() to support options: -pX | -pS | -pC, -pR

I'm closing this issue as I believe I've dealt with your immediate concern. The follow-on matters involving printStructure() will be handled in v0.27 under #1243.

Thank You for bringing this to my attention. Thank You for using Exiv2. Good Luck with your project.

#7 Updated by Herbert Kauer 9 months ago

Hello,

thanks for your investigations and the correction.

Looking for further *.jp2 images I found one on an (rarely useed) USB-stick that also contains an icc-profile.

Best regards
Herb

#8 Updated by Robin Mills 9 months ago

For sure, there's a profile following the '!colr' bytes. All that scnrRGB XYZ stuff is ICC magic:

  0x30     48: ...,............  ->  00 00 01  , 00 00 01 90 00 03 07 07 01 00 00 00
  0x40     64: .!colr..........  ->  01  !  c  o  l  r 02 00 00 00 00 01 16 00 00 00
  0x50     80: .. ..scnrRGB XYZ  ->  00 02    00 00  s  c  n  r  R  G  B     X  Y  Z
  0x60     96:  ............acs  ->     07 d1 00 01 00 01 00 00 00 00 00 00  a  c  s
  0x70    112: p...............  ->   p 00 00 00 00 00 c0 00 00 00 00 00 00 00 00 00
  0x80    128: ................  ->  00 80 00 00 00 00 00 00 00 00 01 00 00 00 00 f6
  0x90    144: ........-.......  ->  d6 00 01 00 00 00 00 d3  - 00 00 00 00 00 00 00
I'll attach this to #1243 and it will be very useful when that issue is serviced. The existing code in jp2image.cpp can read/write Exif/IPTC/Xmp metadata. With this file, I'm sure I can figure out adding ICC profile support. However, I'm going to keep focus on getting v0.26 finished and return to JPEG2000 in v0.27.

#9 Updated by Robin Mills 9 months ago

Herb

Thank You for this very useful file, I have been able to add ICC Profile support to jp2image.cpp. #1243 will be finished today.

I'm particularly pleased to get this done as exiv2 can write/rewrite/modify/delete ICC profiles. I don't believe ExifTool can write ICC profiles! Incidentally, Exiv2 has a friendly working relationship with Phil Harvey. He's inspirational.

Robin

#10 Updated by Phil Harvey 9 months ago

FWIW, I patched ExifTool 10.30 (just released) to read the UUID EXIF box with the incorrect header, but it will issue a warning.

#11 Updated by Robin Mills 9 months ago

Thanks for doing that, Phil. I'm very surprised that this issue has been in the code for a long time without being raised. Clearly very few people use Exiv2 with JP2. I believe Jpeg2000 has not gathered much market acceptance.

I'm glad you issue a warning. I'll update libexiv2 to do the same. Silently forgiving spec violations amounts to changing the specification and is the road to chaos.

#12 Updated by Herbert Kauer 9 months ago

Hello,

thanks to everybody who helped to solve that problem in such a short time.

Best regards
Herb

#13 Updated by Robin Mills 9 months ago

You're welcome. Teamwork is really effective. I have no idea why it's seldom used in the office.

r4630 I've submitted the fix for exiv2 to issue a warning (on errout) when we read a file with the bad box. I've cut'n'pasted Phil's message for consistency.

724 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ exiftool -ver
10.30
725 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ exiftool ~/Downloads/Test.jp2 | grep -e Software -e UUID
Warning                         : Reading non-standard UUID-EXIF_bad box
Processing Software             : digiKam-5.2.0
Software                        : Version 1.0
726 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ 
726 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ bin/Debug/exiv2 --verbose --version --grep svn
exiv2 0.25 001900 (64 bit build)
svn=4629
727 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ bin/Debug/exiv2 -pa --grep Software ~/Downloads/Test.jp2
Warning: Reading non-standard UUID-EXIF_bad box in /Users/rmills/Downloads/Test.jp2
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Exif.Image.ProcessingSoftware                Ascii      14  digiKam-5.2.0
Exif.Image.Software                          Ascii      32  Version 1.0                    
Xmp.tiff.Software                            XmpText    13  digiKam-5.2.0
728 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ 
I hope to get the ICC profile written today and be totally finished with #1243.

#14 Updated by Phil Harvey 9 months ago

Robin Mills wrote:

I'm very surprised that this issue has been in the code for a long time without being raised. Clearly very few people use Exiv2 with JP2.

...and even fewer that use EXIF in JP2.

A bit of history:

In 2007 the "JpgTiffExif->JP2" box was proposed to the JPEG group and added to ExifTool at that time, but never ratified by the JPEG group. This was documented on the Jpeg2000 Wikipedia page, but it was edited out in 2008 (however it still lingers in some language versions of that page).

In 2011 I discovered that the Photoshop JPEG2000 plugin version 1.5 wrote EXIF with a different UUID (05 37 cd ab 9d 0c 44 31 a7 2a fa 56 1f 2a 11 3e) but the same format as the "JpgTiffExif->JP2" box (ie. no "Exif\0\0" header), so I added support for reading this, but ExifTool continued to write the earlier EXIF UUID box. [I just tested this with the JPEG2000 plugin 2.0 and it still does the same thing.]

- Phil

#15 Updated by Robin Mills 9 months ago

That's interesting. Do you have a test file for that guid? It sounds like a tiny change to support both the 16 byte "JpgTiffExif->JP2" and the 16 byte UUID "05 37 cd ab 9d 0c 44 31 a7 2a fa 56 1f 2a 11 3e".

I will only forgive/warn for Exif\0\0 with "JpgTiffExif->JP2". For the "approved UUID", I'll throw an exception if Exif\0\0 is present. The reasons to "forgive and warn" are:

  1. To avoid the embarrassment of admitting that Exiv2 generated illegal files.
  2. I don't want to provide a repair utility for files damaged by exiv2
  3. When the file is to be rewritten it will be repaired.

None of these reasons apply to the UUID. So, I will not forgive/warn for the UUID.

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux