jp2 metadata: Unrecognized UUID EXIF box
|Status:||Closed||Start date:||11 Oct 2016|
|Assignee:||Robin Mills||% Done:|
|Category:||exif||Estimated time:||6.00 hours|
I work with digiKam and ShowFoto 5.2.0 on a win 7 system.
This application uses Exiv2 0.25 for working with metadata.
I opened a *.jpg (that contains only Exif metadata) image with ShowFoto and stored it as *.jp2.
Now Exiftool cannot read the Exif metadata written by Exiv2 (via ShowFoto).
It reports : Unrecognized UUID EXIF box.
Also XnViewMP - which uses an Adobe toolkit for metadata - does not read the metadata of *.jp2 file.
On the other hand ShowFoto/Exiv2 does not read the Exif metadata (in a*.jp2 file) stored by Exiftool.
I asked Phil Harvey, the developer of Exifttol, and he stated:
"ShowFoto has apparently written a JPEG EXIF header in front of the TIFF header (six extra bytes: "Exif\0\0"),
so ExifTool gives the warning and can't process the EXIF.
From interoperability point of view this a very bad situation.
Because Exiftool and XnViewXP write a "common format" of the uuid tag, I guess the problem could be on Exiv2 side.
Please help to solve the big problem, because of interoperability.
Thanks in advance
#1242 Work in progress. Fixing src/jp2image.cpp. Added test file. Test suite to be updated to use Reagan.jp2 (and hopefully additional test files)
#1 Updated by Robin Mills 7 months ago
- Priority changed from High to Normal
- Target version changed from 0.26 to 0.27
Please provide a sample image. It will be investigated and fixed in v0.27. I'm at the end of the v0.26 development cycle and very reluctant to take on anything new at this time.
I understand that you consider this is to be a high priority matter. I disagree. The jp2image.cpp code mostly dates from 2008 and, to my knowledge, this interoperability issue has never been reported. For sure, I have neither sufficient time nor energy to delay v0.26 to investigate this matter.
#2 Updated by Herbert Kauer 7 months ago
thanks for your quick reply.
Attached please find such a test.jp2 image.
Phil Harvey stated also that after removing the 6 byte header the remaining data are a proper Exif block.
And attached find also the verbose output of Exiftool when it tries to read metadata.
Maybe it helps a little bit.
#3 Updated by Robin Mills 7 months ago
- File dmpf.cpp added
- Status changed from New to Assigned
- Target version changed from 0.27 to 0.26
- % Done changed from 0 to 40
- Estimated time set to 10.00
Thank you for providing a test file. I will submit a fix for this tomorrow.
I have never studied the JP2 image format and I'm very surprised to see that we don't have such an image in our test suite. When I dump your file (with the enclosed dmpf.cpp code), I see:
0x50 80: .uuidJpgTiffExif -> 1c u u i d J p g T i f f E x i f 0x60 96: ->JP2Exif..II*.. -> - > J P 2 E x i f 00 00 I I * 00 08I have manufactured a small test file as follows:
667 rmills@rmillsmbp:~/gnu/exiv2/trunk $ cp test/data/Reagan.jpg . 668 rmills@rmillsmbp:~/gnu/exiv2/trunk $ convert Reagan.jpg Reagan.jp2 669 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiftool -tagsfromfile Reagan.jpg -exif:all Reagan.jp2 1 image files updated 670 rmills@rmillsmbp:~/gnu/exiv2/trunk $ dmpf Reagan.jp2 | grep -B 3 -A 3 uuid 0x7fb0 32688: ...........(.c.. -> e7 8b 03 0c 2e c7 c5 b7 f7 80 05 ( ad c 9e a4 0x7fc0 32704: .t..!e.....0... -> 0f t a4 a2 ! e 09 d9 12 ff 00 0 0c d5 d7 0x7fd0 32720: .i....@..d.D.... -> 09 i ed 02 f3 f4 @ ca 1f d cc D 14 ff d9 00 0x7fe0 32736: ..>uuidJpgTiffEx -> 00 05 > u u i d J p g T i f f E x 0x7ff0 32752: if->JP2MM.*..... -> i f - > J P 2 M M 00 * 00 00 00 08 00 0x8000 32768: ................ -> 0d 01 0e 00 02 00 00 01 93 00 00 00 aa 01 0f 00 0x8010 32784: ........>....... -> 02 00 00 00 12 00 00 02 > 01 10 00 02 00 00 00 671 rmills@rmillsmbp:~/gnu/exiv2/trunk $
I believe the 6 bytes
Exif\0\0 should not be there.
However, I've copied the metadata from Reagan.jpg to Reagan.jp2 using ExifTool and based on this discussion: http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=5696.0
I would really like a "honest" JPEG2000 file that has never been touched by Exiv2 or ExifTool. Preview on the Mac can export JPEG2000 however it remove the metadata. My Adobe Tools (PS Elements 12 and LR5) do not support JPEG 2000. Can you provide a small JP2 file that I can add to the test suite?
My implementation of the fix understands and silently forgives the erroneous
Exif\0\0 bytes when it reads such a file. When it writes a JP2 does not write
Exif\0\0. So the fix is forgives sins past and fixes files when they are rewritten.
For exiv2 v0.25, I added the option
-pS to print the structure of an image. For exiv2 v0.26, I added the option
-pR to recursively print the structure of an image. By recursive, I mean that the print will dump image structures embedded in other structures. This is particularly useful in this case as Exif metadata is usually encoded as a TIFF structure. Both options
-pR are implemented by the API
For exiv2 v0.26, I have added support for ICC profiles. I'd like to support
printStructure() and ICC profiles in src/jp2image.cpp. So, although I have the fix for your issue ready, I'd like to add these additional features. More importantly, I'd like to add test files to our test suite to ensure these fixes are monitored.
This issue will be fixed and submitted to the trunk tomorrow (2016-10-12). I would really appreciate a couple of small (50k or so) JPEG 2000 files. I'd be great for those test files to have embedded Exif/IPTC/XMP/ICC metadata. I hope you have a copy of PhotoShop Creative Suite with which to create those test files. Alternatively, you could ask our good friend Phil to help us out with test data.
#5 Updated by Herbert Kauer 7 months ago
thanks for all your investigations.
You asked for test images. But I regret, I do not have many *.jp2 images.
Background: I work on a small GUI that will display and/or modify metadata via Exiftool. I found ShowFoto and hoped to be able to create e.g. some *.jp2 images for testing with my GUI. So I came to this problem.
But nevertheless I have 2 *.jp2 images, which I found somewhere in the www... I do not remember where I found it. I hope it helps
Thanks again and Best regards
#6 Updated by Robin Mills 7 months ago
- Status changed from Assigned to Closed
- % Done changed from 40 to 100
- Estimated time changed from 10.00 to 6.00
Thanks for the files. I don't think there is Exif/IPTC/XML/ICC data embedded in those files. I had an unsuccessful Google for suitable files yesterday. I have not been able to find the specification for JPEG2000. Curiously exiftool does not write ICC profiles in JP2 files. However I submitted the code update because I respect and trust Phil!
Here's what I've done about this:
1) I submitted the code and the test file Reagan.jp2 r4619
2) I have updated our test harness to examine that file on every commit and every nightly build on every platform. r4622
3) I hope to recruit a student in 2017 to work on our raw image support and test (#992).
4) I've opened a new issue (#1243) for v0.27 to implement
jp2image::printStructure() to support options:
-pX | -pS | -pC, -pR
I'm closing this issue as I believe I've dealt with your immediate concern. The follow-on matters involving
printStructure() will be handled in v0.27 under #1243.
Thank You for bringing this to my attention. Thank You for using Exiv2. Good Luck with your project.
#8 Updated by Robin Mills 7 months ago
For sure, there's a profile following the '!colr' bytes. All that scnrRGB XYZ stuff is ICC magic:
0x30 48: ...,............ -> 00 00 01 , 00 00 01 90 00 03 07 07 01 00 00 00 0x40 64: .!colr.......... -> 01 ! c o l r 02 00 00 00 00 01 16 00 00 00 0x50 80: .. ..scnrRGB XYZ -> 00 02 00 00 s c n r R G B X Y Z 0x60 96: ............acs -> 07 d1 00 01 00 01 00 00 00 00 00 00 a c s 0x70 112: p............... -> p 00 00 00 00 00 c0 00 00 00 00 00 00 00 00 00 0x80 128: ................ -> 00 80 00 00 00 00 00 00 00 00 01 00 00 00 00 f6 0x90 144: ........-....... -> d6 00 01 00 00 00 00 d3 - 00 00 00 00 00 00 00I'll attach this to #1243 and it will be very useful when that issue is serviced. The existing code in jp2image.cpp can read/write Exif/IPTC/Xmp metadata. With this file, I'm sure I can figure out adding ICC profile support. However, I'm going to keep focus on getting v0.26 finished and return to JPEG2000 in v0.27.
#9 Updated by Robin Mills 7 months ago
Thank You for this very useful file, I have been able to add ICC Profile support to jp2image.cpp. #1243 will be finished today.
I'm particularly pleased to get this done as exiv2 can write/rewrite/modify/delete ICC profiles. I don't believe ExifTool can write ICC profiles! Incidentally, Exiv2 has a friendly working relationship with Phil Harvey. He's inspirational.
#11 Updated by Robin Mills 7 months ago
Thanks for doing that, Phil. I'm very surprised that this issue has been in the code for a long time without being raised. Clearly very few people use Exiv2 with JP2. I believe Jpeg2000 has not gathered much market acceptance.
I'm glad you issue a warning. I'll update libexiv2 to do the same. Silently forgiving spec violations amounts to changing the specification and is the road to chaos.
#13 Updated by Robin Mills 7 months ago
You're welcome. Teamwork is really effective. I have no idea why it's seldom used in the office.
r4630 I've submitted the fix for exiv2 to issue a warning (on errout) when we read a file with the bad box. I've cut'n'pasted Phil's message for consistency.
724 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ exiftool -ver 10.30 725 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ exiftool ~/Downloads/Test.jp2 | grep -e Software -e UUID Warning : Reading non-standard UUID-EXIF_bad box Processing Software : digiKam-5.2.0 Software : Version 1.0 726 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $
726 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ bin/Debug/exiv2 --verbose --version --grep svn exiv2 0.25 001900 (64 bit build) svn=4629 727 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $ bin/Debug/exiv2 -pa --grep Software ~/Downloads/Test.jp2 Warning: Reading non-standard UUID-EXIF_bad box in /Users/rmills/Downloads/Test.jp2 Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored. Exif.Image.ProcessingSoftware Ascii 14 digiKam-5.2.0 Exif.Image.Software Ascii 32 Version 1.0 Xmp.tiff.Software XmpText 13 digiKam-5.2.0 728 rmills@rmillsmbp:~/gnu/exiv2/trunk/build $I hope to get the ICC profile written today and be totally finished with #1243.
#14 Updated by Phil Harvey 7 months ago
Robin Mills wrote:
I'm very surprised that this issue has been in the code for a long time without being raised. Clearly very few people use Exiv2 with JP2.
...and even fewer that use EXIF in JP2.
A bit of history:
In 2007 the "JpgTiffExif->JP2" box was proposed to the JPEG group and added to ExifTool at that time, but never ratified by the JPEG group. This was documented on the Jpeg2000 Wikipedia page, but it was edited out in 2008 (however it still lingers in some language versions of that page).
In 2011 I discovered that the Photoshop JPEG2000 plugin version 1.5 wrote EXIF with a different UUID (05 37 cd ab 9d 0c 44 31 a7 2a fa 56 1f 2a 11 3e) but the same format as the "JpgTiffExif->JP2" box (ie. no "Exif\0\0" header), so I added support for reading this, but ExifTool continued to write the earlier EXIF UUID box. [I just tested this with the JPEG2000 plugin 2.0 and it still does the same thing.]
#15 Updated by Robin Mills 7 months ago
That's interesting. Do you have a test file for that guid? It sounds like a tiny change to support both the 16 byte "JpgTiffExif->JP2" and the 16 byte UUID "05 37 cd ab 9d 0c 44 31 a7 2a fa 56 1f 2a 11 3e".
I will only forgive/warn for Exif\0\0 with "JpgTiffExif->JP2". For the "approved UUID", I'll throw an exception if Exif\0\0 is present. The reasons to "forgive and warn" are:
- To avoid the embarrassment of admitting that Exiv2 generated illegal files.
- I don't want to provide a repair utility for files damaged by exiv2
- When the file is to be rewritten it will be repaired.
None of these reasons apply to the UUID. So, I will not forgive/warn for the UUID.