Exiv2

This is an interesting matter. Exif metadata is stored as a TIFF and embedded in the JPEG. JPEG files consists of segments which are restricted to 64k max. So, thumbnails are usually small so they and the metadata all live within the 64k limit. The Exif specification forbids the writer from referencing data outside the 64k segment. However, I believe some Sony images break the rule. This is not a good idea because it makes it almost impossible to rewrite the file. So any changes to the metadata will probably result in the preview being lost. I can easily imagine that an Engineer at Sony didn't understand the importance of ensuring that the embedded TIFF must not have an external reference because they are not involved with editing metadata.

If you only wish to read the preview, I may be able to provide a workaround. The command $ exiv2 -pR image-path will "print recursively" and enable you to find the "illegal" preview and the utility dd can extract it. The whole thing can be automated into a short bash script.

Another work-around is to use the Raw ARW file and avoid JPEG.

If you could provide a sample image from your camera, I'll be happy to investigate this for you.

The Exiv2 project has moved to https://github.com/exiv2/exiv2. I haven't yet found a way to make our Redmine server "Read only". May I ask you to open a new issue on GitHub about this for two reasons:

1) More folks read and may join this discussion.
2) A member of Team Exiv2 has accepted the challenge BigTiff (64 bit tiff) for Exiv2 v0.28. He'll be very interested in this matter.

A few years ago, I started to investigate these Sony "external" references. That was my first journey into class TiffReader and I didn't have enough knowledge to preserve those previews from JpegBase::writeMetadata(). However since then, I've solved some puzzles in that class TiffReader, and the time may have come to revisit this topic.

http://dev.exiv2.org/boards/3/topics/1131?r=1137#message-1137

However, the most immediate matter is to a good test file.

Hi Robin,

Thanks for the fast reply. It is not just Sony doing, I think most camera manufacturers are doing something similar.

Here is the link to a Fujifilm JPEG:

https://www.dropbox.com/s/xdnaaprfkpd20vz/XH1F5365.JPG?dl=0

Inside, there is a 320x240 preview image, I can extract it with ExifTool with the -PreviewImage switch. But how do I extract that in a C++ program using Exiv2?

I want to do this with Exiv2 for performance reason.

Thanks,
Albert

Here is the link to a Sony A73 JPEG:

https://www.dropbox.com/s/70lwzp4wk6criv2/DSC00672.JPG?dl=0

Inside, there is a 1610x1080 preview image. Also, I believe all these preview images are JPEGs, not TIFFs.

Thanks for the files. This is going to take some effort at my end. I added "print recursively/-pR" in Exiv2 v0.26 for several reasons:

1) For debugging files.
2) To help me "really" understand tiff.

The code in class TiffReader is inspirational and really hard to understand. We've been asked to support BigTiff (64 bit tiff) and class TiffReader cannot be refactored without understanding. I'm almost there with understanding. class TiffReader is really really good.

However when I wrote "print recursively/-pR" in 2015, I didn't know how many of the MakerNotes work. They are usually another embedded tiff. I know much more about that now. For sure the current code isn't dealing well with your two files. So, I don't have a work-around for you yet. However I am motivated to work on this.

I'm off to the States tomorrow to visit family and enjoy some winter sunshine in Florida. I will spend time on this. I encourage you to report this on GitHub. I don't expect anybody else will jump into this topic, however that's a more visible place to discuss this important matter.

I emailed Phil (author of ExifTool), and he said the PreviewImage are stored in MakerNotes, and different manufacturers do this differently.

He told me to try the -htmldump -v2 switches. I did that on the 2 files above and I simply redirect (>) the output to a html file.

Open this html file in a browser, and you can move your mouse to a tag and it'll highlight the data area for you (very nice). It really does help to see what's inside the JPEG file.

I can see that the Sony preview image is in the MPImage2:MPImage data area. The data size confirms it, as it is the same as the preview image size extracted by ExifTool.

The Fuji one is a bit tricky. I think it is embedded in the APP2 FPXR segment. The data in this area closely resemble the size of the preview image size. See:

https://help.accusoft.com/ImageGear/v18.2/Windows/ActiveX/IGAX-10-23.html

I'm sure the Fuji Preview image is there, because I can see the JPEG header visually :)

The question is: is there a way for Exiv2 to extract the data in those specific tags? If yes, then it won't be too hard to extract this preview image in the MakerNotes.

Thanks. Phil often reads the correspondence concerning Exiv2 and offers advice. He's very helpful. And I often use ExifTool's -v5 option to investigate how he is able to locate data that Exiv2 does not find.

Please understand that the Open Source Community gets no assistance from Camera Manufacturers. We have to deal with this by reverse engineering their files.

The question is: is there a way for Exiv2 to extract the data in those specific tags? If yes, then it won't be too hard to extract this preview image in the MakerNotes.

It's not possible today. However I'm going to work on "print recursively/-pR" and get it correctly dump your files.

I can see the APP2 segments in the Fuji file.

617 rmills@rmillsmbp:~/gnu/github/exiv2 $ exiv2 -pS ~/Downloads/Albert/XH1F5365.JPG 
STRUCTURE OF JPEG FILE: /Users/rmills/Downloads/Albert/XH1F5365.JPG
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   65468 | Exif..II*......................
   65472 | 0xffe2 APP2  |     136 | FPXR........../.F.U.J.I.F.I.L.M
   65610 | 0xffe2 APP2  |      52 | FPXR.......................@...
   65664 | 0xffe2 APP2  |   20864 | FPXR...........................
   86530 | 0xffc4 DHT   |     418 
   86950 | 0xffdb DQT   |     132 
   87084 | 0xffdd DRI   |       4 
   87090 | 0xffc0 SOF0  |      17 
   87109 | 0xffda SOS  
618 rmills@rmillsmbp:~/gnu/github/exiv2 $ 

Hi Robin,

I found that Canon does not embed Preview Image in their JPEG (I looked at 5DIII files, ExifTool does not report any preview image). Nikon does, and it seems that the preview image is always in the same place (PreviewIFD:PreviewImage data).

I also learn today that Exiv2 does not read HEIC files. And I don't know how to get it to read a video file, say I want to get the dimension of the video. I've compiled the source with video option turned ON, but still no luck.

Of course I understand this is Open Source stuff, so it is unrealistic to expect a lot of help. But your replies are certainly very insightful.

Anyway, thanks for the help. Please enjoy your holidays with your family and don't worry about this too much until you have more time.

Albert

I'm happy to help you. I'm 68 and retired. Working on Exiv2 keeps my brain active.

HEIC files. The Apple/iOS native format? We don't support that.

Reading video should work. Which version of the code are you using?

I'm using the latest code.

Yes, HEIC is the Apple iOS native format. ExifTool from Phil Harvey can read the EXIF metadata off of it, but won't allow the values to be edited.

Exiv2 can't read HEIC. I wonder will HEIC be supported ever?

And to Robin: have you had a chance to find out how to extract the preview image yet? It'd be nice if it can be extracted, it'd make creating preview image so much faster than reading the entire JPEG, especially cameras these days have very high MP count.

Apologies. I'm still on vacation and forgot about your requests. Can I ask you to raise two issues on https://github.com/exiv2/exiv2. We'd like to make Redmine "read-only" and deal with users on github.com

1) Previews in your files.

Please link this discussion to your issue report.

743 rmills@rmillsmbp:~/clanmills $ dir ~/Downloads/Albert/
-rw-r--r--@ 1 rmills  staff    12M 22 Jan 09:54 /Users/rmills/Downloads/Albert/XH1F5365.JPG
-rw-r--r--@ 1 rmills  staff   7.0M 22 Jan 09:54 /Users/rmills/Downloads/Albert/DSC00672.JPG
744 rmills@rmillsmbp:~/clanmills $ exiv2 -pa --grep make/i ~/Downloads/XH1F5365.JPG 
Exif.Image.Make                              Ascii       9  FUJIFILM
Exif.Photo.MakerNote                         Undefined 1056  (Binary value suppressed)
Exif.MakerNote.Offset                        Long        1  2176
Exif.MakerNote.ByteOrder                     Ascii       3  II
Exif.Photo.LensMake                          Ascii      13  FUJIFILM
745 rmills@rmillsmbp:~/clanmills $ exiv2 -pa --grep make/i ~/Downloads/DSC00672.JPG 
Exif.Image.Make                              Ascii       5  SONY
Exif.Photo.MakerNote                         Undefined 37520  (Binary value suppressed)
Exif.MakerNote.Offset                        Long        1  994
Exif.MakerNote.ByteOrder                     Ascii       3  II
Exif.Thumbnail.Make                          Ascii       5  SONY
746 rmills@rmillsmbp:~/clanmills $ 

2) HEIC Support

I believe somebody else has raised this. However support for BigTiff and CR3 are higher priority. I don't when, or indeed if, HEIC will be supported. When Apple announced this about 2 years ago, I got the impression that HEIC files are only used for image storage on iOS. The HEIC file can be exported into other formats (JPEG, Tiff etc) for external use.

I would be very helpful if you could provide HEIC example files.

I've just added a new issue for extracting preview image in JPEG.

As for HEIC, there is already an open issue:

https://github.com/Exiv2/exiv2/issues/318

Support for HEIF is the same as HEIC.

BTW, how do I attach a sample HEIC file in GitHub?

Thanks for getting back to me. I believe you can drag'n'drop files into issues on GitHub. Or you put them on a public file server (such as Dropbox) and publish the URL.

GitHub does not support uploading HEIC file :(

So I added a Dropbox link to a HEIC file in the comment.

https://github.com/Exiv2/exiv2/issues/710