Exiv2

There are other tags such as Exif.CanonCs.ImageStabilization (returns 65535), Exif.CanonCs.SpotMeteringMode (returns 65535), Exif.CanonCs.Sharpness (returns 32767) that are also incorrect.

They are also signed short in the exiv2 documentation and return shorts.

In summary, there seems be a bigger bug, probably the same one that is affecting all these variables that incorrectly interprets signedShort variables as short.

I would appreciate this getting fixed soon after 0.26 if we are trying get that out.

Hi Robin,

Sorry for the multiple updates.

I feel like I have more information now.

When I ask exiftool to report on such images (where I get 65535), those tags are not reported unless you pass along a "-v" switch.

This along with those values not being defined (i.e. there is no interpretation for -1), makes me feel that these are tags that are not "written" and thus contain default values.

This reinforces my earlier suggestion of creating a -1, but interpreting it as a new key that we introduce - say Unknown.

I can confirm (for ImageStabilization at least), the lens in question does not have it (and neither does the camera). So when exiv2 returns -1 for that image for this field, it should be interpreted as Unknown. If you can think of a better name for this, please do suggest.

I had suggested "Off" in my original bug report above, but now feel that something like "Unknown" is better because it transcends categories like Exif.CanonCs.SpotMeteringMode and Exif.CanonCs.Sharpness.

Thanks,
Sridhar

Sridhar. I'll put in a generic fix for this. I spent another 2 hours yesterday evening without figuring it out. It will have to wait for 0.27 because I'm fully already fully loaded for v0.26 and I took on another substantial task for v0.26 this week: #1199.

I do not support your suggestion about interpreting -1 as "Unknown" or "Off". I don't want to introduce guesses. Do we reserve "Unknown" for -1, or for every value we don't know? We already have a way in which we report it and I want to leave it alone without have solid information. For example, I'm happy to change 8 to Manual based on Phil's website.

I'm always looking for engineers to help with Exiv2. Are you available and have good C++ skills?

Hi Robin,

Yes, a generic fix is required in this case. Returning -1 and realizing that it is a signed short is better than the way it currently is.

Waiting for 0.27 is not a problem.

However, we need to understand why exiftool segregates between reporting -1 when verbose is specified and not reporting the value at all in the usual case. I feel that this is the crux of the issue. My guess is that FFFF is some code to designate all "Unknown" values - i.e. like when the makernote area is memset with FF and the other values are over written as they are discovered. I have raised this question in the exiftool forum (http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,7446.0.html), let's see if we get any answers. We can look at the exiftool code to understand this as well, but my perl interpretation skills might not get us the answer quickly.

More interesting is the Sharpness case. The value returned is 32767 - a valid signed short, but reported as 3 by exiftool - a value that I recognize from the Canon menus as correct. Now, the verbose exiftool output shows 32767 - maybe a mistake there in that the data is not being interpreted but just reported blindly. Maybe that's how verbose is supposed to behave, but I don't like it. I would have preferred that the output was consistent in both cases and the "final" values reported would the interpreted values and the raw output can be shown as additional data.

I am good in C++ and with some guidance on where to start looking can fix these kind of issues, but my bandwidth is uncertain. I am working on a personal project dealing with keeping my website in sync with my Picasa database and thus have gotten into interpreting tags etc. If you would like, you could make me a developer, but I can't promise reserved bandwidth to exiv2.

Thanks,
Sridhar

I guess my perl interpretation skills were enough (it was easy).

Please take a look a Image-ExifTool-10.25/lib/Image/ExifTool/Canon.pm in the exiftool source (I am using v10.25, latest as of today). Line #2016 (Sharpness) shows that 7FFF means undefined. Similarly line 2208 (SpotMeteringMode) clearly shows -1 is undefined, same for ImageStabilization (#2183) and FocusContinuous (#2163).

It is also apparent that FFFF (-1) means undefined for a lot of the other tags and 7FFF is also a special value for some of them.

Now, exiftool differentiates by reporting the RawValue (its terminology) in the verbose mode and not reporting an undefined value in normal mode. I feel that exiv2 should a "better" mix of the two - i.e. retrieve -1 correctly, but interpret it as undefined. This could mean an additional function in Exiv2::Metadatum that could let us know that the retrieved data is undefined and should not be used.

Regards,
Sridhar

I'm happy to mentor you in the ways of the force. Adding stuff like 8/Manual is almost trivial (#1202). And we should add something to the test suite for #1202. I'll probably do that over the weekend and then you'll have a model of what to do with stuff like that. I can't grant you write access to the Exiv2 repository, Andreas (the project founder) reserves that (and other) privileges. When new contributors volunteer, we normally ask them to provide patches for changes as we get to know each other.

http://dev.exiv2.org/projects/exiv2/wiki/Contributing_to_Exiv2
http://dev.exiv2.org/projects/exiv2/wiki/Guidelines_for_submitting_patches

#1203 is difficult. I spent a couple of hours on that last night. Not a job for a newbie.

I've added Phil (of exiftool fame) to the watchers of this bug. Phil: Can you give any insight concerning -1/unsignedShort metadata values?

Sounds fine regarding contributing patches etc.

I have been corresponding with Phil separately on the same question. His answers are quoted below.

Summary: -1 is undefined and is not shown in exiftool output. 7fff is also undefined and is not shown. The sharpness value of 3 that I see in exiftool is actually Exif.CanonPr.Sharpness - or at least something like this. I say "something like this" because I do not see a CanonPr category in the exiv2 canon tags documentation. Though this is in the code.

I will use that going forward.

However, fixing Issue #1203 the right way would involve introducing terminology like I suggest above to tell users that these values that are being returned are really not defined and should not be used. Or, we go the exiftool way and do not return a value at all - i.e. set the returned iterator to ed.end(). It might be easier to accomplish the latter.

Regards,
Sridhar

Hi Sridhar,

Sorry, I missed this in the email, but I did respond about this in the forum post.  Also, here is my response to the exiv2 bug email (but I'm not sure if an email response is valid for this bug list because it hasn't shown up on the web page yet):

On Aug 12, 2016, at 8:38 AM, Phil Harvey <phil@...> wrote:

Nothing further than what I mentioned here:

http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,7446.msg37852.html#new

- Phil

> On Aug 12, 2016, at 8:08 AM, Sridhar Boovaraghavan <sridhar_ml@...> wrote:
>
> Thanks, Phil.
>
> Could you please comment on the Sharpness case?
>
> After sending the original e-mail, I went to see the underlying perl source and see that Sharpness values of 7fff is undefined. How is that interpreted as 3 in the non-verbose output? Or is it another sharpness value that is being interpreted?
>
> Regards,
> Sridhar
>
>
> From: Phil Harvey <phil@...>
> To: Sridhar Boovaraghavan <sridhar_ml@...>
> Sent: Friday, 12 August 2016 5:19 PM
> Subject: Re: ImageStabilization/Sharpness/SpotMetering - verbose output vs non-verbose
>
> Hi Sridhar,
>
> Some values in the metadata indicate "not applicable" or similar.  Typically these are -1.  The -v output shows these raw values, but ExifTool may sometimes choose to not generate a tag when this happens.  For a bit more insight, see the Value Conversions section of this page:
>
> http://owl.phy.queensu.ca/~phil/exiftool/under.html#conversions
>
>     - Phil
>
>> On Aug 12, 2016, at 3:26 AM, Sridhar Boovaraghavan <sridhar_ml@...> wrote:
>>
>> Hi Phil,
>>
>> I would like to understand the difference between values returned with the verbose option and those without.
>>
>> I am attaching the output of exiftool -a of two images along with the second file containing the output of exiftool -a -v3 on those same images.
>>
>> Specifically, I see ImageStabilization reported as -1 in the verbose output and not reported at all in the "normal" output.
>>
>> In both these images, ImageStabilization was not available and thus, I understand the "normal" output to be "correct".
>>
>> How am I to interpret the verbose output?
>>
>> Is FFFF some kind of default value stored in fields that are not used and thus those values should be "discarded" or ignored?
>>
>> For Sharpness, it is a more interesting question. The normal output returns a sharpness of 3 - which is correct. The verbose output returns 7FFF (or 32767). How is this translated to 3?
>>
>> Thanks,
>> Sridhar

Here is a list of explicit tag values that exiftool ignores. The resolution of this issue should address this as well.

Note that this doesn't encompass other cases where values that are <= 0 are being ignored. Ideally, source:canonmn.cpp should do the latter as well. I think it does do this in many cases, but a check with Canon.pm of exiftool might show up cases that it doesn't.

Exif.CanonCs.RecordMode -1
Exif.CanonCs.Contrast 0x7fff
Exif.CanonCs.Saturation 0x7fff
Exif.CanonCs.Sharpness 0x7fff
Exif.CanonCs.CameraISO 0x7fff
Exif.CanonCs.AFPoint 0
Exif.CanonCs.FlashActivity -1
Exif.CanonCs.FocusContinuous -1
Exif.CanonCs.AESetting -1
Exif.CanonCs.ImageStabilization -1
Exif.CanonCs.SpotMeteringMode -1
Exif.CanonCs.PhotoEffect -1
Exif.CanonCs.ColorTone 0x7fff
Exif.CanonCs.SRAWQuality -1
Exif.CanonSi.FlashGuideNumber -1
Exif.CanonSi.AFPointsInFocus 0
Exif.CanonMi.FrameRate 65535
Exif.CanonMi.FrameCount 65535
Exif.CanonFi.FilterEffect -1
Exif.CanonFi.ToningEffect -1

Are you asking for those tags to be ignored by Exiv2, or to be not reported when they have those specific values?

We seem to be discussing several different things simultaneously:
1) The signed/unsigned conflict
2) How to "translate" undefined values such as -1
3) The list of tags above

Can you open different issues for those topics. Redmine has a feature for "Related Issues" which can be used to group these matters together. You may wish to open an "Umbrella" which is a virtual issue to collect numerous related issues. The title could be "Umbrella: Canon MakerNote Undefined Values"

As requested, I have created Issue #1206 to cover the case of ignoring specific values for specific Canon makernote tags.

#2 and #3 should be addressed by Issue #1206.

It will be good if Issue #1203 stuck to resolving why these tags return unsigned short and fixing that.

Sorry, Robin. I think you are talking about #1204 and #1205.

Thanks for offering to update the canonical test results to match the newly-set defaults that ignore undefined values.

However, #1203 is still a problem. We have requested Andreas to address this.

Thanks,
Sridhar

The investigation of the test harness anomalies introduced by r4438 is being deal with in #1218.