Bug #1035

Lens model not detected ( exiv2 -> LensFun -> darktable )

Added by Rodrigo De Leon over 2 years ago. Updated over 1 year ago.

Status:ClosedStart date:19 Feb 2015
Priority:NormalDue date:
Assignee:Robin Mills% Done:

100%

Category:metadataEstimated time:6.00 hours
Target version:0.26

Description

Hi.

Background details:
1) http://www.darktable.org/redmine/issues/10331
2) https://sourceforge.net/p/lensfun/bugs/18/

Is there anything that can be done in exiv2 to fix this?

Thanks for your time.

Regards.

19mm.tar.gz_part_aa (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ab (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ac (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ad (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ae (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_af (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ag (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ah (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ai (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_aj (19 MB) Rodrigo De Leon, 14 Jan 2016 20:58

19mm.tar.gz_part_ak (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_al (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_am (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_an (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_ao (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_ap (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_aq (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_ar (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_as (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_at (19 MB) Rodrigo De Leon, 14 Jan 2016 21:01

19mm.tar.gz_part_au (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_av (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_aw (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_ax (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_ay (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_az (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_ba (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_bb (19 MB) Rodrigo De Leon, 14 Jan 2016 21:05

19mm.tar.gz_part_bc (4.44 MB) Rodrigo De Leon, 14 Jan 2016 21:05

30mm.tar.gz_part_aa (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ab (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ac (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ad (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ae (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_af (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ag (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ah (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ai (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_aj (19 MB) Rodrigo De Leon, 14 Jan 2016 21:08

30mm.tar.gz_part_ak (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_al (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_am (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_an (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_ao (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_ap (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_aq (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_ar (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_as (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_at (19 MB) Rodrigo De Leon, 14 Jan 2016 21:13

30mm.tar.gz_part_au (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_av (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_aw (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_ax (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_ay (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_az (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_ba (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_bb (19 MB) Rodrigo De Leon, 14 Jan 2016 21:16

30mm.tar.gz_part_bc (10.4 MB) Rodrigo De Leon, 14 Jan 2016 21:16


Related issues

Related to Exiv2 - Bug #1143: Unable to extract embedded preview from jpg for Sony a77 Closed 24 Dec 2015

Associated revisions

Revision 4189
Added by Robin Mills over 1 year ago

#1035 test files

History

#1 Updated by Niels Kristian Bech Jensen over 2 years ago

Sony makernote support is in serious need of an update to support E-mount lens detection amongst other things. I do not know when such an update will happen due to lack of time.

Best regards,
Niels Kristian Bech Jensen

#2 Updated by Robin Mills over 2 years ago

  • Category set to metadata
  • Status changed from New to Assigned
  • Assignee set to Robin Mills
  • Target version set to 0.26

I'm going to accept this assignment for v0.26 as we can't consider this for v0.25. I'll have to investigate what's involved in this. So, all I am agreeing to do for v0.26 is consider the effort involved. If it's a lot of work, it might be pushed beyond v0.26.

#3 Updated by Robin Mills over 2 years ago

  • Target version changed from 0.26 to 0.25

#4 Updated by Robin Mills over 2 years ago

  • Target version changed from 0.25 to 0.26

Pushed back to v0.26. I don't why I moved this to v0.25 yesterday.

#5 Updated by Robin Mills over 2 years ago

  • Assignee deleted (Robin Mills)

#6 Updated by Robin Mills almost 2 years ago

  • Assignee set to Robin Mills

#7 Updated by Robin Mills over 1 year ago

  • % Done changed from 0 to 10
  • Estimated time set to 10.00

I'm adding a SWAG to my outstanding issues for v0.26 to get a more accurate assessment of the status of v0.26. I can't remember anything about this - so I'll guess about a day's work.

#8 Updated by Robin Mills over 1 year ago

  • % Done changed from 10 to 50
  • Estimated time changed from 10.00 to 5.00

Good News, I believe I can fix this. It's almost identical to #1153 in which .ARW file correctly identifies the Lens and the .JPG file does not. Please attach the DSC00050.JPG (to match DSC00050.ARW) and I'll fix this.

It doesn't have to be DSC00050.JPG - however I need matched pairs of JPG/ARW for every camera/lens combination. It's also very useful to shoot ARW/JPG pairs at every aperture setting of the camera/lens combo.

The lens is correctly identified in the RAW image:

1678 rmills@rmillsmbp:~ $ exiv2 -pa DSC00050.ARW  | grep -i lens
Exif.Sony2.LensID                            Long        1  Manual lens
Exif.Photo.LensSpecification                 Rational    4  190/10 190/10 28/10 28/10
Exif.Photo.LensModel                         Ascii      12  E 19mm F2.8
$ 
However, the LensID is ambiguously set to 0xffff (65535).
1679 rmills@rmillsmbp:~ $ exiv2 -pv DSC00050.ARW  | grep -i lens
0xb027 Sony2        LensID                      Long        1  65535
0xa432 Photo        LensSpecification           Rational    4  190/10 190/10 28/10 28/10
0xa434 Photo        LensModel                   Ascii      12  E 19mm F2.8
$ 

#9 Updated by Robin Mills over 1 year ago

  • Estimated time changed from 5.00 to 6.00

#10 Updated by Rodrigo De Leon over 1 year ago

Hi.

Thanks for the feedback.

I no longer have the original DSC00050.JPG, but I can take a new set of photos and upload them.

Don't remember if this only happen with the 19mm or also with the 30mm, but I'll upload both sets.

Just give me a few minutes...

#11 Updated by Rodrigo De Leon over 1 year ago

Hi.

There seems to be a 20 MB file upload limit for this issue tracker, but all ARW files measure about 24-25 MB.

I'm gonna have to archive and split to be able to upload.

Hang on...

#12 Updated by Rodrigo De Leon over 1 year ago

Batch 1...

#13 Updated by Rodrigo De Leon over 1 year ago

Batch 2...

#14 Updated by Rodrigo De Leon over 1 year ago

Batch 3...

#15 Updated by Rodrigo De Leon over 1 year ago

Batch 4...

#16 Updated by Rodrigo De Leon over 1 year ago

Batch 5...

#17 Updated by Rodrigo De Leon over 1 year ago

Batch 6 (final)...

#18 Updated by Rodrigo De Leon over 1 year ago

Done.

Let me know if you need something else.

Cheers.

#19 Updated by Robin Mills over 1 year ago

Thanks, Rodrigo. It's 23:03 in England. I'm off to bed. I'll look at this tomorrow.

#20 Updated by Robin Mills over 1 year ago

Apologies, Rodrigo. It's been a very busy day and I didn't get to your issue. I will deal with it on Saturday.

I had another thought however! Can you get your camera to take lower resolution photos which are within the 15mb limit? After all, I'm only interested in the 50k of metadata - the rest of the file isn't useful!

#21 Updated by Robin Mills over 1 year ago

  • % Done changed from 50 to 70

Rodrigo

Thanks for the files. I've got them, unzipped them and submitted them to our test file repository: svn://dev.exiv2.org/svn/testdata/trunk/1035

What is your expectation here? We are detecting Exif.Photo.LensModel as:

520 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ exiv2 -pa -K Exif.Photo.LensModel *.JPG
DSC05005.JPG          Exif.Photo.LensModel                         Ascii      12  E 30mm F2.8
...
DSC05043.JPG          Exif.Photo.LensModel                         Ascii      12  E 19mm F2.8
521 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ exiv2 -pa -K Exif.Photo.LensModel *.ARW
DSC05005.ARW          Exif.Photo.LensModel                         Ascii      12  E 30mm F2.8
...
DSC05043.ARW          Exif.Photo.LensModel                         Ascii      12  E 19mm F2.8
522 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ 
I see exiftool reports it's a Sigma Lens:
$ for i in DSC05043.*; do echo --- $i ---- ; (exiftool -all $i| grep -i Lens) ; done
--- DSC05043.ARW ----
Lens Type                       : E-Mount, T-Mount, Other Lens or no lens
Lens Spec                       : E 19mm F2.8
Lens Zoom Position              : 0%
Lens Mount 2                    : E-mount
Lens Type 3                     : Unknown E-mount lens or other lens
Lens E-mount Version            : 1.00
Lens Firmware Version           : Ver.01
Lens Mount                      : E-mount
Lens Format                     : APS-C
Lens Type 2                     : Unknown E-mount lens or other lens
Lens Spec Features              : E
Lens Info                       : 19mm f/2.8
Lens Model                      : E 19mm F2.8
Lens ID                         : Sigma 19mm F2.8 [EX] DN
--- DSC05043.JPG ----
Lens Type                       : E-Mount, T-Mount, Other Lens or no lens
Lens Spec                       : E 19mm F2.8
Lens Zoom Position              : 0%
Lens Mount 2                    : E-mount
Lens Type 3                     : Unknown E-mount lens or other lens
Lens E-mount Version            : 1.00
Lens Firmware Version           : Ver.01
Lens Mount                      : E-mount
Lens Format                     : APS-C
Lens Type 2                     : Unknown E-mount lens or other lens
Lens Spec Features              : E
Lens Info                       : 19mm f/2.8
Lens Model                      : E 19mm F2.8
Lens ID                         : Sigma 19mm F2.8 [EX] DN
524 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ 
For sure, the 'Sigma' is not in the metadata:
527 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ exiv2 -pa --grep lens/i DSC05043.*
DSC05043.ARW          Exif.Sony2.LensID                            Long        1  Manual lens
DSC05043.ARW          Exif.Photo.LensSpecification                 Rational    4  190/10 190/10 28/10 28/10
DSC05043.ARW          Exif.Photo.LensModel                         Ascii      12  E 19mm F2.8
DSC05043.JPG          Exif.Sony1.LensID                            Long        1  Manual lens
DSC05043.JPG          Exif.Photo.LensSpecification                 Rational    4  190/10 190/10 28/10 28/10
DSC05043.JPG          Exif.Photo.LensModel                         Ascii      12  E 19mm F2.8
528 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ 
However it could be deduced from other metadata such as the crop factor (or the LensSpecification above):
 530 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ exiv2 -pa --grep focal/i DSC05043.*
DSC05043.ARW          Exif.Photo.FocalLength                       Rational    1  19.0 mm
DSC05043.ARW          Exif.Photo.FocalLengthIn35mmFilm             Short       1  28.0 mm
DSC05043.JPG          Exif.Photo.FocalLength                       Rational    1  19.0 mm
DSC05043.JPG          Exif.Photo.FocalLengthIn35mmFilm             Short       1  28.0 mm
531 rmills@rmillsmbp:~/gnu/exiv2/testdata/trunk/1035 $ 
I've never used DarkTable nor LensFun, so I don't understand the information you reported to them. I've checked back a few years to exiv2 v0.23 (about 2012 or so). exiv2 v0.23 reports the same lens data as the current trunk - so upgrading the version of in DarkTable and/or LensFun will change nothing.

I've looked at the exiftool lens identification code and I don't know how then discovered it was a Sigma Lens, I can put more effort into that - or cheat and ask Phil! The (perl) code is here: http://dev.exiv2.org/issues/1143#note-11

However, before I roll up my sleeves, or involve anybody else, I'd like to know what you are looking for here.

#22 Updated by Rodrigo De Leon over 1 year ago

Hi.

Really sorry for the late update.

OK, to summarize:

  1. darktable is a raw photo developer, similar to lightroom, capture one, etc.
  2. they use exiv2 for lens identification and lensfun for lens correction (distortion, TCA, vignetting). Detailed info here:
  3. they match lens name to (apparently) normalized exiv2 names, list here:
  4. for the 19mm, 30mm and 60mm Sigma lenses, there seem to be no Sony entries:
  5. there seem to be valid entries for these lenses on the Olympus side, though:
  6. so, basically, the "fix" is for exiv2 to return the normalized names for the Sony version of those lenses, so that they can be matched against the names on the lensfun list here:

Hope that was clear enough.

Anything else, just let me know.

Cheers.

#23 Updated by Rodrigo De Leon over 1 year ago

Just for for completeness, here are the expected normalized names (complete lens list, here -> http://lensfun.sourceforge.net/lenslist/):

19mm:
  • Sigma 19mm f/2.8 EX DN
30mm:
  • Sigma 30mm f/2.8 EX DN
60mm:
  • Sigma 60mm f/2.8 DN

Cheers.

#24 Updated by Robin Mills over 1 year ago

Thanks, Rodrigo

Niels has been dealing with makernote/lens matters and has started a college course in addition to his very demanding job. So, I agreed last September to take over this stuff. It's been quite a learning curve to understand how this works and fix a few lens. I've reached a conclusion: http://dev.exiv2.org/issues/1160#note-1

As I've started to work on your stuff, I'll have a look at this at the weekend. If you have any thoughts about my plan to use exiftool over a web service, I'm interested to hear your comments.

Robin

#25 Updated by Rodrigo De Leon over 1 year ago

Interesting.

If I understand correctly:

  • that method would be for initial identification (exiv2 does not initially know of the lens)?
  • it's required to be online.
  • afterwards, the info is incorporated to exiv2 for the next release?

Or am I understanding wrong?

Cheers.

#26 Updated by Robin Mills over 1 year ago

I think your summary is about correct. I'm saying that when the LensID is unique, we will identify it. When we don't know what it is, we'll use a webservice to ask exiftool. I will maintain and add new lens which have a unique LensID.

I don't know how lens recognition arrived in the code base. We read/write metadata. The meaning of that metadata isn't our concern. Guessing the lens from the focal length/aperture doesn't feel good. And it's consuming a lot of my time. At the moment, I'm performing about 70% of the work on Exiv2 and it adds up to an unpaid full-time job. I'd like to reduce the burden that I am carrying.

#27 Updated by Robin Mills over 1 year ago

  • Status changed from Assigned to Resolved
  • % Done changed from 70 to 100

I've added a new feature with #1034 to read from the file ~/.exiv2. For example:

[sony]
255=My wiz-bang new lens

I intend to add the web-service feature, however this ~/.exiv2 will be faster and can be customized/localized as you think best.

#28 Updated by Robin Mills over 1 year ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux