Exiv2

Actually the status quo is really annoying. Mots times I svn update the exiv2 repo I can see makernote updates corresponding to camera/lense changes. Since exiv2 is released very rarely this means the hard-coded camera/lense data are out of sync most of the time exiv2 is used when it comes to new camera/lense models.

The enhanced feature actually is fairly simple to implement. One should decide on how to textual store the camera/lens data and how and when to read it from file. This is no big issue at all. One thing I have in mind is performance though. As a user of exiv2.lib (as opposed to exiv2.exe) (Yes, Win user I am.) I can easily life with reading the camera/lens data from file only once at the life time of the application. But having exiv2.exe as a stand-alone command-line tool reading this file with every start may decrease performance. So maybe one should think of splitting this feature for exiv2.lib and exiv2.exe.

Why 1.0? Milestone 1.0 translates to never (in my opinion as observer and part-time contribuor of exiv2). Let's see what Niels suggests for the design and discuss it here.

Are you offering to undertake this project? I don't want to commit this to the v0.26 release and I haven't given a single thought to v0.27. So, I've declared it as v1.0.

A few thoughts about this. Not a real design specification, but more like a unstructured brain-storm. I think it can be developed in two stages:

1. I think such a system should be designed around XML. It is a mature and portable format (stage 1).
2. The text files should be considered temporary, meaning that they should be converted to source code and included in each release (stage 1).
3. The XML file(s) should supercede built in lens definition with same lens ID number to enable setting a "prefered lens" when several lenses share the lens same ID number (stage 1).
4. The XML file(s) must be placed in a standard directory structure with priority to the users home directory. On Linux/Unix systems the search order should be: ~/.local/share/exiv2, /usr/local/share/exiv2, /usr/share/exiv2. Similar standards should be followed on other systems.
4. Scripts for automatic conversion of XML files to source code should be developed (stage 2). Perl or Python could be used for this and the next item.
5. Scripts for generation of XML files from ExifTool updates should be developed (stage 2).

Only camera models and lens models should be defined this way. All other metadata tags should be hardcoded for safety reasons.

Please comment on this and volunteer to implement it.

Best regards,
Niels Kristian Bech Jensen

If I read it correctly you want to have the XML data on top of the hard-coded data and not instead, right?

My idea is that the library should search the overflow files first and the hardcoded data second. For releases we will only include hardcoded data.

Best regards,
Niels Kristian Bech Jensen

And the discussion here.
http://www.darktable.org/2015/02/on-lens-detection-and-correction/comment-page-1/#comment-42822

Pascal's comment you're replying to is spot-on.

Re Niels' thoughts:

1. I think such a system should be designed around XML. It is a mature and portable format (stage 1).

Very much agree this should be limited to lenses. So if the format is simple, I'd prefer a simple text file that can be parsed without an XML parser. (To avoid the dependency on Expat - currently only needed for XMP - and the parsing overhead it introduces.)

Besides my main concerns are

a. Technical: This will require the library to perform some kind of one-time initialization, something which is ugly and hairy. We do have a precedent for that for XMP, but it's really not something I'd like to promote.
b. Conceptual: Chances are we won't get feedback from users anymore for lens definitions, they'll all just update their own files. So we'll rely on Exiftool completely. What if Phil adds a similar feature?

My idea is that the library should search the overflow files first and the hardcoded data second. For releases we will only include hardcoded data.

Agreed

b. Conceptual: Chances are we won't get feedback from users anymore for lens definitions, they'll all just update their own files. So we'll rely on Exiftool completely.

This is of cause pure guessing and I would expect the opposite. I expect more users to contribute new lenses, because they can test it directly and benefit faster. I wrote the long version of this argument here:
http://www.darktable.org/2015/02/on-lens-detection-and-correction/comment-page-1/#comment-42822

At the moment I, as a user, have to wait ~1 year or longer to get benefit from a bugreport with a new lens. (New exiv2 version + new distribution version) I think in Exiftool I can very easily add a lens to the Perl file. (Not tested, my lens was already there.)

Is it at all possible to map exiv2's lens detection data to a not-too-complicated XML schema? As a related question, are we talking about moving the whole of exiv2's lens detection data into external text files, or only about infrastructure to add simple cases ad hoc?

It is meant as an ad hoc solution to add new lenses between releases.

Best regards,
Niels Kristian Bech Jensen

Here's another idea to consider for v0.25+.

We can create a new branch 'stable' which will begin life as a copy of v0.25. When any maker-note code is modified on 'trunk', the same code is synced on 'stable'. Anybody can download and build 'stable' during the life of v0.25 and get the latest makernote updates. The 'overflow' file isn't needed. And ... (drum roll) we can bring this to life as soon as v0.25 is published (taa-raa). This doesn't conflict with adding the 'overflow' file feature in v0.26.

As we now have a Jenkins build server (http://exiv2.dyndns.org:8080) we can publish the latest binary build of 'stable' for download by anybody. For applications which use Exiv2 dynamic libraries, it's possible to 'patch' the application by downloading the built library.

This doesn't put a burden on Niels to keep trunk/makernotes -> stable/makernotes in perfect sync. There are only 11 makernote .cpp/.hpp file pairs and if we forget to sync a file occasionally, it will 'catch up' later. To relieve Niels of the sync burden, I can add a script to Jenkins to promote trunk/makernotes -> stable. To ensure quality, the build server builds and runs the test suite on all monitored branches.

638 rmills@rmillsmm:~/gnu/exiv2/foo/trunk $ finder "*mn*" | rev | sort | rev
./src/sigmamn.cpp
./src/minoltamn.cpp
./src/panasonicmn.cpp
./src/samsungmn.cpp
./src/fujimn.cpp
./src/nikonmn.cpp
./src/canonmn.cpp
./src/casiomn.cpp
./src/olympusmn.cpp
./src/pentaxmn.cpp
./src/sonymn.cpp

./src/tiffmn-test.cpp

./src/sigmamn_int.hpp
./src/minoltamn_int.hpp
./src/panasonicmn_int.hpp
./src/samsungmn_int.hpp
./src/fujimn_int.hpp
./src/nikonmn_int.hpp
./src/canonmn_int.hpp
./src/casiomn_int.hpp
./src/olympusmn_int.hpp
./src/pentaxmn_int.hpp
./src/sonymn_int.hpp
639 rmills@rmillsmm:~/gnu/exiv2/foo/trunk $ 

Robin, does this really mean that stable binaries of library and command line tool are released (and provided by official distro repos like OpenSuse etc.) every time a maker note changes? Or does it mean that the source is provided and users can use and build them on their own?

Well, the long term plan is to add the 'overflow' file. However that's not going to be available until v0.26 which hopefully will ship in 2016. Niels has explained how the 'overflow' file will work after v0.26 ships.

v0.26 is quite a long way off. At the moment, I can't even say when v0.25 will ship. I don't have credentials for exiv2.org, so I have to wait for the Exiv2 Release Engineers (Andreas and Shawn) to make that release.

However, I'm suggesting that v0.25 can have simple "dot releases" only involving makernote changes. We can have a very high confidence that the quality of v0.25.n = v0.25.0 if we only automatically sync changes to the above files into the "dot" branch.

This is just a suggestion. Thanks for asking for clarification. As nobody else has commented, I don't know if my suggestion is:

• Not understoood
• Everybody's too busy to think about this
• Stupid and everybody's too polite to say so
• Something else!

However the intent would be that we would have lots of v0.25 "dot" releases. Any "dot" could be picked up by Suse etc. Every distro would have the latest makernote changes on the day they collect our code.

I've had a new idea about this while working on the camera lens issues such as #1143 and #1145.

We don't need an overflow file! We can have an SQL server on the internet and when an unidentified lens appears, libexiv2 can send a request using HTTP (which is part of all versions of libexiv2) along the lines of:

SELECT LensID where Exif.Image.Model = 'NIKON D5300' [AND key relop value]+"

1. Don't need an overflow file on the user's computer
2. We don't need dot releases with new lenses/accessories/cameras
3. Failing to identify a lens (or returning multiple lenses) can be logged so that the database notifies us of new devices

This could be an interesting project. Perhaps we can get a Student from GSoC 2016 to implement this.

And here's another even simpler solution. exiftool understands .exv files (exiv2 extracted metadata files). If we don't know the lens, we can send the .exv file to an http server (such as exiv2.org) that has a CGI script to run exiftool on the .exv file. So simple. It's 100% cross platform because our http code is already cross platform. So we set this up and live happily ever after. We inherit all the wonderful work done by Phil and have no maintenance effort. No files on the client computer. No horrible dos box on Windows. No need for the client to install exiftool. And, we register failures and send the metadata to Phil - so he learns about new devices.

I'm going to add the following to v0.26. The user can put the file .exiv2 (in .ini format) in their home directory to define a new lens:

[canon]
169=Sigma 35mm F1.4 DG HSM ART

Exiv2 will consult that file and use definitions in preference to what's compiled into the library. Let's not discuss the performance implication. I'll find a way to make it very fast (don't read if not necessary, then read/cache).

In a conversation with Martin, he raised the subject of line-endings in ~/.exiv2 http://dev.exiv2.org/issues/1212#note-13

The code uses library function fgets(...) to read lines from the .ini file. On the Mac, that handles UNIX (\n) and Windows (\r\n) and doesn't handle Mac (\r) lines. MSVC, Cygwin and Linux behave the same.

The code in src/ini.cpp code can use a local function instead of fgets(...). I'm not going to do anything about this for v0.26 and probably never will. The "Mac Legacy \r" line-ending is a long buried relic. In the process of testing line endings on Linux, I consulted the man page for unix2dos and unix2mac. Gosh, there's a mine-field of ways to complicate this file including character sets, BOM markers and all manner of stuff. This feature works well enough on "simple/plain" Unix and DOS files.