Exiv2

More info from my side :

The digiKam maintenance tool has an option to use all CPU core at the same time, to process file metadata synchronization. Typically, each thread run same code to operate with Exiv2. Each code is re-entrant of course.
Each thread operate one file, as to read image metadata and update the database (and vis versa).
We have lock in Database interface to prevent no re-entrancy cases, depending of database backend used.
Other code not using Exiv2 already operate with database in separated threads without problem.

We use Qt to manage a queue of threads to fork on separate CPU core. One thread is taken from the queue and executed on one available core. Onether one is taken if more than one core is available, etc. When a thread is completed, another one is taken from the queue, until the queue is empty. The goal of course is to parallelize operation to reduce execution time, especially with large collection of images.

We have small classes to do it. A test code based on these classes can be written quickly, if you is interested.

I hope to be enough clear with the internal digiKam code.

So the questions are :

Exiv2 code is really fully re-entrant everywhere ?
Did you have already tried this kind of use-case into test code ?

PS : The report from digiKam bugzilla listed in this entry is not alone. A lots of reports have been already send, with similar traces. Remember that whole digiKam use multicore/multithreading everywhere when it's possible.

Gilles Caulier

Robin Mills wrote:
[...]

Is it possible for you:
1) To confirm that your test file is good and works OK from the terminal/command-line

It is possible, but I need some time....

2) To update to at least Exiv2 v0.25

Installierte Versionen: 0.25-r2(0/14)(16:57:49 14.02.2016) -> this is ther running exiv2 here - sorry my posting isn't well easy to read. I will try to make it better.

uhai

Uwe

Thanks for getting back to me. If you can attach the test files to this issue, I'll test them on my machines. If you want to keep the test files confidential please share them on Dropbox so I can collect them. I'll let you know when I get them from Dropbox and you are remove them.

Robin

If this PNG image crash Exiv2, why the library do not generate a C++ exception.

In digiKam, Exiv2 API are all wrapped in C++ exception catcher. All Exiv2 API are only wrapped in dedicated classes to complete exception handling. Exiv2 API are not used in whole digiKam but only located in low level classes metaengine_* :

https://quickgit.kde.org/?p=digikam.git&a=tree&f=libs%2Fdmetadata

Gilles Caulier

Good news:

1) Your file is good.
2) I've written a very simple mt-test.cpp and it's working (code below)

1 Your file

1065 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ exiv2 -pa --grep Exif http://dev.exiv2.org/attachments/download/1034/2011-07-28T12-14-33SW.png 
Exif.Image.ProcessingSoftware                Ascii      14  digiKam-5.0.0
Exif.Image.Software                          Ascii      14  digiKam-2.9.0
Exif.Image.DateTime                          Ascii      20  2011:08:01 07:11:14
Exif.Image.ExifTag                           Long        1  110
Exif.Photo.DateTimeOriginal                  Ascii      20  2011:08:01 07:11:14
Exif.Photo.DateTimeDigitized                 Ascii      20  2011:08:01 07:11:14
1066 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ exiv2 -pa --grep Exif http://dev.exiv2.org/attachments/download/1034/2011-07-28T12-14-33SW.png | wc
       6      27     452
1067 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ 

1067 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ make -B mt-test "CXXFLAGS=-std=c++11 -stdlib=libc++" 
g++ -std=c++11 -stdlib=libc++ `pkg-config exiv2 --cflags` -c -I../include -I../include/exiv2 -o mt-test.o mt-test.cpp
mkdir -pv ../bin 2>&1 > /dev/null
../libtool --mode=link g++ `pkg-config exiv2 --libs` -rpath /usr/local/lib -L/usr/local/lib -o ../bin/mt-test mt-test.o
libtool: link: g++ -o ../bin/mt-test mt-test.o -Wl,-bind_at_load  -L/usr/local/lib /usr/local/lib/libexiv2.dylib -L/Users/rmills/gnu/exiv2/trunk/xmpsdk/src /usr/local/lib/libintl.dylib -lc -liconv -lz -lpthread
1068 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ ../bin/mt-test http://dev.exiv2.org/attachments/download/1034/2011-07-28T12-14-33SW.png ~/Stonehenge.jpg ~/Stonehenge2.webp
file: /Users/rmills/Stonehenge2.webp n: 3 count: 209 
file: /Users/rmills/Stonehenge.jpg n: 2 count: 209 
file: http://dev.exiv2.org/attachments/download/1034/2011-07-28T12-14-33SW.png n: 1 count: 6 
1069 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ 

// ***************************************************************** -*- C++ -*-
// mt-test.cpp, $Rev: 3090 $
// Sample program to test multi-threading

#include <exiv2/exiv2.hpp>
#include <iostream>
#include <iomanip>
#include <cassert>
#include <string>
#include <thread>
#include <mutex>

std::mutex m;

void reportExifMetadataCount(int n,const char* argv[])
{
    int count = 0 ;
    std::string what;
    try {
        Exiv2::Image::AutoPtr image = Exiv2::ImageFactory::open(argv[n]);
        assert(image.get() != 0);
        image->readMetadata();

        Exiv2::ExifData &exifData = image->exifData();
        if (!exifData.empty()) {
            Exiv2::ExifData::const_iterator end = exifData.end();
            for (Exiv2::ExifData::const_iterator i = exifData.begin(); i != end; ++i)
                count++;
        }
    } catch (Exiv2::Error& e) {
        what = e.what() ;
        count = -1;
    }

    m.lock();
    std::cout << "file: "  << argv[n] << " " 
              << "n: "     << n       << " " 
              << "count: " << count   << " " 
              << (count < 0 ? "exception: " : what)
              << what                 << std::endl;
    m.unlock();
}

int main(int argc,const char* argv[])
{
    int result = 0;
    if ( argc != 4 ) {
        std::cerr << "syntax: " << argv[0] << " path1 path2 path3" << std::endl;
        result = 1 ;
    } else {
        std::thread t1(reportExifMetadataCount,1,argv);
        std::thread t2(reportExifMetadataCount,2,argv);
        std::thread t3(reportExifMetadataCount,3,argv);
        if (t1.joinable())    t1.join();
        if (t2.joinable())    t2.join();
        if (t3.joinable())    t3.join();
    }
    return result;
}

4 Next steps

I haven't submitted this code to the repository.

Gilles: you said you might have some simple code to simulate your use case. I'd prefer not to get sucked into dealing with Qt if it can be avoided. All help appreciated.

Even better news. I think I've found the fix:

1) Upgrade to the trunk version of exiv2 which includes the fix for #1187
2) Explicitly call Exiv2::XmpParser::initialize() before spinning up the threads.

Here's my analysis. It's crashing consistently with a 'vanilla' v0.25. Down in the dungeons of XMPsdk:

0   libsystem_kernel.dylib            0x00007fff9bf1dde6 __psynch_mutexwait + 10
1   libsystem_pthread.dylib           0x00007fff9d54ee4a _pthread_mutex_lock_wait + 89
2   libexiv2.14.dylib                 0x000000010bfaf859 XMP_EnterCriticalRegion(_opaque_pthread_mutex_t&) + 9
3   libexiv2.14.dylib                 0x000000010bfa64a2 WXMPIterator_Next_1 + 66
4   libexiv2.14.dylib                 0x000000010bf9b759 TXMPIterator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::Next(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, unsigned long*) + 137
5   libexiv2.14.dylib                 0x000000010bf96ec1 Exiv2::XmpParser::decode(Exiv2::XmpData&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 545
6   libexiv2.14.dylib                 0x000000010bf17906 Exiv2::JpegBase::readMetadata() + 2006
7   mt-test                           0x000000010beb6bd0 reportExifMetadataCount(int, char const**) + 1776

I'm a little blind here. It almost feels as though you are getting an exception. Then your code crashes when it reports the exception.

digikam.metaengine: Cannot load metadata from file (Error # 14 : Failed to read image data
digikam.dimg: "/home/uhai/Bilder_alt/2011/2011-07-28/2011-07-28T14-02-52.png" : PNG file identified
Catchpoint 1 (exception thrown), __cxxabiv1::__cxa_throw (obj=0x7fff2d356640, tinfo=0x7ffff71a8340 <typeinfo for Exiv2::BasicError<char>>, dest=0x7ffff6818ce0 <Exiv2::BasicError<char>::~BasicError()>)

I've rebuilt the trunk with C++11 and getting curious results:

1095 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ ../bin/mt-test ~/Stonehenge.jpg ~/Stonehenge.jpg ~/Stonehenge.jpg 
Error: XMP Toolkit error 102: Unknown namespace prefix for qualified name
Warning: Failed to decode XMP metadata.
file: /Users/rmills/Stonehenge.jpg n: 1 count: 209 
Error: XMP Toolkit error 102: Unknown nameEsrpraocre:  prXeMfPi xT ofoolrk iqtu aelrirfoire d1 0n2a:m eU
nknownW anranmiensgp:a ceF apirleefdi xt of odre cqoudael iXfMiPe dm entaamdea
ta.
Warning: Failed to decode XMP metadata.
file: /Users/rmills/Stonehenge.jpg n: 2 count: 209 
file: /Users/rmills/Stonehenge.jpg n: 3 count: 209 
1096 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ ../bin/mt-test ~/Stonehenge.jpg ~/Stonehenge.jpg ~/Stonehenge.jpg 
file: /Users/rmills/Stonehenge.jpg n: 3 count: 209 
file: /Users/rmills/Stonehenge.jpg n: 2 count: 209 
file: /Users/rmills/Stonehenge.jpg n: 1 count: 209 
1097 rmills@rmillsmbp:~/gnu/exiv2/trunk/samples $ 

Gentlemen: I've run out of time for today. Share your thoughts and I'll spend more time on this during the week.

Gilles Caulier wrote:

If this PNG image crash Exiv2, why the library do not generate a C++ exception.

In digiKam, Exiv2 API are all wrapped in C++ exception catcher. All Exiv2 API are only wrapped in dedicated classes to complete exception handling. Exiv2 API are not used in whole digiKam but only located in low level classes metaengine_* :

Is this a crash? or an exception? I think I can continue in gdb....

(gdb) c
Continuing.
digikam.metaengine: Cannot load metadata from file (Error # 14 : Failed to read image data
digikam.dimg: "/home/uhai/Bilder_alt/2011/2011-07-28/2011-07-28T14-02-52.png" : PNG file identified
Catchpoint 1 (exception thrown), __cxxabiv1::__cxa_throw (obj=0x7fff2d356640, tinfo=0x7ffff71a8340 <typeinfo for Exiv2::BasicError<char>>, dest=0x7ffff6818ce0 <Exiv2::BasicError<char>::~BasicError()>)
at /var/tmp/portage/sys-devel/gcc-4.9.3/work/gcc-4.9.3/libstdc++-v3/libsupc++/eh_throw.cc:63
63 in /var/tmp/portage/sys-devel/gcc-4.9.3/work/gcc-4.9.3/libstdc++-v3/libsupc++/eh_throw.cc
(gdb)

uhai

Exception handlers don't catch signals. This is discussed here: http://dev.exiv2.org/issues/1164 Signal handlers are a process responsibility and should be dealt with by the application and not a library. In this case digiKam.

There is an exception involved. That's how you have the information:

Cannot load metadata from file (Error # 14 : Failed to read image data
digikam.dimg: "/home/uhai/Bilder_alt/2011/2011-07-28/2011-07-28T14-02-52.png" :

1) Upgrade to the trunk version of exiv2 which includes the fix for #1187
2) Explicitly call Exiv2::XmpParser::initialize() before spinning up the threads.

Robin Mills wrote:

Exception handlers don't catch signals. This is discussed here: http://dev.exiv2.org/issues/1164 Signal handlers are a process responsibility and should be dealt with by the application and not a library. In this case digiKam.

There is an exception involved. That's how you have the information: [...]However there's more to this an uncaught exception. I'd really like you to try my 2 step recipe and let me know how that works for you:

1) Upgrade to the trunk version of exiv2 which includes the fix for #1187

Sorry, I'm not sure... I'm only a user, no dev. Is this the version I shall use:

Installierte Versionen: 0.25-r2(0/14)(16:57:49 14.02.2016)(nls png xmp -doc -examples -webready ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32" LINGUAS="de -bs -es -fi -fr -gl -ms -pl -pt -ru -sk -sv -ug -uk -vi")

2) Explicitly call Exiv2::XmpParser::initialize() before spinning up the threads.

This part I don't understand. Will you be so kind to give me an detailed howto? or an link to an howto for this?

Sorry, no programming skills, bad english - your job is hard :-)

uhai

Here's how I called that in the mt-test.cpp code listed above:

int main(int argc,const char* argv[])
{
    int result = 0;
    if ( argc != 4 ) {
        std::cerr << "syntax: " << argv[0] << " path1 path2 path3" << std::endl;
        result = 1 ;
    } else {
        Exiv2::XmpParser::initialize();
        std::thread t1(reportExifMetadataCount,1,argv);
        std::thread t2(reportExifMetadataCount,2,argv);
        std::thread t3(reportExifMetadataCount,3,argv);
        if (t1.joinable())    t1.join();
        if (t2.joinable())    t2.join();
        if (t3.joinable())    t3.join();
    }
    return result;
}

2) Explicitly call Exiv2::XmpParser::initialize() before spinning up the threads.
This part I don't understand. Will you be so kind to give me an detailed howto? or an link to an howto for this?
Sorry, no programming skills, bad english - your job is hard :-)

This is already done in digiKam since a while (more than few years)... into metadata engine wrapper line 75.

https://quickgit.kde.org/?p=digikam.git&a=blob&f=libs%2Fdmetadata%2Fmetaengine.cpp

So the problem is not here...

Gilles Caulier

Robin,

You MT code is too simple. It read by 3 thread the same file.

It's not enough. The read must be done on different files from different type. On i7 CPU, 8 core want mean 8 threads at the same time.

Also, i think reading is not the most critical stage. Writing is more and less tedious. Imagine the case where a thread update metadata and another one read metadata at the same time, on the same file of course...

But as we have seen in backtrace, the crash appear while read metadata. If you test file can support to read metadata from a huge collection, this will be a good step.

My tips : in you MT test code, pass a path as argument to a collection a file. parse the path recursively an make a list of file path to read with Exiv2. Loop over the list step by step depending of CPU cores available and read file into separated threads until the list is completed. This is how digiKam process in Maintenance tool.

Gilles Caulier

Uwe,

An important question : when the crash appear, did you synchronize database with file metadata (read files and update DB), or the inverse (read DB and write in files).

The behavior is completely different as reading metadata from files do not change files contents. The inverse of course, yes...

Gilles Caulier

Please correct me, if it is wrong:

exiv2-0.25-r2 is the version I shall try.

First step - save your code as mt-test
Second step - make mt-test -> make complains about "for this aim is nothing to do"

Is this the correct make command:
make -B mt-test "CXXFLAGS=-std=c++11 -stdlib=libc++"
g++ -std=c++11 -stdlib=libc++ `pkg-config exiv2 --cflags` -c -I../include -I../include/exiv2 -o mt-test.o mt-test.cpp
mkdir -pv ../bin 2>&1 > /dev/null
../libtool --mode=link g++ `pkg-config exiv2 --libs` -rpath /usr/local/lib -L/usr/local/lib -o ../bin/mt-test mt-test.o
libtool: link: g++ -o ../bin/mt-test mt-test.o -Wl,-bind_at_load -L/usr/local/lib /usr/local/lib/libexiv2.dylib -L/Users/rmills/gnu/exiv2/trunk/xmpsdk/src /usr/local/lib/libintl.dylib -lc -liconv -lz -lpthread

So I had to correct the paths - ok so far? What's next?

uhai

Gilles Caulier wrote:

Uwe,

An important question : when the crash appear, did you synchronize database with file metadata (read files and update DB), or the inverse (read DB and write in files).

No, I have only marked maintenance - quality. Nothing else. I've tested both directions of metadata-synchronizing a couple of days ago.

I can start an new test, please tell me what maintenance-function I shall run. Multicore or no multicore?

Uwe

Uwe,

Maintenance Quality Sort tool will only read metadata from file and Tag image in database accordingly with quality parser result.

So the current problem is only when read metadata.

For the Metadata Synchroniser, we will see later, step by step, but both directions and both threaded processing are important to test. If problem disappear in single thread and appear in multi threads, there is really a re-entrance problem somewhere.

Gilles Caulier

So I test now Maintenance quality sort without multicore-support? Perhaps only with the folder with the uploaded picture?

Uwe

Gilles

In the last week, I have spent about 20 hours working on stuff for you and DigiKam. Perhaps you'd like to spend 20 seconds to say "Thank You". My multi-threaded test may not deal with every nuance of your use case. However I have identified useful short-comings on Exiv2 multi-threading support. I've asked you to modify your code in a couple of ways and let me know the outcome. Please help me to help you. And please have the courtesy to occasionally say "Thank You".

In comment #12 you said :

Exception handlers don't catch signals. This is discussed here: http://dev.exiv2.org/issues/1164 Signal >handlers are a process responsibility and should be dealt with by the application and not a library. In this >case digiKam.

I don't understand well the incidence if a signal handler is not implemented in digiKam metadata engine. It's not the case currently.

Do i need to do it ? This can prevent a crash from Exiv2 ? If yes, why exactly ? Why Exiv2 send a signal to client application ?

This signal are Posix signals. Right ? If yes, how do you deal under Windows which is not a Posix compliant system ?

Gilles... (who thank you sir for your support (:=)))

Gilles. This conversation is not going well. I don't know what's wrong. It's possible for the signal to come from your exception handler. Here's where we are:

1) I've asked you to try two changes and report back.
2) You've asked me to do additional work on my mt-test.cpp code. I'll look at that on Monday.
3) I can't build and debug the whole assembly of DigiKam, Qt, KDE and Exiv2 working together. I can't for a couple of reasons:
a) I don't know how to.
b) I don't have the time.
c) I don't believe it is necessary for me to take on ownership of the whole problem.

There is another approach. Please reduce the problem to a modest little test program based on mt-test.cpp, or any code you have available. Please provide a simple definition of the issue that does not involve Qt, boost, DigiKam or any other huge frameworks and then we can fix this.

I assure you that I am trying to help you.

yes, tomorrow i will look to code test implementation about multi-core and Exiv2.

But my question about the signal still valid. You said :

I don't know what's wrong. It's possible for the signal to come from your exception handler.

No signal are emitted from digiKam metadata engine wrapper. We don't use Posix signals in digiKam.

In http://dev.exiv2.org/issues/1164, it said to implement a signal handler in digiKam. The question is why ?

Gilles Caulier

uwe,

in digiKam source code i written a simple CLI test tool named "metareaderthread" :

https://quickgit.kde.org/?p=digikam.git&a=blob&f=tests%2Fdmetadata%2Fmetareaderthread.cpp

Binary is located in build/core/tests/dmetadata/. It's compiled if you enable tests at configuration time through Cmake, with option "-DBUILD_TESTING=ON". If you use bootstrap.linux script to configure whole digiKam source code, this option is enabled by default.

In command line, just call this tool with a path where is located your collection of images files. The tool will parse the collection recursively and will load metadata from all jpg, png, and tif files. This will use all CPU cores available to run threads where Exiv2 will load metadata and print a resume of main photo info.

Here, on 4 cores CPU running on a VM, 2000 images from my test collection is parsed in 5 secondes :

[gilles@localhost dmetadata]$ pwd
/home/gilles/Devel/5.x/build/core/tests/dmetadata
[gilles@localhost dmetadata]$ ls al
total 6524
drwxr-xr-x 3 gilles gilles 4096 Aug 14 19:08 ./
drwxr-xr-x 20 gilles gilles 4096 Aug 14 17:21 ../
drwxr-xr-x 24 gilles gilles 4096 Aug 14 17:21 CMakeFiles/
-rw-r--r- 1 gilles gilles 997 Aug 10 06:49 cmake_install.cmake
rwxr-xr-x 1 gilles gilles 1027736 Aug 14 17:43 commentReadWriteTest*
-rw-r--r- 1 gilles gilles 85 Aug 14 13:41 commentReadWriteTest_automoc.cpp
rw-r--r- 1 gilles gilles 305 Aug 10 06:49 CTestTestfile.cmake
rwxr-xr-x 1 gilles gilles 393936 Aug 14 17:46 erasetag*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 erasetag_automoc.cpp
rwxr-xr-x 1 gilles gilles 355104 Aug 14 17:45 loadfromba*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 loadfromba_automoc.cpp
rw-r--r- 1 gilles gilles 48923 Aug 14 17:21 Makefile
rwxr-xr-x 1 gilles gilles 806040 Aug 14 19:08 metareaderthread*
-rw-r--r- 1 gilles gilles 81 Aug 14 19:00 metareaderthread_automoc.cpp
rw-r--r- 1 gilles gilles 4231 Aug 14 13:41 moc_commentreadwritetest.cpp
rw-r--r- 1 gilles gilles 4234 Aug 14 19:00 moc_metareaderthread.cpp
rw-r--r- 1 gilles gilles 4210 Aug 14 13:41 moc_ratingreadwritetest.cpp
rw-r--r- 1 gilles gilles 4806 Aug 14 13:41 moc_tagsreadwritetest.cpp
rwxr-xr-x 1 gilles gilles 425432 Aug 14 17:45 printtagslist*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 printtagslist_automoc.cpp
rwxr-xr-x 1 gilles gilles 849120 Aug 14 17:43 ratingReadWriteTest*
-rw-r--r- 1 gilles gilles 84 Aug 14 13:41 ratingReadWriteTest_automoc.cpp
rwxr-xr-x 1 gilles gilles 392904 Aug 14 17:49 readimagewritexmpsidecar*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 readimagewritexmpsidecar_automoc.cpp
rwxr-xr-x 1 gilles gilles 552896 Aug 14 17:45 setiptcpreview*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 setiptcpreview_automoc.cpp
rwxr-xr-x 1 gilles gilles 462872 Aug 14 17:43 setxmpface*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 setxmpface_automoc.cpp
rwxr-xr-x 1 gilles gilles 857880 Aug 14 17:46 tagsReadWriteTest*
-rw-r--r- 1 gilles gilles 82 Aug 14 13:41 tagsReadWriteTest_automoc.cpp
rwxr-xr-x 1 gilles gilles 393120 Aug 14 17:46 usexmpsidecar*
-rw-r--r- 1 gilles gilles 93 Aug 10 06:49 usexmpsidecar_automoc.cpp
[gilles@localhost dmetadata]$ ./metareaderthread /mnt/data/
digikam.general: Using 4 CPU core to run threads
"/mnt/data/2014/2014-07-29/pano4/DSC04692.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04690.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04678.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04700.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04695.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04699.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04702.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04679.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04676.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04682.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04698.jpg"
"/mnt/data/2014/2014-07-29/pano4/DSC04675.jpg"
...
Photo info:
digikam.metaengine: DateTime => Exif.Photo.DateTimeOriginal => QDateTime(2014-04-07 16:08:01.000 EDT Qt::TimeSpec(LocalTime))
PhotoInfoContainer::make: "SONY", PhotoInfoContainer::model: "SLT-A77V", PhotoInfoContainer::lens: "70-200mm F2.8", PhotoInfoContainer::exposureTime: "1/2000 s", PhotoInfoContainer::exposureMode: "Auto", PhotoInfoContainer::exposureProgram: "Aperture priority", PhotoInfoContainer::aperture: "F2.8", PhotoInfoContainer::focalLength: "70.0 mm", PhotoInfoContainer::focalLength35mm: "105.0 mm", PhotoInfoContainer::sensitivity: "100"PhotoInfoContainer::flash: "No, compulsory"PhotoInfoContainer::whiteBalance: "Auto"PhotoInfoContainer::dateTime: QDateTime(2014-04-07 16:08:01.000 EDT Qt::TimeSpec(LocalTime))PhotoInfoContainer::hasCoordinates: true
digikam.general: One job is done
2000 files processed
digikam.general: Cancel Main Thread
[gilles@localhost dmetadata]$

This tools work fine for you with your collection, without to see a crash ?

Gilles Caulier

Sorry, got troubles with bootstrap.linux:

-- The following REQUIRED packages have been found:

• KF5Config (required version >= 5.1.0) * KF5XmlGui (required version >= 5.1.0) * KF5Service (required version >= 5.1.0) * KF5 (required version >= 5.1.0) * JPEG * PNG * TIFF * Qt5Gui (required version >= 5.6.1) * Qt5Widgets * Gettext * PythonInterp * KF5I18n (required version >= 5.1.0) * KF5Wallet (required version >= 5.1.0) * KF5WidgetsAddons (required version >= 5.1.0) * KF5TextWidgets (required version >= 5.1.0) * ECM (required version >= 1.5.0) * Qt5Network * KF5CoreAddons (required version >= 5.5.0) * Qt5Core * Qt5Test * Qt5 (required version >= 5.2.0)

CMake Error at extra/libkvkontakte/CMakeLists.txt:34 (find_package):
By not providing "FindKF5WebKit.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"KF5WebKit", but CMake did not find one.

Could not find a package configuration file provided by "KF5WebKit" 
  (requested version 5.3.0) with any of the following names:

KF5WebKitConfig.cmake
    kf5webkit-config.cmake

Add the installation prefix of "KF5WebKit" to CMAKE_PREFIX_PATH or set
  "KF5WebKit_DIR" to a directory containing one of the above files.  If
  "KF5WebKit" provides a separate development package or SDK, be sure it has
  been installed.

-- Configuring incomplete, errors occurred!
See also "/root/digikam-5.1.0/dk/build/CMakeFiles/CMakeOutput.log".

What's wrong? Do I need KF5WebKit?

Uwe

Uwe

You're welcome to use this issue to continue to discuss this matter. However this is mostly about building something. DigiKam maybe? Can you restrict discussion here to Exiv2/Threading.

Hi Robin,

I try to build digikam from git to test the metreaderthread-code from Gilles. Sorry I#am only an user . This is my first time cloning git repository and build without portage. Any help is welcome...

Uwe

Uwe,

edit bootstrap.linux and change-DDIGIKAMSC_COMPILE_LIBKVKONTAKTE=ON to OFF, and restart it.

Gilles

Robin,

This is to configure digiKam and compile my multi-cores/mult-threads test CLI tool to check if Exiv2 crash while scanning all images from Uwe in parallel. The goal is to test with a limited code from digiKam to reproduce the problem. Ideally, it's to identify if problem is in digiKam or Exiv2.

On my computer, it do not crash with current implementation from svn/trunk. I just tested with 2000 items (jpg, png, and tif). Perhaps Uwe will observe another behavior.

Gilles Caulier

I'm very pleased to hear you're making good progress with this. This is not easy for Uwe. Building a big application like digiKam is a challenge. Ben and I are making very good progress with WebP. I've been working on testing WebP all day. Hoping to declare WebP solid for reading and writing metadata in the next few days.

It's not easy - but I'm learning a lot :-)

Now bootstrap.linux runs fine. make too - be back soon.

I enjoy digikam since 0.7 and I like help to make it better.

Uwe

If you work in software development, you never stop learning. Every day brings surprises and new technology to be mastered.

I have a job as facility manager. Every day brings surprises and new problems to be mastered.

I try to learn programming for a while but after work I'm too tired for thinking hard. All I do is taking photos and managing the collection with gentoo (hard enough sometinmes). So bughunting is an new thing for me.

Now make and make install are all ok. metareadthread is running. First files are done well, now there are some *.png from 2011 giving error 6 invalid key Xmp.exif.Exposureindex & xmp.exif.ISOSpeedratings - but still running.

Should I start it in gdb?

Uwe

Oh, boy. gdb isn't a tool for users. The problem of the multi-threaded crash appears to have gone away. So I think we've resolved this issue.

Collect some of the files giving messages into a ZIP and attach them to this issue. I'll look at them. Errors/warnings are probably correct.

Reading metadata from 105516 files took 8793.28 seconds

Great. Exiv2 parallelization work fine. It's a good point.

The exception printed on 3 files are not errors in fact. The digiKam Exiv2 interface try to get a sets of tags in image to populate a photo info container grouping common photograph details of shot. Exiv2 do not find some tags checked by interface. That all..

Gilles Caulier

Right. Well done everybody. Can I confirm that the resolution for this is:

1 Exiv2 svn 4350+ to include fix for #1187
2 Call Exiv2::XmpParser::initialize(); before spinning up threads
(digiKam has done this for years in its exiv2 interface wrapper)

Gilles:
If you are satisfied that no code changes are required in Exiv2, please assign this back to me. I will add mt-test.cpp to our test suite AND I'll test with more than 3 threads AND many different files.

Robin,

Typically the result while reading on multicore with Exiv2 sound good now.

It rest a test to do which will be more problematic : writing operations in parallel.

Uwe,

I will modify the test CLI tool in digiKam to be able to write something in images. Of course this must be done on a copy of your collection. This will touch your file of course.
I think to write one Exif, one Iptc, and one XMP tags at the same time on all image.

Gilles Caulier

Uwe ,

In your collection, do you have video files. If yes, the curretn tool do not try to read video files.

It's also the case with RAW files. I will at least oatch the code to play with RAW files.

Gilles Caulier

Uwe,

Implemented with commit http://commits.kde.org/digikam/4e9884e4439ba73dd19a600693ac49488b33a4ef

new CLI tool syntax :

bash-3.2$ ./metareaderthread
metareaderthread - test to load metadata from images through multi-core threads
Usage : <direction: READ | WRITE> <images path> <image file filter> ... <image file filter>
Example: READ /mnt/photos *.jpg *.png *.tif *.nef *.dng
Warning: Write direction will touch file matadata contents!

Gilles Caulier

Robin,

The speed depend a lots of drive device type with Exiv2.. With my multicore CLI tool, here i use SSD, and 2000 items are parsed in few seconds. If i do the same in a real hard drive, it's note really the same.... Time is at least multiplied by 10.

Also, the cache memory from device and driver have a lots of influence. I recommend to check with more items. As in statistic in real world, use more than 1000 items to start to have a real sample measurement results..

Gilles Caulier

2000 in just over a second:

842 rmills@rmillsmbp:~/gnu/exiv2/trunk $ time find ~/Pictures/2016 -print0 -name "*.jpg" | xargs -0 bin/mt-test 2>/dev/null  | wc
    2236   25236  272275

real    0m1.293s
user    0m2.758s
sys    0m1.201s
843 rmills@rmillsmbp:~/gnu/exiv2/trunk $ 

Robin,

Did you use a SSD ?

Gilles

Yes. That was running on my MacBook Pro with an SSD. Not so quick on my elderly 2008 iMac with a 3TB HD where it dies a horrible death:

libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable

I don't want to spend any more time on this.