Project

General

Profile

Problem with Universal libs ]

Added by Dominik Pietrzak about 9 years ago

Dear All,

I am working with QtCreator and qmake project under Mac OSX and have built and installed Exiv2 with basic steps and it works fine.
The build runs perfectly ok using linking steps from (http://clanmills.com/exiv2/qt.shtml).
My qmake project contains:
macx {
INCLUDEPATH += /usr/local/include
LIBS += -L/usr/local/lib -lexiv2
}

Then, I have build the "Universal" libraries of Exiv2-0.23 on Mac OSX 10.6 (Snow Leopard) with steps explain here:
http://clanmills.com/exiv2/macosx.shtml
However what I am trying to do is to link statically (or/and dynamically) with the universal libraries located in project directory.
For this I deleted the libs (.a and .dylib) from the deafault locations and copied the Universal build (.a and .dylib) to project folder so my new qmake looks like :
macx {
INCLUDEPATH += /project/include
LIBS += -L/project/lib -lexiv2
}

The project process make, build and compilation without errors and warnings.
Although, at the beginning of run the application crashes with crash log:

  • /usr/local/bin/exiv2 ; exit;
    dyld: Library not loaded: /usr/local/lib/libexiv2.12.dylib
    Referenced from: /usr/local/bin/exiv2
    Reason: image not found
    Trace/BPT trap*

That means that exiv2 library is still searching in the local deafault location of libraries (.a and .dylib).
The app crashes either while dynamic or static link.
Static link I obtain by adding CONFIG += staticlib to the qmake.

Did I missed something with configuration process? Honestly, I am not very familiar with this kind of operations and I just followed the steps from mentioned webpage.
Any help, tip would be appreciated.


Replies (12)

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

You'll have to use the tool install_name_tool to modify the dylib when you copy it.

Dynamic libraries in MacOSX have records which tell their customers where they live. The linker copies the record into the executable. At run time, your executable will use this information to find the library. You can inspect the information inside an executable with otool -L foo.dylib

Sounds complicated? That's the way it is!

Several things you have to know how to make this work:
1) Inspect the records with otool -L foo.dylib
2) Modify the records with install_name_tool
3) The copy from the dylib to the executable occurs during linking

If you're no using the "standard" locations for the library (such as /usr/local/lib), you can specify a relative path with the record @executable_path/.../libexiv2.12.dylib. An application on Mac-OSX is a directory with the extension .app (usually called a bundle). It's normal for the bundle executable to be the file foo.app/Contents/MacOS/foo. If you add a directory foo.app/Libraries, then you can put your dylib into that location and use @executable_path/../../Libraries/libexiv2.12.dylib

If you'd like to a model of this being used, I recommend you look in any of the Adobe applications (such as Reader.app).

If you adopt this approach, then the user can put your bundle (foo.app) anywhere on the file system and it will work. As a bonus, the library that your application loads is guaranteed to the one you shipped and not an arbitrary library in /usr/local/lib. When you're debugging this, you'll find the various environment strings documented in 'man dyld' very useful. export DYLD_PRINT_LIBRARIES=1 is your friend. You should start your application from the Terminal as $ .... foo.app/Contents/MacOS/foo to see the messages. When you use 'open foo.app' (or double click the icon), the UI hides those messages. I think they are written to log and you can use console.app to see/inspect them. However I prefer to use the Terminal for this purpose.

Lastly, I recommend that you do this by hand to learn/discover how this works. Once you are comfortable with this, you'll want to write a script and add it to your build machinery. The script should copy libexiv2.12.dylib to somewhere and use install_name_tool to modify the copy. Then you link (and your executable will have your install_name), and finally you'll execute your app. It's fun when you get this to work.

Good Luck. Let me know how it goes.

Robin

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

On my way to work, I thought of something very obvious that I should said. The "quick fix" to get your application to run is to use:

$ otool -L foo.app/Contents/MacOS/foo
... inspect the link records in your app ...

$ install_name_tool -change /usr/local/lib/libexiv2.12.dylib .../your-project/.../libexiv2.12.dylib
$ otool -L foo.app/Contents/MacOS/foo
... inspect again ...
$ open foo.app (or better: $ env DYLD_PRINT_LIBRARIES=1 foo.app/Contents/MacOS/foo)

.. and you should be happy. However your happiness won't last when you have to do this every time you link your application.

By the way, when you run in the terminal $ foo.app/Contents/MacOS/foo

stdout is in your terminal. You can then use the much maligned, but very useful printf to help you debug your code.

RE: Problem with Universal libs ] - Added by Dominik Pietrzak about 9 years ago

Thank you Robin for the extensive answer. This is exactly what I was asking for!
This is the deployment topic and I am recently becoming familiar with it. I will play around with tips you gave me.
I have read these tips briefly and already have one additional question.
You focused here on dynamic linking for libexiv2.12.dylib only.
What about the static linking ?
After I run the script exiv2-0.23/contrib/buildForMac I got 3 directories with "Universal" libs :
exiv2-0.23/src/i.386 (for Intel 32-bits Mac OSX)
exiv2-0.23/src/.libs (for 32-bits Mac OSX)
exiv2-0.23/src/x86_64 (for 64-bits Mac OSX)

Each location contains binary file exiv2, dynamic libs with .dylib extension and static libraries with .a extension.
I can copy static libraries ( libexiv2.a file) to the desired folder where I set my application.
Thus my question is if there is any way to link the project with static linking libexiv2.a ?
I work with QtCreator and would build and link with following qmake script:

CONFIG += staticlib
macx {
INCLUDEPATH += /project/include
LIBS += -L/project/lib -lexiv2
}

where necessary include headers are in include directory and libexiv2.a in project/lib directory.
Previously, I work with different libraries and this was the way I link them in my project and everything was ok.

RE: Problem with Universal libs ] - Added by Dominik Pietrzak about 9 years ago

I forgot to mention to you that I am using DESTDIR in my qmake, so the final script looks as follows:

DESTDIR = /project/executables
CONFIG += staticlib
macx {
INCLUDEPATH += /project/include
LIBS += -L/project/lib -lexiv2
}

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

When you link <exiv2-dir>/src/.libs/..libexiv2.a and <exiv2-dir>/xmpsdk/src/.libs/libxmpsdk.a you should be done. The linker switch -lexiv2 will link libexiv2.a. The linker switch -L src/.libs add to the library search path.

$ make distclean ; make config --disable-shared --enable-static ; make 
$ ls -alt xmpsdk/src/.libs/libxmpsdk.a
-rw-r--r--  1 rmills  staff  509440 Oct 22 07:49 xmpsdk/src/.libs/libxmpsdk.a
$ ls -alt src/.libs/libexiv2.a 
-rw-r--r--  1 rmills  staff  4615312 Oct 22 07:50 src/.libs/libexiv2.a
$ cd samples
$ g++ exifprint.cpp -o exifprint -lexiv2 -L../src/.libs -lxmpsdk -L../xmpsdk/src/.libs -lexpat -lz -liconv -lintl
$ ls -alt exifprint
-rwxr-xr-x  1 rmills  staff  2416932 Oct 22 07:55 exifprint
$ ./exifprint
Usage: ./exifprint file
$ otool -L exifprint
exifprint:
    /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
    /usr/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
$ 

You can see that I've statically linked exifprint with the static build of libexiv2.a and libxmpsdk.a. However we're dynamically linked to libexpat, libz, libiconv (in /usr/lib). I'm a little surprised to see I'm linked to /usr/local/lib/libintl.

Static libraries reduce build complexity. No need to bother with otool, install_name_tool or any other complication. I personally prefer to link with shared libraries, even with the install_name_tool complication. It is however your choice.

Please be aware that universal binaries are almost dead with Mountain Lion (which does not have 32 bit support). If you're using Mountain Lion, it's better to forget about Universal binaries. I discovered then when I tested 0.23 with the developer preview of Mountain Lion. I didn't publish this because I was under NDA. Just do a regular build on ML. The script contrib/buildForMac was tested on Lion with static and shared libraries and passed our test suite.

RE: Problem with Universal libs - Added by Robin Mills about 9 years ago

One more thing (before I head off for work). I don't have Qt on my Mac at the moment. If you're still stuck, let me know and I will download Qt this evening and create a sample project for you.

RE: Problem with Universal libs ] - Added by Dominik Pietrzak about 9 years ago

Thank you very much for extensive support, Robin!

Although, still have some troubles with linking.
I have tried to build statically via QtCreator with settings you posted. Thus my qmake .pro file looks as follows:

CONFIG += static
macx {
INCLUDEPATH += exiv2-0.23/include/exiv2
LIBS += exiv2-0.23/xmpsdk/src/.libs/libxmpsdk.a \
exiv2-0.23/src/.libs/libexiv2.a
}
what surely should do the job you mentioned.
However it turns the linking error again :
_error: duplicate symbol XML_Node::IsWhitespaceNode() const
in exiv2-0.23/xmpsdk/src/.libs/libxmpsdk.a(XML_Node.o) and exiv2-0.23/lib/libexiv2.a(XML_Node.o)

It seems that there is some kind of symbols confict in these two static libs...
The linking errors are pretty hard to track and I am not sure what am I doing wrong.

By the way, I am working with Snow Leopard OSX 10.6, so it handles either 32 or 64 bit modes.

RE: Problem with Universal libs ] - Added by Dominik Pietrzak about 9 years ago

If you could create sample Qt projects for OSX with:
1) static linking
2) dynamic linking

I would be very happy and I think it would be useful for the others.
Possibly you may add it to wiki so anyone could take advantage of that.

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

Dominic

Are you sure that QtCreator is building "Universal" binaries. Create a "hello-world" command line program and run $ lipo -info hello to see if he's building universal.

It's going to take me most of this evening for me to load Snow Leopard, QtCreator and so on. Can I ask you to setup a command-line program in QtCreator and use the source code samples/exifprint.cpp and see if you can get that to link and run?

It looks as though both libxmpsdk.a and libexiv2.a have the same entry points. However I know that all our samples programs are working fine and don't give this linking error.

$ nm xmpsdk/src/.libs/libxmpsdk.a | grep IsWhitespaceNode
0000000000000000 T __ZNK8XML_Node16IsWhitespaceNodeEv
00000000000020a8 S __ZNK8XML_Node16IsWhitespaceNodeEv.eh
                 U __ZNK8XML_Node16IsWhitespaceNodeEv
$ nm src/.libs/libexiv2.a | grep IsWhitespaceNode
                 U __ZNK8XML_Node16IsWhitespaceNodeEv
0000000000000000 T __ZNK8XML_Node16IsWhitespaceNodeEv
00000000000020a8 S __ZNK8XML_Node16IsWhitespaceNodeEv.eh
$ c++filt __ZNK8XML_Node16IsWhitespaceNodeEv
XML_Node::IsWhitespaceNode() const
$ find . -name "*.hpp" -exec grep -H IsWhitespaceNode {} ";" 
./xmpsdk/src/XMLParserAdapter.hpp:    bool IsWhitespaceNode() const;
$ 

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

Dominic

The good news is that this is working just fine for me. I created a new partition on my hard drive and installed Snow Leopard from CD + the developer tools. Here's all the stuff I downloaded (I probably don't need cmake). I've downloaded/installed QtCreator 2.5.2 and Qt SDK 4.8.3.

48 rmills@Robin-Millss-iMac:~/Downloads $ dir
-rw-------  1 rmills  staff   545K Oct 24 19:51 About Downloads.pdf
-rw-r--r--@ 1 rmills  staff   182M Sep 13 01:59 qt-mac-opensource-4.8.3.dmg
-rw-r--r--@ 1 rmills  staff    93M Sep 11 00:40 qt-creator-mac-opensource-2.5.2.dmg
-rw-r--r--@ 1 rmills  staff    33M Aug  9 12:36 cmake-2.8.9-Darwin64-universal.dmg
-rw-r--r--@ 1 rmills  staff    68K Nov  2  2011 libintl-0.18.1.1-4.pkg
-rw-r--r--@ 1 rmills  staff   2.3M Oct 23  2011 PkgConfig.pkg
49 rmills@Robin-Millss-iMac:~/Downloads $ 

I built the shared libraries and installed them with:

$ contrib/buildForMac
$ sudo make install
$ make samples
$ make tests

Everything built and the test suite passed with no errors.

I configured QtCreator to use the qmake in /usr/bin/qmake and he figured out the 4.8.3 for himself. I created a console application 'exiv-console', removed the wizard's C++ and copied in exifprint.cpp from <exiv2dir>/samples/. I cut'n'pasted your code above into exiv-console.pro. It compiled, linked and ran first time!

macx {
    INCLUDEPATH += /usr/local/include
    LIBS += -L/usr/local/lib -lexiv2
}

Here's what was built:

76 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug/ $ ls -alt exiv-console 
-rwxr-xr-x  1 rmills  staff  43736 Oct 24 21:24 exiv-console
76 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug/ $ otool -L exiv-console
exiv-console:
    /usr/local/lib/libexiv2.12.dylib (compatibility version 13.0.0, current version 13.0.0)
    QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.3)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 103.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 123.0.0)
78 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug/ $ 

You can see that exiv-console was 43736 bytes and links /usr/local/lib/libexiv2.12.dylib.

So then I build the static library, removed the shared libraries and installed the static libraries:

$ contrib/buildForMac --disable-shared --enable-static
$ sudo rm -rf /usr/local/lib/*exiv2*
$ sudo make install

Back to QtCreator. Rebuild/relink. Linker errors. No problem, tell him to link zlib and friends. Modified exiv-console.pro:
macx {
    INCLUDEPATH += /usr/local/include
    LIBS += -L/usr/local/lib -lexiv2 -lexpat -lz -liconv -lintl
}

And things were just as happy.
32 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug $ ls -alt exiv-console 
-rwxr-xr-x  1 rmills  staff  4002344 Oct 24 21:54 exiv-console
33 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug $ lipo -info exiv-console 
Non-fat file: exiv-console is architecture: x86_64
34 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug $ otool -L exiv-console 
exiv-console:
    /usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 7.2.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.3)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
    /usr/local/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
    QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.3)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
    /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 103.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 123.0.0)
35 rmills@Robin-Millss-iMac:~/Projects/Qt/exiv-console-build-desktop-Qt_4_8_3__System__Debug $ 

This time, exiv-console was 4002344 bytes (4mb), however he doesn't need libexiv2.12.dylib.

Things learned

You don't need universal binaries to make this work. QtCreator doesn't build universal binaries by default. I think you'd have to add some -arch i386 -arch x86_64 arguments in various places to the .pro file.

So, forget contrib/buildForMac and simply use $ make config ; ./configure ; make ; sudo make install ; make samples ; make tests and you'll be fine.

What didn't go too well

1. /usr/local/lib/libexiv2.la was corrupted when installed by the shared library build

libexiv2.la is some kind of magic file used by libtool/pkgtool to determine how to link the code. I don't why he didn't copy correctly from the build tree to /usr/local/lib/libexiv2.la. I got messages about "couldn't find file ???" from libtool. So I just copied libexiv2.la from the build tree to /usr/local/lib/libexiv2.la and everything was fine.

2. libexiv2.a wasn't build correctly as a universal binary

I had to look in contrib/makeUniversal and figured out to execute the following commands:

$ lipo -arch i386 src/.i386/libexiv2.a -arch x86_64 src/.x86_64/libexiv2.a -create -output src/.libs/libexiv2.a
$ sudo cp src/.libs/libexiv2.a /usr/local/lib

I don't know why contrib/buildForMac didn't do this correctly. I know that script was good on Lion. Perhaps it needs a little fix for SL.

What I'm going to do next

With both my confessions (about the copy of libexiv2.la and not executing lipo on the universal static build), I will investigate and submit a fix if necessary. However you don't need it immediately, because I'm encouraging you to simply build for x86_64.

How do move you forward?

I've given you a considerable amount of time here. I don't believe your troubles are related to the exiv2 software. You have Mac/Qt/Build Engineering issues. If you want me to do more work with you on this, you will have to share your code with me. I assure you that I will treat it confidentially and with respect.

Finally, may I ask why you are doing this? If this is for another open-source project, I'm happy to help without charge. However I'm reluctant to give you very much more free time if this is a commercial project.

RE: Problem with Universal libs ] - Added by Dominik Pietrzak about 9 years ago

Thank you very much for your time, Robin !
I have learned a lot from your posts.
I am surprised you put so much time into this and provide such extensive explanation. During my work here I also invested quite much time to figure out how the things works (lack of experience in general). Therefore I am really thankful for your involvement.

Actually I am doing some research for a company which considers commercial license of Exiv2. Therefore I tried different ways of linking since the possible usage would be a part of large, cross-platform project (OSX, Windows, 32 and 64 mode for both platforms). Thus the Universal library was a choice.

Your posts were extremely helpful to me and I believe some other developers will also appreciate them.

RE: Problem with Universal libs ] - Added by Robin Mills about 9 years ago

Dominic

I am quite certain that you can create a successful cross-platform OSX/Win32/64 and even Linux/32 Linux/64 software component using Qt. And exiv2 will work perfectly in the mix. The reason I am so certain is because I have done this successfully.

I'm a freelance software guy and do C++, Build Engineering, Installers and Web. http://clanmills.com/files/CV.pdf

I had a couple of weeks at home a couple of years ago and spent them on a Qt project involving exiv2. This was when Snow Leopard was the current Mac-OSX (before Lion and Moutain Lion). The project wasn't finished when I got a call to do some work (for money) and the project has been sitting unfinished on the SVN server ever since. Linking on Windows was much harder than Mac or Unix. At that is when I wrote the articles: http://clanmills.com/exiv2/mingw.shtml and http://clanmills.com/exiv2/qt.shtml

So I've documented this already, however I might add a link from those articles to this thread as we've discussed many additional details here.

Please assure your customer that this project is feasible. A commercial license for exiv2 is very inexpensive. Email Andreas for details. I'm not looking for work, however I'm always willing to discuss opportunities for the future. If you wish to discuss how to involve me in the project then email me privately at

Best Wishes

    (1-12/12)