How to build exiv2 32-bit libraries using Cmake

Added by Arnold Wiegert 21 days ago

Running under Win 10 - 64-bit using MSVC 2015 Community edition.
Installed Cmake and have downloaded the latest trunk exiv2-0.26
I have been able to build the 64-bit code (which seems to be the default), both static & dynamic versions - release and debug, but have been unable to guess at what command line arguments or environment variable to set to make it compile 32-bit versions.
Even cmakeBuild --help gives no hints
Any help or pointer will be very much appreciated


Replies (43)

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 16 days ago

Arnold

Where are you located? Alison and I will be in the States 29 September until November 6. Perhaps we can meet.

You know more about the Windows/UNICODE puzzle than you think. With a little encouragement, you and the student will have fun and learn lots. I believe Unicode support in XML is provided by the processing instruction: https://en.wikipedia.org/wiki/Processing_Instruction So, Adobe are saying "internally XMP is UNICODE and when we serialise it (write it as XML) it is encoded according to UNICODE/xml standards.".

I joined the Exiv2 project in 2008 when I ported Exiv2 and pyexiv2 to Mac and Windows. http://clanmills.com/articles/gpsexiftags/ pyexiv2 died because it was obsoleted by gexiv2 which I have never investigated. Ask the Gramps people to contact me. I know something about pyexiv2.

Remarkably, I know very little about how people use Exiv2. I only hear from people who want something. I haven't thought about UNICODE_PATH until recently. In v0.27, all windows builds will support that. It will be unconditional. It adds UNICODE path APIs. Existing (ascii) users will see no change. Existing (unicode) users will not require special builds.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 14 days ago

I've had a busy and successful week working on CMake/Exiv2 contrib/cmake/msvc/cmakeBuild.cmd. The script, all CMakeLists.txt files, and documentation have been overhauled.

The code is available from https://github.com/clanmills/exiv2. On Friday night, I successfully built the following 56 "flavours".

{ 2005 | 2008 | 2010 | 2012 | 2013 | 2015 | 2017 } x { 32 | 64 } x { static | dll } x { debug | release }

This morning I have successfully run the test suite on:

cmd/c "vcvars 2005 64 && cmakeBuild --test --dll --release --unicode"
cmd/c "vcvars 2015 32 && cmakeBuild --test --static --debug --unicode"

I'll run run the test suite everywhere tonight.

TODO: I build exiv2 + zlib + expat. I haven't looked at --webready, --video and they will be tested/investigated next week.

I decided not to add a 'u' to the UNICODE_PATH libraries because this changes the Exiv2 ABI. In v0.27, all Windows builds will enable UNICODE_PATH.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 14 days ago

Sounds like you made a lot of progress.
I will try to test the new libraries, if possible, before the weekend is over, but if not, it most likely won't be til late next week.

As for the Summer of Code project etc. I'll have regretfully decline.
I had a brief look at the terms of reference & I see no way I can take on more commitments. I've already had to back off from helping with the Windows version of Gramps.
As, and if, time permits, I do want to keep an eye on both projects.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 14 days ago

That's fine, Arnold. You've made a very useful contribution to Exiv2 by getting me to revisit cmakeBuild.cmd (which I invented) and to think about UNICODE_PATH (which has nothing to do with me). And you've caused me to think about UNICODE in metadata and flagged metadata chunks in PNG files. Please know that you have helped. If you test/discover/report issues with Visual Studio, I assure you of attention. Good Job, buddy. Thanks very much.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 14 days ago

Downloaded the zip file from the link you provided.
Also looked at the 'contributors' - looks like things are pretty quiet other than commits by yourself ;-) :-(

Tried the static build: cmd/c "vcvars 2015 32 && cmakeBuild --test --static --debug --unicode"

One surprise it forced me to install cygwin - that was not necessary for my earlier trials.
That command ran to completion, as far as I can tell; there is a bit of a puzzle at the end of the run:
------------
Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:08.96
---------------------------------------------------------------------------------------------------------
----- /cygdrive/d/pkg/C++/MSVC2015/exiv2-master/contrib/cmake/msvc/dist/2015/Win32/static/Debug/bin -----
----- D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc\dist\2015\Win32\static\Debug\bin -----
---------------------------------------------------------------------------------------------------------
./testMSVC.sh: line 35: make: command not found
exiv2 0.26 001a00 (32 bit build)
Copyright (C) 2004-2017 Andreas Huggel.

.....

exiv2=0.26.0
platform=windows
compiler=MSVC
bits=32
dll=0
debug=1
cplusplus=199711
cplusplus11=0
version=13.00 (2015/x86)
date=Sep 9 2017
time=12:34:35
ssh=0
curl=0
id=$Id$
executable=D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc\dist\2015\Win32\static\Debug\bin\exiv2.exe
library=C:\WINDOWS\SYSTEM32\ntdll.dll
library=C:\WINDOWS\System32\KERNEL32.DLL
----8X-------------
xmlns=xmpGImg:http://ns.adobe.com/xap/1.0/g/img/
xmlns=xmpMM:http://ns.adobe.com/xap/1.0/mm/
xmlns=xmpNote:http://ns.adobe.com/xmp/note/
xmlns=xmpRights:http://ns.adobe.com/xap/1.0/rights/
xmlns=xmpT:http://ns.adobe.com/xap/1.0/t/
xmlns=xmpTPg:http://ns.adobe.com/xap/1.0/t/pg/
xmlns=xmpidq:http://ns.adobe.com/xmp/Identifier/qual/1.0/

D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>
----------------
Not sure what that all means.

Other than having run plain exiv2.exe on a test file without errors, I have not had a chance to use or test the newly built library. ================
Running: cmd/c "vcvars 2005 64 && cmakeBuild --test --dll --release --unicode"
gives me:

_D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>cmd/c "vcvars 2005 64 && cmakeBuild --test --dll --release --unicode"
The system cannot find the path specified.
'vcvarsamd64.bat' is not recognized as an internal or external command,
operable program or batch file.
calling cmakeDefaults.cmd
checking that ..\..\.. exists

------ cmakeBuild Settings ----------
bash = c:\cygwin64\bin\bash.exe
binpath =
build =
builddir = D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc
config = Release
curl = curl-7.45.0
dryrun =
exiv2 = D:\pkg\C++\MSVC2015\exiv2-master
expat = expat-2.1.0
generator =
install =
incpath =
libpath =
libssh = libssh-0.7.2
nls = 0
openssl = openssl-1.0.1p
platform =
samples =
shared = 1
test = 1
unicode = 1
vc =
video =
vs =
webready =
work = work
zlib = zlib-1.2.8

EXIV2 = D:\pkg\C++\MSVC2015\exiv2-master
testing VSINSTALLDIR ""
"VSINSTALLDIR not set. Run vcvars32.bat or vcvarsall.bat or vcvars.bat ***"

D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>


As for the 'path' issue, two comments:

one - to investigate and debug, one would need specific files which cause issues.
It is much easier to debug specific problems and then fixing the cause, usually cleans up much more than that issue.

two - aside from checking on the requirements for GoC (trying to figure out what would be involved), I also had a quick look into the code to see what UNICODE_PATH?? controls and my first impression is that there are two string conversion routines called c2ws & ws2c?? which are invoked.

While I did not chase that code, just the naming seems to imply that the conversion is to & from wide string, which is not the same as Unicode string.

_

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 13 days ago

Thanks for investigating this, Arnold. Very helpful.

The argument --test in cmakeBuild.cmd says "run the test suite" and that requires Cygwin (the test suite is written in bash). The message ./testMSVC.sh: line 35: make: command not found is make stumbling as it tries to get the test suite to run. This is odd because the output xmlns=... comes from the test suite. One day we'll find the culprit. For sure, I've never seen this message!

The command: cmd/c "vcvars 2005 64 && cmakeBuild --test --dll --release --unicode" fails because you do not have Visual Studio 2005 installed. It is a free download if you want to build with that. You can imagine that my focus is the build. I haven't given much thought to the error message if you do not have the build tools installed.

The UNICODE_PATH support uses wchar strings. The Exiv2 UNICODE API uses wchar to talk to the file system. I've forgotten the purpose of ws2s and s2ws. It's minor - for reporting error messages or something like that. The error system isn't in UNICODE. If a file cannot be opened, the error system uses ws2s to get an ascii string.

Take a look at Google Summer of Code: https://developers.google.com/open-source/gsoc/ It's a program to sponsor students to work on open-source. I mentored Tuan in 2013 and Alison and I went to visit him in Singapore last year. We've been invited to his wedding in Vietnam in November and we're looking forward to attending. Both Tuan and Alice are Vietnamese and live in Singapore. If you were to mentor a student, I would be involved with the project and help steer you to success. However, I'd prefer not to mentor as I am already very busy with Exiv2. I was intending to retire from the project at the end of 2017 to study for a PhD at the Open University. I've decided not to do a PhD and will continue to work on Exiv2 in 2018. I've put 10,000 hours into Exiv2 over 10 years and I will stay with the project until I am confident of the project's survival without me.

I kicked off a "full build" before going to bed on Saturday night. All 112 configurations including --test. If every build+test is 10 minutes, it'll run for 112/6 = 18 hours. It's still running after about 8 hours. Very pleased with our progress this week.

{ 2005 | 2008 | 2010 | 2012 | 2013 | 2015 | 2017 } x { 32 | 64 } x { --static | --dll } x { --debug | --release } x { --ascii | --unicode }

7 Versions of Visual Studio x 2 bits x 8 permutations of options. = 7 x 2 x 8 = 112.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 13 days ago

The problem with the 'test' not running, is something I should have expected since I doubt that I have the proper setup for that.
Same with the other 'complaint' - again, just trying to rush trough the command had me seeing 2015 where your text read 2005 - my apologies.
When I ran the 'proper' command, corrected to call for the proper tool chain 2015 and omitted --test, all worked as expected.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 10 days ago

Just downloaded the latest master - I'm still far to inexperienced with git to use it - and ran
cmd/c "vcvars 2015 32 && cmakeBuild --static --debug --unicode"

---------
LINK : warning LNK4044: unrecognized option '/DCMAKE_LINK=static'; ignored [D:\pkg\C++\MSVC2015\exiv2-master\contrib\cm
ake\msvc\work_Debug\zlib-1.2.8\zlib.vcxproj]
....

LINK : warning LNK4044: unrecognized option '/DCMAKE_LINK=static'; ignored [D:\pkg\C++\MSVC2015\exiv2-master\contrib\cm
ake\msvc\work_Debug\zlib-1.2.8\example.vcxproj]

...
also for minizip

Can these be ignored??

it finishes with:

-- ------------------------------------------------------------------
-- exiv2 0.26 configure results <http://www.exiv2.org>
-- Building shared library: NO
-- Building PNG support: YES
-- XMP metadata support: YES
-- Building static libxmp: YES
-- Native language support: NO
-- Conversion of Windows XP tags: YES
-- Nikon lens database: YES
-- Commercial build: NO
-- Building video support: NO
-- Building webready support: NO
-- Dynamic runtime override: NO
-- Unicode paths (wstring): YES
-- Building exiv2 command: YES
-- Building samples: NO
-- Building PO files: NO
-- ------------------------------------------------------------------
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:

CMAKE_LINK

-- Build files have been written to: D:/pkg/C++/MSVC2015/exiv2-master/contrib/cmake/msvc/work_Debug/exiv2
The system cannot find the file specified.
"*** build errors in EXIV2 ***"

D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 10 days ago

Arnold

The "latest" code is on https://github.com/clanmills/exiv2 and should work. I haven't integrated the "latest" to https://github.com/exiv2/exiv2 yet. I expect to do that integration early next week.

Robin

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 9 days ago

My apologies, https://github.com/clanmills/exiv2 is where I got the code I was testing.
To confirm, I downloaded it again and I am getting the same errors.

Using this last download, I also get errors for
D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>cmd/c "vcvars 2015 64 && cmakeBuild --dll --release "

or
D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc>cmd/c "vcvars 2015 64 && cmakeBuild --dll --release --ascii"

------
...
-- Looking for inttypes.h - found
-- Performing Test EXV_HAVE_DECL_STRERROR_R
-- Performing Test EXV_HAVE_DECL_STRERROR_R - Failed
-- None: /DWIN32 /D_WINDOWS /W3 /GR /EHsc
-- Debug: /MDd /Zi /Ob0 /Od /RTC1
-- Release: /MD /O2 /DNDEBUG
-- RelWithDebInfo: /MD /Zi /O2 /DNDEBUG
-- MinSizeRel: /MD /O1 /DNDEBUG
-- Compiler info: MSVC (C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe) ; version: 19.0.24215.1
-- ------------------------------------------------------------------
-- exiv2 0.26 configure results <http://www.exiv2.org>
-- Building shared library: YES
-- Building PNG support: YES
-- XMP metadata support: YES
-- Building static libxmp: YES
-- Native language support: NO
-- Conversion of Windows XP tags: YES
-- Nikon lens database: YES
-- Commercial build: NO
-- Building video support: NO
-- Building webready support: NO
-- Dynamic runtime override: YES
-- Unicode paths (wstring): NO
-- Building exiv2 command: YES
-- Building samples: NO
-- Building PO files: NO
-- ------------------------------------------------------------------
-- Configuring done
-- Generating done
-- Build files have been written to: D:/pkg/C++/MSVC2015/exiv2-master/contrib/cmake/msvc/work_Release/exiv2
The system cannot find the file specified.
"*** build errors in EXIV2 ***"


One of the issues I see, that when I use --unicode, the output seems to end up in the same directory tree as with --ascii.
This would mean, that one cannot have both coexist at the same time?
Or is this related to the fact that the 'make' does not run to completion?

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 9 days ago

Ah, this isn't good news. I'll have a look. Testing is good stuff, that's how we find out what's broken!

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 9 days ago

Good News: It's broken for me. Better News: I know why. Great News: The fix is easy!

It's in cmakeBuild.cmd at line#356. Can you comment off this line (rem = remark = comment)

    rem copy/y "%_WORK_%\exiv2\exv_conf.h" "%_INCPATH_%" 

Apologies. I never intended to submit that line of code.

I have a buildserver and it uses cmakeBuild.cmd to build on Windows. After I moved the Exiv2 code from SVN to GIT in April, the buildserver was not updated to build from GIT. It's on the TODO list and I haven't done it yet. The whole point in having a buildserver is to find stuff like this before users discover it!

CAUTION: I'm working on CMake Mac/Linux stuff to enable the Adobe XMPsdk to be built and linked into Exiv2. I don't intend to break the build - however it's possible. I appreciate your help by looking at this stuff. Very much appreciated. Thank You.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 9 days ago

Removing that line lets it all run to completion, but .... ;-)

the --unicode flag does not make it into the .sln file.
For the most part in any of my projects, the .sln files would also need the Multi-threaded Debug DLL (/MDd) option set for debugging code or Multi-threaded DLL (/MD) for release versions.
I have no idea what it would take to implement that.

I manually set those two options in the .sln file generated by

cmd/c "vcvars 2015 32 && cmakeBuild --static --debug --unicode" 

for exiv2.lib, which is all I really was interested in.

After importing the exiv2d.lib to my project, I end up either with undefined symbols or multiply defined symbols, depending on how much of the exiv2 code I also import.
The one other exiv2 item I seem to need is xmp_object.lib, but it conflicts with some symbols in the XMP SDK code I use. :-(

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 9 days ago

Great feedback, Arnold about --unicode which I will investigate on Friday. The xmp_obect was introduced recently by Luis. Luis is on vacation at the moment. Lots of broken stuff to be fixed at the moment. Thank you for your contribution.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 8 days ago

Ummmm. This seems mostly OK here. I'm wondering if you have a use case for the libraries that different from our sample apps.

Anyway, here's what I see:

I submitted a fix to cmakeBuild.cmd in github. I pulled down a fresh copy of the code from https://github.com/clanmills/exiv2: git clone https://github.com/clanmills/exiv2

I've built --dll with the command: cmd/c "vcvars 2015 64 && cmakeBuild --dll --unicode --samples --build"

I pulled down my test file curl -O http://clanmills.com/Stonehenge.jpg to c:\Users\rmills\Stonehenge.

I launched VS2015 and
1) opened the project: C:\Users\rmills\gnu\github\clanmills\exiv2\contrib\cmake\msvc\work_Debug\exiv2
2) set exifprint as the "startup target"
3) set the debugger command to C:\Users\rmills\gnu\github\clanmills\exiv2\contrib\cmake\msvc\dist\2015\x64\dll\Debug\bin\exifprint.exe
4) working directory to C:\Users\rmills\gnu\github\clanmills\exiv2\contrib\cmake\msvc\dist\2015\x64\dll\Debug\bin
5) set the command argument to c:\Users\rmills\Stonehenge.jpg
5) set the breakpoint in exifprint\exifprint.cpp #41 and debugged.

Perfect, it stepped in the UNICODE entry in the library.

I did much the same with --static. The debugger grumbled about being unable to locate .PDB files, however the program executed OK.

//////////

I don't know why UNICODE isn't set for you. CMake generates the file include/exiv2/exv_conf.h and that has UNICODE set for me:

C:\Users\rmills\gnu\github\clanmills\exiv2\contrib\cmake\msvc>grep UNICODE ..\..\..\include\exiv2\exv_conf.h
#define EXV_UNICODE_PATH

C:\Users\rmills\gnu\github\clanmills\exiv2\contrib\cmake\msvc>

//////////

My "policy" about /MD and that stuff is "ALL DLLs" or "No DLLs". So the --dll --debug is linked with /MDd. It's possible to force the setting in CMake with options such as: -DCMAKE_C_FLAGS_RELEASE=/MT -DCMAKE_C_FLAGS_DEBUG=/MTd and this is done by cmakeBuild.cmd when tell CMake to build expat --static build.

There's something wrong with the --static linking of libz. I fixed that some time ago, it should be linking zlibstaticd.lib. Luis has rewritten our CMakeLists.txt files and I think that subtle fix has been lost. I'll investigate.

//////////

Yesterday, I reverted Luis's code about xmp_object because I couldn't debug the XMP code using Xcode on the Mac. So, with today's code, we create and link xmpd.lib I'll discuss xmp_object with Luis when he's back from vacation next week. When he made that change, he requested a code review which I approved.

//////////

I am aware Exiv2 conflicts with Adobe XMPsdk. The major feature being added to Exiv2 v0.26.1 is to build and link with Adobe XMPsdk (2013 or 2014 or 2016).

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 8 days ago

Downloaded (cloned) the latest from https://github.com/clanmills/exiv2
as well as the test image and built the 64 bit versions first, ran the test with exivprint and all worked as you described.

Next I built the 32 bit versions and had some warnings re static not being defined, I believe it was for (some of) the samples.

When I tried to verify Unicode related compiler macros, I ran into a bit of an issue.
If possible, it would be helpful - for me, at least - if both output from 32 & 64 bit versions were separate so that they can exist side by side.
As it is right now, one overwrites the other and I would have to recompile one or the other as I want to switch between them.
-------------

At this time, in the 32 bit output from
cmd/c "vcvars 2015 32 && cmakeBuild --static --unicode --samples --build --debug"
in
D:\pkg\C++\MSVC2015\exiv2-master\contrib\cmake\msvc\work_Debug\exiv2\exiv2.sln

I see no trace of anything Unicode related, though I am unsure what the exiv2 --unicode build is expected to insert.

OTOH, perhaps, I should modify my expectations and usage of exiv2.lib.
Because my app(s) is/are all built using UTF-8 strings, I sort of blindly expected to also link a Unicode build of the exiv2 library.
Perhaps I need to stay with the plain 'ASCII' version and convert any strings I feed to the library to plain old MS wide strings.
I'll try to modify my test app and see how that works out.
This is usually how I have to treat the native Win32 API interface, with very few exceptions; IIRC correctly it is also how I have to treat the XMP SDK code - at least that what gave issues to start with. Those seems to have gone away, without my doing anything about my interface.
-------------

As to the DLL/static 'policy', I an only speak to what I have been 'taught' from my earliest attempts with wxWidgets.
We have the choice of linking the wxWidgets libs statically or dynamically, but the OS run-time library is usually linked dynamically.
-------------
As part of this, I have tried to build 2 separate versions of my app - one with XMP SDK and no exiv2 code, the other with exiv2, but no XMP SDK.
This is still a work in progress and I'll have to see how it works out.

RE: How to build exiv2 32-bit libraries using Cmake - Added by Robin Mills 8 days ago

I'm tucked up in bed (it's 10:40pm here) and I'm off for the weekend. Home Sunday evening.

cmakeBuild.cmd creates the dist/2015/mumble/bumble/jumble directories. 8 flavours of {win32|x64}x{static|dll}x{debug|release}. Your application should compile/link the one that's appropriate. The builds are created in the work_release or work_debug directories. The build artefacts are copied to dist for your use. (dist = Distribution).

I'll have a look at 2015/32 in the morning before I leave. I'm not aware of anything special about that configuration. I encourage you to use --unicode. I'm confident that it's working fine.

On the MSVC builds, I decided to have "All DLLs" or "All Static". With CMake, somebody complained that (static) exiv2.lib was using expat.dll and zlib.dll, so I fixed that to be 100% static. Somebody once requested exiv2.dll with static run-time libraries. That's one of the few requests that I have ever refused and is discussed in detail somewhere on the forum.

Curiously, I discovered this week that all builds on Mac and Linux (both static and shared versions of exiv2), use shared zlib and expat. Nobody's ever mentioned this. I think many folks have total fear of DLLs on Windows and like 100% static linking. I'm totally opposite and prefer 100% dlls. We can discuss my reasons another day.

I'm very happy that you are working with me on this. Luis is very enthusiastic to discontinue support for our msvc solution/project files and the autotools build environment. You've really challenged our CMake/Visual Studio support and the product is much stronger for it. Thanks very much. And I hope you'll continue to work with me when I get the external Adobe XMPsdk into a state in which it can be tested.

I'll take the laptop this weekend and will do email. I'm hoping to a have a couple of days away from building Exiv2!

RE: How to build exiv2 32-bit libraries using Cmake - Added by Arnold Wiegert 8 days ago

.... I was surprised you replied (already) :-)
No rush at this end really, since I have got the XMP SDK working.
Work on exiv2 now is more or less just out of interest ;-)
At this time I don't see anything I can't do with what I have, but as long as you are working along on this, I don't mind doing some testing and learning on the side.

Not to worry about taking a break; I have plenty to keep me occupied in the mean time and we'll carry on next week or whenever.

« Previous 1 2 (26-43/43)

Redmine Appliance - Powered by TurnKey Linux