Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2

Added by Tobias E. 3 months ago

Hi,

I am currently working on porting some software to Windows and there I need support for unicode filenames. I found EXIV2_ENABLE_WIN_UNICODE which seems to do what I want, however, I am compiling using msys2/mingw, and there is an #error that tells me it's not supported. Removing the #error line and not building the samples (exifprint.cpp fails due to the name of main()) I get a library that seems to work for me. Are there any caveats or known bugs wrt. unicode filename handling in msys2, or is that just unsupported code that was disabled to be on the safe side?

Tobias


Replies (13)

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 3 months ago

Tobias

You're a jump ahead of me. I haven't investigate Exiv2 + msys2 yet. After a great deal of frustration with Msys/1.0, I deprecated support for MinGW/1.0 when Exiv2 v0.26 was released in April. Although I hope to support msys2 with Exiv2 v0.27, I haven't started working on that yet.

The #error for unsupported build platform is for my benefit. However, this is open-source, the code is likely to build and run on unsupported platforms. I believe msys/2 is derived from Cygwin which is supported.

The sample exifprint.cpp is the only sample which supports EXV_UNICODE_PATH. It's 100% intentional that it defines _wmain() which is "wide" main() in Windows. It's possible that's not supported in msys2. I did that to have a "sanity" test that the EXV_UNICODE_PATH works! However, you won't be able to run the test suite in UNICODE as all the samples will require "wide" treatment.

I know you're busy (and so am I). If you want to provide patches relating to msys2, I would be very pleased to have your contribution.

Robin

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Tobias E. 3 months ago

Thanks for the quick reply Robin!

So far I haven't found anything that would require patching. Apart from the sample of course, but for just using the library that's not really an issue. And I have to admit that I know next to nothing about Windows, so I am not able to fix the sample anyway. :-)

I'll ask the msys2 folks if they can update their package to include Unicode support. That way it will get wider testing and we will see what breaks. Crowd sourced CI.

Tobias

PS: Am I right in assuming that adding the wstring functions to the classes will break the ABI of the library, requiring a recompile of every program using it?

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 3 months ago

Yes, I think you'll have to rebuild the library. Functions such as Exiv2::ImageFactory::open(...) will require unicode strings.

You should be able to put something like this into samples/exifprint.cpp to make it compile.

#if defined(_MSYS2_) and defined(EXV_UNICODE_PATH)
#undef EXV_UNICODE_PATH
#endif
I don't know what pre-processor symbols msys2 will put into your code. You can inspect them with
$ echo '' > empty.cpp ; gcc -E -dM empty.cpp | less

EXV_UNICODE_PATH was added before I joined the project 10 years ago. An alternative, and in my opinion better, way to do this is to support both UNICODE and ASCII/Utf8 entry points for all file system functions. The ascii version (on Windows), "bump ups" path arguments to UNICODE and calls their wide brother. However, I don't believe that is what was done. I've never dealt with unicode paths on Linux. I'm not sure there is such a thing. I believe fopen and the like support Utf8. So on Linux, the wide versions should "bump down" their path arguments to Utf8 and call their brother.

I'm not sure that it's a good idea to ask the msys/2.0 people to build using EXV_UNICODE_PATH. I suspect it will break the command-line program exiv2(.exe).

Perhaps, I'll revisit EXV_UNICODE_PATH for v0.27 so that Windows builds of Exiv2 (cygwin, msys/2.0 and msvc) offers both ascii and wide path APIs. I won't change MacOS-X and Linux.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 3 months ago

I've had a look at that code and built it on msys/1.0 with/without EXV_UNICODE_PATH. I commented off the error:

#error EXV_UNICODE_PATH is not supported for MinGW builds
The library builds. It fails to compile/link exiv2(.exe)

libexiv2 always offers the "ascii/utf8" file system interface.

244 -64- /home/rmills/temp/trunk> nm -g --demangle src/.libs/libexiv2.a|grep ImageFactory|grep open|grep ' T '
0000000000001dc0 T Exiv2::ImageFactory::open(unsigned char const*, long)
0000000000001c90 T Exiv2::ImageFactory::open(std::string const&, bool)
0000000000001a40 T Exiv2::ImageFactory::open(std::auto_ptr<Exiv2::BasicIo>)
245 -64- /home/rmills/temp/trunk>
When you build with -DEXV_UNICODE_PATH, you get:
248 -64- /home/rmills/temp/trunk> nm -g --demangle src/.libs/libexiv2.a|grep ImageFactory|grep open|grep ' T '
00000000000035a2 T Exiv2::ImageFactory::open(unsigned char const*, long)
0000000000003430 T Exiv2::ImageFactory::open(std::basic_string<wchar_t... > const&, bool)
00000000000032be T Exiv2::ImageFactory::open(std::string const&, bool)
000000000000370a T Exiv2::ImageFactory::open(std::auto_ptr<Exiv2::BasicIo>)
249 -64- /home/rmills/temp/trunk>
As you see, the additional entry point to open wchar_t has been added.

I modified samples/exifprint.cpp a couple of years ago to be test that EXV_UNICODE_PATH really worked on msvc builds. All the other sample applications (including exiv2.exe) use the ascii/utf8 file system interface.

EXV_UNICODE_PATH/Cygwin does not build, although I don't explicitly error and report. I don't believe this should build. Cygwin isn't Windows. It's a Unix sandwich above Windows. However, MinGW msys/1.0 is Windows and should offer unicode paths. I don't know if msys/2.0 should be able to support EXV_UNICODE_PATH, however I think it's rather likely. I will not support msys/1.0 in future. It has been a time sink.

EXV_UNICODE_PATH is only ready for use on msvc builds. If you'd like us to investigate offering this on msys/2.0, can you open an issue in Redmine "Provide support for msys/2.0 with EXV_UNICODE_PATH".

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 3 months ago

Tobias

I've tried to built exiv2 v0.26 with msys 2.0

1) build with ./configure

I had some difficulties building/installing expat and zlib and had to create and copy files to /usr/local/lib and /usr/local/include. I had to use ./configure --prefix=/usr/local with expat and exiv2.

I got Exiv2 to compile and link the library. There were lots of grumbles about auto-ptr being deprecated. I think the default compiler on msgs/2.0 is C++11. We're adding support for C++11 in Exiv2 v0.26.1

The exiv2 command-line utility doesn't link. There are messages coming from the version of libtool included with exiv2. It builds a <exiv2>/bin/.libs/exiv2.exe which appears to be dead on arrival.

2) building with CMake

CMake complained a lot, however it did produce a makefile

When I run make on the CMake/makefile, I get thousands of warnings and messages. I've seen this kind of thing on MacOS-X being caused by c++/gnu++ library conflict. I think this will resolve when Exiv2 has support for C++11.

3) UNICODE support on Exiv2 v0.26

I've been working with Arnold on CMake UNICODE and static builds and discovered issues with the v0.26 release. We're working together to resolve/fix/test these and hope they will be working correctly on Exiv2 v0.26.1. http://dev.exiv2.org/boards/3/topics/2922

4) How to move forward from here

There are too many things to fix here to include support for msys/2.0 in Exiv2 v0.26.1. I hope to revisit this in 2018.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Tobias E. 3 months ago

2) building with CMake

CMake complained a lot, however it did produce a makefile

When I run make on the CMake/makefile, I get thousands of warnings and messages. I've seen this kind of thing on MacOS-X being caused by c++/gnu++ library conflict. I think this will resolve when Exiv2 has support for C++11.

I didn't try 0.26 but current git. And that compiled without any complaints.

3) UNICODE support on Exiv2 v0.26

I've been working with Arnold on CMake UNICODE and static builds and discovered issues with the v0.26 release. We're working together to resolve/fix/test these and hope they will be working correctly on Exiv2 v0.26.1. http://dev.exiv2.org/boards/3/topics/2922

When compiling with Unicode support there was the one issue I mentioned in my initial post (samples don't compile but I don't care about those in this case). Apart from that it was smooth sailing. Maybe I didn't enable some feature that might be broken. In subsequent tests the resulting exiv2.exe didn't seem to be more broken than the one without Unicode support. May I ask you for the cmake output when running in a clean build directory? I am curious to see the differences to mine and maybe understand why I don't see any problems.

4) How to move forward from here

There are too many things to fix here to include support for msys/2.0 in Exiv2 v0.26.1. I hope to revisit this in 2018.

To be honest, I don't see what changes there are needed.

Unfortunately I have to use msys2/mingw64 as the Microsoft compiler is lacking too many essential things to be considered useful for me. So gcc and clang are my only real options.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 3 months ago

To be honest, I don't see what changes there are needed.

I'm giving myself a way out. I don't want to promise this and be unable to deliver! I opened an issue on Redmine to support this. http://dev.exiv2.org/issues/1310

If you're building successfully from GitHub, that's wonderful. When I add/fix stuff for Arnold, it's possible everything will work fine! And I'm making good progress for Arnold and it's likely that the new code will be available later this week. Perhaps you could try again at the weekend and give me feedback.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Tobias E. 3 months ago

Sure thing, I will have an eye on all new commits. :-)

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 2 months ago

Tobias:

I've done a LOT of work this week on contrib/cmake/msvc/cmakeBuild.cmd and all the CMakeLists.txt files: http://dev.exiv2.org/boards/3/topics/2922?r=2962#message-2962

I'm working very hard on Exiv2 v0.26.1 and don't want to support msys/2.0 at this time. You are welcome to test the new CMakeLists.txt files on msys/2.0 and provide feedback.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Tobias E. 2 months ago

After removing the #error in config.h compilation worked. I didn't try the resulting .dll. I had to pass -DBUILD_SHARED_LIBS=On though as that is no longer the default case and not even an explicit CMake option. I didn't try using the static lib so far, maybe that would be even easier.
The complete cmake command used was thus:

$ cmake -G "MSYS Makefiles" -DEXIV2_ENABLE_WIN_UNICODE=On -DBUILD_SHARED_LIBS=On ..

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 2 months ago

Thanks for looking at this. Yes -DBUILD_SHARED_LIBS=1 is a CMake flag, so we removed -DEXIV2_ENABLE_SHARED https://cmake.org/cmake/help/v3.0/variable/BUILD_SHARED_LIBS.html

I built Exiv2 on msys/2:

$ mkdir build ; cd build ; cmake .. ; make 
I think it used the "Unix Makefiles" generator because when I tried -G "MSYS Makefiles" it said it didn't have that generator! I've tried unsuccessfully to build CMake 3.9.2 and CMake 3.8.2 from source, so I'm stuck with the default $ pacman -S cmake with is 3.6.2

The preprocessor symbol for msys2 appears to be __MSYS__. I didn't trigger the "MinGW escape" in include/exiv2/config.h

After working with you and Arnold on EXV_UNICODE_PATH, I think I will make EXV_UNICODE_PATH the default in Exiv2 v0.27. Then you don't need a private build. I won't change this in v0.26.1 because "dot" releases should have the same API & ABI.

This looks promising and I hope to support msys/2 soon. Thank You for working with me on this.

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Tobias E. 2 months ago

That sounds great!

When trying to compile in msys2 you should run the MinGW shell (64 to build 64bit binaries, 32 to build for 32bit) and do your stuff in there, not in the MSYS shell. The latter is mostly for system maintenance, like installing software and updating with pacman. Maybe that's where the differences come from?

RE: Support of EXIV2_ENABLE_WIN_UNICODE in MSYS2 - Added by Robin Mills 2 months ago

Ah, I didn't know about the difference between MSYS and the MinGW offerings. I mistakenly thought MinGW was msys/1.0. I'll spend time investigating this tonight. We're almost there, thanks to you.

(1-13/13)

Redmine Appliance - Powered by TurnKey Linux