Bug #776
Some tests depend on system settings (locale, path, etc)
100%
Description
The following test drivers fail if another system locale is set:
- bugfixes-test.sh
- conversions.sh
- exiv2-test.sh
- modify-test.sh
- xmpparser-test.sh
This is because the messages are translated, and because the ordering of files is changed due to a different collation.
In addition, the following error occurs:
Running exiv2-test.sh ... Files ./tmp/exiv2-test.out and ./data/exiv2-test.out differ 5,6c5,6 < /home/.../exiv2-trunk/src/exiv2 < exiv2 0.21.1 (32 bit build) --- > /usr/local/bin/exiv2 > exiv2 0.21.1
It seems that the exiv2-test.sh driver expects the exiv2 binary not to be used from the source tree, and expects the system not to be 32-bit.
Associated revisions
#776: Adjust the exiv2-test.sh driver to respect the " (__ bit build)" suffix of the first line of "exiv2 -V"
#776: Unified the mechanism by which the test drivers look for binaries (exiv2, samples, etc.), added support for EXIV2_BINDIR, removed the call to "which exiv2"
History
Updated by Volker Grabsch over 10 years ago
I just fixed the locale issue (r2481), but I'm not sure how to fix the remaining issue with the exiv2-test.sh driver.
Updated by Volker Grabsch over 10 years ago
- File use_always_local_exiv2_binary_in_tests.patch added
Okay, so I also fixed the "(32/64 bit build)" issue via sed. (r2495)
The remaining issue is the "/usr/local/bin/exiv2" stuff. I think there is a deeper issue here. Some test drivers are using the local exiv2 binary ("../../src/exiv2"), while other drivers are using the system-wide installed exiv2 binary ("which exiv2").
I'm not sure which of those is the intended way, but I personally think that it is inconventient to always have to run "make install" before running the test suite. So I propose to use always the local exiv2 binary. If we do that, there is no need to call "which exiv2" anymore, which solves the "/usr/local/bin/exiv2" issue. See the attached patch. Should I commit it?
As a side issue: This wouldn't fix the whole issue, as some tests depend on binaries in ../samples which need "make install" before they can be built. This is because their Makefile uses pkg-config. So maybe we should fix that, too?
Updated by Andreas Huggel over 10 years ago
This is a bit a hairy issue.
- The library must be installed to build the samples. This means the samples are like an external project and serves as a test to ensure that the installed library and headers can actually be used to build binaries. I think this is an important test and it shouldn't be dropped.
- When compiling with MSVC on Windows, all binaries end up in some other directory (msvc\bin\<release>\) and the test scripts can't be used as they are currently even though with MinGW for example, they would actually run.
So ideally it should be able to easily switch the location where exiv2 and test drivers are expected. The default location should remain the exiv2 install path and samples/ and it should be possible to use a different location (like msvc\bin\<release>\) for all.
Andreas
Updated by Volker Grabsch over 10 years ago
I'm not sure if I understand you correctly here.
Under Windows, the test suite can only run when MinGW/MSYS is installed. However, if Exiv2 is built with MinGW/MSYS, the exiv2.exe would be created in src/exiv2.exe, not in msvc/bin/<release>/. Also, the samples couldn't be built at all, unless MSYS provides enough support for "make install" and pkg-config. In every case, the test suite would either run without change, or would not run at all.
Also note that giving a correct path to exiv2 binary and samples is tricky, because those have to be relative to test/tmp/, i.e. not relative to test/. However, this can be solved by requesting absolute paths. And this issue already exists with VALGRIND, too, so it is probably a non-issue.
Nevertheless, it should be possible to introduce some variable like EXIV2_BINDIR that could be set to /absolute/path/to/exiv2/msvc/bin/<release>/ or similar. If it is set, exiv2 and samples are taken from there. Otherwise, they are taken from src/ and samples/. Is that the solution you were thinking of?
Updated by Andreas Huggel over 10 years ago
I'm not sure if I understand you correctly here
Agreed, compiling and testing with MinGW/MSYS works like on Linux, i.e., the test scripts can (generally) be used.
However, when compiling with MSVC, the test scripts cannot be used. Of course the scripts won't run at all without MinGW/MSYS but besides that they expect the compiled binaries in the wrong locations. If MinGW/MSYS is available it is possible, e.g., to copy the MSVC-compiled binaries into the folders expected by the test scripts and then run them.
Nevertheless, it should be possible to introduce some variable like EXIV2_BINDIR that could be set to /absolute/path/to/exiv2/msvc/bin/<release>/ or similar. If it is set, exiv2 and samples are taken from there. Otherwise, they are taken from src/ and samples/. Is that the solution you were thinking of?
Yes, that sounds like a possible solution.
Andreas
Updated by Volker Grabsch over 10 years ago
- Status changed from New to Feedback
Andreas Huggel wrote:
Nevertheless, it should be possible to introduce some variable like EXIV2_BINDIR that could be set to /absolute/path/to/exiv2/msvc/bin/<release>/ or similar. If it is set, exiv2 and samples are taken from there. Otherwise, they are taken from src/ and samples/. Is that the solution you were thinking of?
Yes, that sounds like a possible solution.
Okay, so I implemented this. I had to touch all test drivers for that purpose. I removed some variables such as "$binpath" which had different meanings in various test drivers.
Please review. (r2506)
Updated by Volker Grabsch over 10 years ago
- File deleted (
use_always_local_exiv2_binary_in_tests.patch)
Updated by Volker Grabsch over 10 years ago
One more notice:
EXIV2_BINDIR is not required to be an absolute path. It may also be a relative path, in which it is meant to be relative to the test/tmp/ directory, such as: ../../msvc/bin/<release>/
Updated by Volker Grabsch about 10 years ago
- Status changed from Feedback to Resolved
I just confirmed that the test suite runs well under Debian and Mac OS X.
#776: ensure that the test drivers are not confused by the locale settings