Project

General

Profile

Feature #858

SVN repository download size

Added by Daniel Kaneider about 9 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Robin Mills
Category:
miscellaneous
Target version:
0.25
Start date:
11 Oct 2012
Due date:
% Done:

100%

Estimated time:
12.00 h

Description

Hi,

in every project where features are continuously added, extensive testing is pretty much important. It is also understandable that exiv2 needs a lot of input images, in order to achieve useful testing. But do you really need nearly 80MB of test data?
Checking out the svn repository takes ages. And since I prefer (fixed) svn versions for building, building time goes up. I observed that many images are tried to keep small, but the recently added videos alone take nearly 50MB!!!

Therefore I suggest:

  1. Reduce the size of the test videos! IMHO they don't need to be 2 minutes long. Also reducing them in size could help a lot. Moreover I'm not sure if you are allowed to include a Vanilla Sky sequence in your repo (video-matroska.mkv). But again, I don't see a big benefit with those, which couldn't have achieved with smaller test videos.
  1. Split the repo up. Keeping your repo all together is important, I agree. Providing a possibility to exclude the test data can be important, too. Since you are using a subversion repository, you should have a look on svn:externals.
    In this way you could define the test data external to the main repository. If users simply do checkout/updates they get those test data included by default. Then doing a checkout using svn co --ignore-externals, is a way to reduce the download size.

Best,
Daniel

Associated revisions

Revision 2918 (diff)
Added by Robin Mills about 9 years ago

Fix: #858. Download video data on demand. More discussion in issue report.

History

#1

Updated by Robin Mills about 9 years ago

  • Assignee set to Robin Mills
  • Target version set to 0.24
  • Estimated time set to 12.00 h

Daniel

I think I'm in agreement about this. I'd quite like to keep the test data in a different branch and the test scripts can get the test files on demand. In that way, the user doesn't need to know about 'svn co --ignore-externals' and a regular 'svn checkout' is sufficient. The data will be checked out once of course.

I hear your comments about the video test data. Our video code engineer is a student and developed this as a Google Summer of Code Project. He's back at college and has many other matters on his mind. So while I'm sure the video data could be reviewed and put on a diet, I'm not optimistic about having that done any time soon. We're not a business, so we cannot prioritize/assign tasks to team members and then hold them accountable in their annual review!

I'll assign this issue to myself and do something about having the test data downloaded (once) on demand. This should decrease the size of our build tree from 93 to 16mb - a worthwhile improvement for all users.

#2

Updated by Daniel Kaneider about 9 years ago

In the case of a splitted repo, the size of the test data becomes less important.
Btw. this download process could also be done inside CMake using an option. If you split the repo up, I can provide you a patch.

#3

Updated by Robin Mills about 9 years ago

  • Status changed from New to Assigned
  • % Done changed from 0 to 20

Daniel

Thanks for you offer for a patch. I've set up a new branch in our repository to hold test data. My intention is to pull down the test files on demand from test/Makefile and avoid involving CMake (or CTest) in this.

Our friend Shawn in China is working on Issue #850 which to make the test suite run correctly from CMake. Shawn and I have been talking about using CTest, and adding code to CMakeLists.txt to run the tests from CMake. However Shawn I don't want to complicate Shawn's work at this time. I believe that pulling down the tests in test/Makefile is a simple way forward.

#4

Updated by Robin Mills about 9 years ago

  • % Done changed from 20 to 60

Stage 1 fix submitted: SVN: 2918.

I've moved the video test files to a different location in our repository and they are downloaded on demand by the test suite. This is removes 45mb of video data. When we make our 0.24 release (expected late November), I'll move the eps data (24mb). I can't remove that at present because it will break the published test notes on our web site at: http://www.exiv2.org/download.html

I have retained all other tests for convenience. With 0.24, the total size of a clean export of the trunk will be just over 20mb.

Additionally, I've subdivided the tests into:

make tests - runs the 'basic' tests
make teste - runs eps tests
make testv - runs video tests

#5

Updated by Daniel Kaneider about 9 years ago

Thank you Robin. Even though I consider downloading the test branch in CMake the better solution, your solution is fine for me. I'll see forward for the 0.24 release!

#6

Updated by Robin Mills about 9 years ago

Daniel

I know I'd like to learn more about CMake from you, so I'm interested to know how to achieve this with CMake. However we're not ready yet to abandon ./configure build and test. As you know, changing config/configure.ac is seldom enjoyable.

However good ideas are gold. If you want to share, I'm happy to learn. On the other hand, I know you're busy and I don't want to burden you with another task.

Robin

#7

Updated by Daniel Kaneider about 9 years ago

Well, downloading the repo in CMake is quite easy with
FIND_PACKAGE(Subversion)
Even for windows it's easy, since the program path can be passed to CMake on the command line, if the defaults don't work. I would say you would need around 5 lines of code to checkout the repo in CMake. However this shouldn't replace all the work which is currently done in config/configure, and this definitely requires much more effort. But if you do a svn co in configure or in CMake is more or less the same effort to do, except the fact the CMake could replace the makefiles somedays...

#8

Updated by Robin Mills about 9 years ago

Daniel

Thank you for those suggestions. I'm hoping that we'll be able to dump Makefile/.vcproj/.sln/.ac files by the end of next year. I hope to say in the 0.24 release notes (Nov 2012) that 0.25 (May 2013) will be the final release in which msvc/msvc64/config directories will be be provided. When we have a CMake only build, we'll use CTest/CDash to drive the tests. However between now and then, what's been done is sufficient and works on all our platforms.

Robin

#9

Updated by Robin Mills over 8 years ago

  • Target version changed from 0.24 to 0.25

The video code have been moved in 0.24. The EPS test files will be moved in 0.25.

#10

Updated by Robin Mills almost 7 years ago

  • Status changed from Assigned to Resolved

eps files have been moved to testdata/trunk and make teste downloads files on demand.

The data copied using svn export svn://dev.exiv2.org/svn/trunk has been decreased from 52 to 29mb 

$ rm -rf trunk; svn export svn://dev.exiv2.org/svn/trunk --revision 3200 >/dev/null ;du -m trunk | tail -1 
52    trunk
$ rm -rf trunk; svn export svn://dev.exiv2.org/svn/trunk --revision HEAD >/dev/null ;du -m trunk | tail -1 
29    trunk
This has reduced a "compressed" trunk from 12 to 7mb. 
$ tar czf t.tar.gz t 
$ ls -alt t.tar.gz
-rw-r--r--  1 rmills  staff  7350265 27 Dec 11:42 t.tar.gz

#11

Updated by Robin Mills over 6 years ago

  • % Done changed from 60 to 100
#12

Updated by Andreas Huggel over 6 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF