Exiv2

Good News: I've set this up on a Mac Mini between Ubuntu 14.04 (client Virtual Maching on Mac) and Windows7 (server Virtual Machine on Mac). 3822 jpgs (all small files from my web site) - 358mb total. So an average of about 100k/file. No errors, no hangs using your command-line.

Tomorrow, I'll test:

1. Ubunutu (Virtual Machine on Mac) <-> Ubuntu (Real machine) on Laptop.
2. Larger files

Robin Mills wrote:

Thanks for reporting this, Calvin. Destroying your file isn't good. I'll have a look at this in the next few days.

Two thoughts come to mind:

1. Have you tried this on v0.24, or (even better) on our trunk?
2. File is mounted with CIFS. There was an issue with Windows/Virus checker locking files. http://dev.exiv2.org/issues/984 I see you've said you're using Ubuntu server. Is there any possibility that another process is locking any of those files? For example, is shotwell or DigiKam busy indexing/thumbnailing files while you're updating all this metadata?

So - initially it manifested itself via shotwell, and I suspected some kind of locking issue.
Which is why I went to the above.
The only possible other thing interacting with the files would be plex, but that's periodic, read only and direct on the server, and doesn't fit the behaviour.

Just to be clear, it's Ubuntu 14.04.2 LTS on both the CIFS client & server.

I'll see about getting 0.24 going and test that.

Random faults are always hard to diagnose. However I will try to reproduce this with Ubunutu <-> Ubunutu

Ok - it happens every time (ie before the end of the 4000 files). Which one it picks is random. It's been the first one, it's been the nth (where n is <4000), but it's been every time.

Thanks for the effort so far.

Robin

~/exiv2-0.24/bin/exiv2 -V
exiv2 0.24 001800 (64 bit build)

for i in `ls`; do ~/exiv2-0.24/bin/exiv2 -v -M"set Exif.Photo.UserComment charset=Ascii New Exif comment4" $i; done
File 1/1: 1393827856454.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment4" (Comment)
File 1/1: 1393837729884.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment4" (Comment)

(hanging:

ls l 1393837729884.jpg
-rw-r--r- 1 calvin 500 0 Mar 16 07:39 1393837729884.jpg
)

hmmmm.... now where do i find the trunk...... lemme look.

Calvin Browne wrote:
<SNIP>

)

hmmmm.... now where do i find the trunk...... lemme look.

oh dear - no ./configure file with......

svn checkout svn://dev.exiv2.org/svn/trunk

Calvin Browne wrote:

Calvin Browne wrote:
<SNIP>

)

hmmmm.... now where do i find the trunk...... lemme look.

oh dear - no ./configure file with......

svn checkout svn://dev.exiv2.org/svn/trunk

of course - reading the README helps ;-)

~/exiv2-trunk/trunk/bin/exiv2 -Vexiv2 0.24 001800 (64 bit build)
Copyright (C) 2004-2013 Andreas Huggel.

About 50 lines.......

File 1/1: 20150131_192840.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment5" (Comment)

(hanging
ls l 20150131_192840.jpg
-rw------ 1 calvin 500 0 Mar 16 08:09 20150131_192840.jpg )

:(

Calvin

There is almost no useful information in your last 4 or 5 reports. To be clear, I said I would set up and test Ubunutu <-> Ubuntu today and also investigate with larger files.

Can I infer that you have built the trunk and are using that now? And ls of hanging/empty file does not tell me anything new. Is this the first file, or after 500 files. Is it always the same file.

Please help me to help you. I'm going to have my breakfast (it's 7:42am in England) and perform the tests I said I would try. I'll report when I've done that.

Robin Mills wrote:

Calvin

There is almost no useful information in your last 4 or 5 reports. To be clear, I said I would set up and test Ubunutu <-> Ubuntu today and also investigate with larger files.

Can I infer that you have built the trunk and are using that now? And ls of hanging/empty file does not tell me anything new. Is this the first file, or after 500 files. Is it always the same file.

Please help me to help you. I'm going to have my breakfast (it's 7:42am in England) and perform the tests I said I would try. I'll report when I've done that.

Apologies for not being very articulate.

Yup - I first tried the 2.4 tarball, and then the svn trunk.

The file truncation can happen on the first file, or the n-th file. It seems random.

I've reproduced this using the following setup. Client is a Linux VM running on the MacMini. Server is a laptop. Both are connected to the router by ethernet cable.

client: Linux rmillsmm-kubuntu 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
server: Linux rmills-ubuntu 3.13.0-30-generic #54-Ubuntu SMP Mon Jun 9 22:45:01 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

As before, I've copied some images from my website to a test directory. 8000 images for a total size of 600mb. Average about 100k/image.

As you've said, it hangs (for me on the third file) and ^C breaks - level the file with length zero. Not Good!

1043 rmills@rmillsmm-kubuntu:~/smb4k/RMILLS-UBUNTU/rmills-ubuntu/home/rmills/temp/exiv2test $ for i in `ls`; do exiv2 -v -M"set Exif.Photo.UserComment charset=Ascii New Exif comment3" $i; done
File 1/1: 0001_2_3.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment3" (Comment)
File 1/1: 0001.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment3" (Comment)
File 1/1: 0011_2_3.jpg
Set Exif.Photo.UserComment "charset=Ascii New Exif comment3" (Comment)
^C
1044 rmills@rmillsmm-kubuntu:~/smb4k/RMILLS-UBUNTU/rmills-ubuntu/home/rmills/temp/exiv2test $

1063 rmills@rmillsmm-kubuntu:~/smb4k/RMILLS-UBUNTU/rmills-ubuntu/home/rmills/temp/exiv2test $ ls -alt 001*.jpg 0001*.jpg
-rwxrwxr-x 1 rmills rmills 68998 Mar 16 08:08 0001_2_3.jpg
-rwxrwxr-x 1 rmills rmills 75647 Mar 16 08:08 0001.jpg
-rwxrwxr-x 1 rmills rmills 77930 Mar 16 08:08 0011_2_3.jpg
-rwxrwxr-x 1 rmills rmills 82040 Mar 16 08:08 0011.jpg
-rwxrwxr-x 1 rmills rmills 28702 Mar 16 08:04 001.jpg
1064 rmills@rmillsmm-kubuntu:~/smb4k/RMILLS-UBUNTU/rmills-ubuntu/home/rmills/temp/exiv2test $

Here's what I'm thinking:

1. Is this peculiar to Ubuntu <-> Ubuntu CIFS, or can it impact any/every network
2. Is it Memory Mapping?

Concerning the first, I will try Ubuntu <-> Windows (smb/cifs) and Ubuntu <-> Ubuntu (NFS) and Ubuntu <-> Mac (Parallels magic and smb/cifs).

The second point is that we use Memory Mapping to perform I/O when it is available. When MM is not available, we use the "C" standard I/O fopen,ftell,fseek etc. Perhaps MM or the "switch to fopen" mechanism is not working correctly over CIFS. I'll build exiv2 and to always ignore MM and see what happens.

Please be patient and give me time to investigate this. I'm an unpaid volunteer and have other things in my life. I thank you for bringing this to my attention and assure you that I will work on this and report my findings.

Glad you were able to reproduce - shout if there's something you want me to try - and I completely understand how this works, so no time pressure from my side ;-)

Robin Mills wrote:

Calvin

Thanks for understanding. I've tested the following:

1. Linux <-> Linux with Memory Mapping Disabled. Hang. As earlier.
2. Linux <-> Windows. Given up. Can't get any smbclient on Ubuntu to accept my password for the Windows Laptop
   (I've tried smb4u, konqueror, dolphin and command-line)

Ok - so I resurrected my old windows xp laptop, shared a directory, mounted it under my linux client ( sudo mount -o uid=1000,gid=500 //192.168.2.145/test /media/test/ -t cifs ), copied a couple of hundred pics across and ran the command, and it hung and nulled the 29th file, and again about the 100th one.

Is there a windows binary for exiv2 that I should try this the other way round?
I'm going to try nfs and see what happens there...

Thank you for making the effort to test and report your result. Much appreciated. Together we can find and fix this matter.

I've build v0.23, v0.24 (both from the tar-ball on exiv2.org) and trunk this morning as MSVC 32bit static builds. No DLLs needed. I've zipped and posted them to: http://clanmills.com/files/exiv2-32bit.zip The trunk is the most interesting build. If that fails, it's unlikely that v0.24 and v0.23 will be OK.

I've provided those builds, so there's no need to run the command exiv2 -vV (verbose version) which reveals much more build information than plain exiv2 -V.

I did some googling into Ubuntu 14.04/Samba client connection. Lots of unhappy users - however this could be a coincidence. We keep digging for a while yet!

Thanks for working on this.

Robin Mills wrote:

http://clanmills.com/files/exiv2-32bit.zip The trunk is the most interesting build. If that fails, it's unlikely that v0.24 and v0.23 will be OK.

Redirect loop :(

--Calvin

Robin Mills wrote:

How odd. It has indeed be removed from the server. I've restored it. And I've attached a copy.

gotit - will test in the morning...
will also finish the nfs test...

hi,

Robin Mills wrote:

1. Is it Memory Mapping?

probably related: http://dev.exiv2.org/issues/1043

according to man mount.cifs, cache=none aka directio "precludes mmaping files on this mount […] Note too that no matter what caching model is used, the client will always use the pagecache to handle mmap'ed files. Writes to mmap'ed files are only guaranteed to be flushed to the server when msync() is called, or on close()." that might be relevant :-)

with kind regards,
t.