Exiv2

I can't think of any reason why exiv2 would "fail to open file" on a remote drive.

Two questions:
1) Can you edit/update ascii files on the remote drive with your favourite text editor?
2) Can you run regular linux utilities on the file. For example:

$ od -a /media/300gb/RPi1-Pictures/test1.jpg | head -4 

I have a Linux VM on my MacBook-pro running Parallels. Parallels installs networking magic that exposes the Mac's ~ directory as /media/psf/Home. I can seamlessly update files on the Mac from Linux. For example (using my favourite test file which you can download from https://clanmills.com/Stonehenge.jpg.

568 rmills@rmillsmbp:~/bin $ exiv2 -pa --grep datetime/i ~/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28   <-- 20:25
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54

On Linux:

rmills@rmillmbp-ubuntu:~$ od -a  /media/psf/Home/Stonehenge.jpg | head -4
0000000 del   X del   a   ;   8   E   x   i   f nul nul   I   I   * nul
0000020  bs nul nul nul  vt nul  si soh stx nul dc2 nul nul nul dc2 nul
0000040 nul nul dle soh stx nul  ff nul nul nul   $ nul nul nul dc2 soh
0000060 etx nul soh nul nul nul soh nul nul nul sub soh enq nul soh nul
rmills@rmillsmbp-ubuntu:~$ exiv2 -pa --grep datetime/i  /media/psf/Home/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
rmills@rmillsmbp-ubuntu:~$ rmills@rmillmbp-ubuntu:~$ exiv2 -a -00:01:00 /media/psf/Home/Stonehenge.jpg # Adjust age by negative one minute
rmills@rmillsmbp-ubuntu:~$ exiv2 -pa --grep datetime/i  /media/psf/Home/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:24:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:37:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:37:54
rmills@rmillmbp-ubuntu:~$ 

Back on the Mac, Stonehenge is a minute older!

569 rmills@rmillsmbp:~/bin $ exiv2 -pa --grep datetime/i ~/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:24:28 <-- 20:24
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:37:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:37:54
570 rmills@rmillsmbp:~/bin $ 

I vaguely remember a discussion about Exiv2 being unable to correctly access CIFS file-system files. I think CIFS is SMB, is it not? I use SMB to connect my MacOS-X machines. Since the last couple of upgrades to the OS (High Sierra and Mojave), AFS doesn't seem to work well for any purpose and this has nothing to do with Exiv2 - it's system wide.

I have thoughts about this matter and I'm sure we'll discuss this more in the days ahead. Together we'll solve this. I'm confident of our success in discovering the reason and remedy.

I can confirm this problem with CIFS file system mounted under Linux :

https://bugs.kde.org/show_bug.cgi?id=338462

The digiKam console trace indicate that image editor stop while metadata are writen in remote file when image is saved after modification.

note : and yes, CIFS is Samba stuff...

Gilles Caulier

Thanks for this response, Gilles.

I'll look at this next week (starting Monday 2018-11-19). I'm on vacation (long weekend running in Wales) and when I get home, my priority is Exiv2 v0.27 RC2 for Friday 2018-11-16. Incidentally, we're aiming for RC3 on 2018-12-07 and GM on 2018-12-28. I hope there will be no code changes between RC3 and GM. Thank You very much for your input on Localisation. As always, Luis is doing very good work.

Thanks Robin and merci Gilles,

Yes indeed my 'nas' router (linux / opensource) choose Samba/CIFs to share files using USB port.

Happy to say I kept my Digikam and Piwigo images and db on linux/NFS.

I needed to keep recoding hedgehog hibernation activity so last night I switched my images recording away from the CIF/Samba and confirm my metadata is back! So it seems the issue lies with the CIFs/Samba handling of metadata.

In response to Robin's question all seems OK:-

1. can edit simple text file

andrew@VESA-ubuntu:/nfs/all/USB1Tb$ touch test.txt
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ nano test.txt
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ cat test.txt
hello
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ ls -l test.txt
-rwxrwx--- 1 andrew andrew 6 nov.  11 13:54 test.txt
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ 

2. od utility

andrew@VESA-ubuntu:/nfs/all/USB1Tb$ od -a test.txt
0000000   h   e   l   l   o  nl
0000006

With this success I tried a manual simulation of my workflow using exiv2 on a single image, captured on raspberrypi (raspbian) copied to CIFs/Samba and this seems to work:-

andrew@VESA-ubuntu:/mnt/RPi3-home-pi/RPi1-Pictures$ exiv2 -g DateTime -pt 154192999801.jpg
Exif.Image.DateTime                          Ascii      20  2018:11:11 10:53:18
Exif.Photo.DateTimeOriginal                  Ascii      20  2018:11:11 10:53:18

andrew@VESA-ubuntu:/mnt/RPi3-home-pi/RPi1-Pictures$ cp 154192999801.jpg /nfs/all/USB1Tb/
andrew@VESA-ubuntu:/mnt/RPi3-home-pi/RPi1-Pictures$ cd /nfs/all/USB1Tb/

andrew@VESA-ubuntu:/nfs/all/USB1Tb$ exiv2 -g DateTime -pt 154192999801.jpg
Exif.Image.DateTime                          Ascii      20  2018:11:11 10:53:18
Exif.Photo.DateTimeOriginal                  Ascii      20  2018:11:11 10:53:18

andrew@VESA-ubuntu:/nfs/all/USB1Tb$ exiv2 -g DateTime ex -f 154192999801.jpg
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ ls -l 154192999801.*
-rwxrwx--- 1 andrew andrew   181 nov.  11 14:03 154192999801.exv
-rwxrwx--- 1 andrew andrew 61249 nov.  11 14:02 154192999801.jpg

andrew@VESA-ubuntu:/nfs/all/USB1Tb$ exiv2 -g DateTime in -f 154192999801.JPG
andrew@VESA-ubuntu:/nfs/all/USB1Tb$ exiv2 -g DateTime -pt 154192999801.JPG
Exif.Image.DateTime                          Ascii      20  2018:11:11 10:53:18
Exif.Photo.DateTimeOriginal                  Ascii      20  2018:11:11 10:53:18

The difference from my failed workflow is that the image is recorded directly to the CIFs/samba from the Raspberrypi. I dare not switchback to confirm this since the hedgehog are busy. But once everything calms down and if you need it I can check.

AB
PS . Thanks again for your work.

I suspect this has something to do with memory-mapping the file.

Exiv2 uses BasicIo to perform I/O and has sub-classes such as FileIo , MemIo, HttpIo and others to use images on different media. On the desktop, exiv2(.exe) usually uses memory-mapping and perhaps that's not valid for CIFS mounted files.

I'll look at this next week as releasing RC2 is the priority for this week. I'm not sure how to set up CIFS from Linux as I've never given that matter any thought. I'll send you some code to implement a sub-set of the utility od which I'll implement with/without memory mapping to validate/confirm/deny my hunch.

Andrew

We don't want to disturb the hedgehogs. Gosh no. And I'm in Wales to run in the Mountains. And apparently this issue has been present for years. So, this isn't urgent. Interesting and curious. If we need a code change, it'd be great to get it into exiv2 v0.27 RC3 (scheduled 2018-12-07).

I'm curious about the relationship with the rasperry π

I thought your issue is that the exiv2 command line program will not read (nor write) on the nas storage connected to Linux by a USB cable and mounted using CIFS.

So why discuss the π ? Are you saying that if you mount the nas using NFS, exiv2 works? Or am I confused?

Thanks Robin, no rush for sure. Sorry for confusion.

The camera was back recording last night, the hedgehogs are close to shutdown anyhow, despite global warming..... which is part of this study but don't let me start on that.....

So the RaspberryPi is the IR camera, I just gave example of command line to demo the 'bug' in actual workflow which is:-

1. Raspberrypi (Zero W in field with IR cam) captures 2 stream of jpgs, the low-res has DateTime stamp.
2. Until last week the stream was saved over WiFi using NFSv4 to another Raspberrypi3 server.
3. At dawn the Raspberrypi3 runs FFmpeg and exiv2 on the jpgs to create timestamped video of action for research/archive.

This has worked well for 2 years or more. Then due to router upgrade, we decided to the router USB3 NAS to store the jpg stream thinking it would be easier/larger etc. But we lost exiv2 functions.

So, yes if we mount the NAS (USB HDD) using NFS exiv2 works as expected.
My manual attempt to run exiv2 on 11th also seemed to work via CIF storage also seems to work, which is confusing.

I'll double check everything and report back if you think it might help you, but on my side I'll stick with original Linux based NFS4 EXT4 storage.

Andrew

Thanks for the clarification. So, my task is to un-puzzle exiv2/CIFS refusing to open files. How odd! I'll investigate next week. When I get home from Wales, I want to focus my attention on Exiv2 v0.27 RC2 and to release that this week.

Robin

Andrew and Gilles

I haven't had time to look into this issue this week. However it isn't forgotten. Luis and I are both working hard to get the Exiv2 v0.27 GM by 28 December 2018. We hoped to arrive at zero open issues for v0.27 today. I suspect this CIFS issue has been in the code for years and doesn't impact every Samba mount. If I can't investigate this next week, I'll open an issue on GitHub to be sure it's never forgotten.

Robin

I'm pleased to say that Team Exiv2 is on track to release Exiv2 v0.27 on 28 December 2018 (as scheduled).

I've spend an afternoon investigating this puzzle. I can't get Samba to work between my Linux VMs. Very frustrating. They can see each other's shares with $ smbclient -L machine-name. Why can't they mount each other's shares? I don't know.

I can mount a Windows share from the Mac. Windows 10 is mounted over SMB. (the w7 name is historical, it's a up-to-date Windows 10).

513 rmills@rmillsmm:~ $ df
Filesystem                                                 512-blocks       Used  Available Capacity  iused               ifree %iused  Mounted on
...
//rmills@rmillsmm-w7._smb._tcp.local/%5BC%5D%20rmillsmm-w7  226025464  196924712   29100752    88% 24615587             3637594   87%   /Volumes/[C] rmillsmm-w7
514 rmills@rmillsmm:~ $ exiv2 -pa --grep datetime/i /Volumes/\[C\]\ rmillsmm-w7/Users/rmills/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54 <--- 15:38
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
515 rmills@rmillsmm:~ $ exiv2 -a -00:01:00 /Volumes/\[C\]\ rmillsmm-w7/Users/rmills/Stonehenge.jpg 
516 rmills@rmillsmm:~ $ exiv2 -pa --grep datetime/i /Volumes/\[C\]\ rmillsmm-w7/Users/rmills/Stonehenge.jpg 
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:24:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:37:54 <--- 15:37
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:37:54
518 rmills@rmillsmm:~ $ 

I ran the command in the debugger. Exiv2 uses a FileIo object to perform the I/O. This is a BasiIo wrapper on FILE*. He also used a FileIo object when I ran/debugged a copy of Stonehenge.jpg locally on the Mac.

I know the code can use Memory Mapped File (and MemIo object). Perhaps he only uses that for tiff files.

I have a suggestion. When you build Exiv2, it generates a file exv_conf.h which is generated by the build system. exv_conf.h is tricky because he's a generated file and it depends how you build (CMake/autotools/msvc) how he get created. I can offer help with that if you're in doubt about the setting.

For version v0.27, Gilles will be happy to hear, we have our CMake support in very good shape (thanks to outstanding work by Luis) and will on support CMake in future. You can build as follows:

$ <exiv2dir>
$ mkdir build ; cd build ; 
$ cmake ..
...... shake rattle and roll .....
$ grep MM exv_conf.h 
#define EXV_HAVE_MMAP
#define EXV_HAVE_SYS_MMAN_H
$ bbedit "exv_conf.h"             # EDIT and comment off EXV_HAVE_MMAP
$ make
...... shake rattle and roll .....
$ bin/exiv2 -vVg mm   # --verbose --version --grep mm
exiv2 0.27.0.3
library=/usr/lib/system/libcommonCrypto.dylib
have_mmap=0   <----------- Correct mmap isn't in the build
have_sys_mman=1
xmlns=xmpMM:http://ns.adobe.com/xap/1.0/mm/
$ make test
$ sudo make install

I'm now stuck. Gilles has produced evidence that CIFS/Exiv2 can "stall". However without being able to reproduce this, it's hard for me to make progress.

I'll be happy to meet with you one-to-one on Skype and walk the builds with you. You (or one of your co-workers) may be offer to help me with the CIFS networking.

I strongly encourage you to use the pre-release of Exiv2 v0.27 http://exiv2.dyndns.org I hope (and expect) the only change from RC3 to GM will be the version number. No code/test/documentation changes.

A good night's sleep, a few reboots, some googling and I've made progress with Samba. I've put the following at the bottom of /etc/samba/smb.conf

[sambashare]
comment=https://tutorials.ubuntu.com/tutorial/install-and-configure-samba
path=/home/rmills/sambashare
read only=no
browsable=yes
writeable = yes
public = yes
create mask = 0644
directory mask = 0755
force user = rmills

I also modified workgroup = RMILLSMM in /etc/samba/smb.conf

And I added a samba user from the terminal:

$ sudo smbpasswd -a rmills

I've done this on rmillsmm-ubuntu (Ubuntu 18.04 64 bit) and rmillsmm-ubuntu32 (Ubuntu 14.04 32 bit) and Mac, Windows, Linux can read and write on the samba shares.

I've stepped the code in the CLion debugger on Ubuntu 18.04. It's using a FileIo object, so it's performing I/O with FILE*. So, that brings my investigation to a dead-end. I can't think of anything else that I can try in my office. This puzzle has to reproduced on Andrew's system.

Andrew, what else can you tell me about your platform and version of exiv2? Did you build exiv2 yourself? Can you download exiv2 v0.27 RC3 from http://exiv2.dyndns.org and try with that, please? You should be able to do that without disturbing your production system.

Gilles: I applied for a free open-source license for CLion. JetBrains gave us three one-year licenses for all their products (Robin, Luis and Dan). I'm delighted with CLion. I've used it on Linux and Mac and it works really well. Very good integration with CMake (and cppcheck and clang-tidy). I haven't tried it on Windows yet. It's great to have the same tools on every platform and, for the same reason, I like SmartGit and SmartSVN. I'm confident of Exiv2 v0.27 GM on 28 December. I know Luis wants to submit some changes inspired by cppcheck. I'd prefer to modify the revision number in 0.27 RC3 and release that. Luis can deal with cppcheck in the v0.28 project.

Thanks for your efforts and latest release Robin.

As you suggested I've built v0.27 on Raspbian Stretch and Ubuntu 18.04 for testing and conclude never work with animal or samba!

I just couldn't get exiv2 to process CIF mounted share images on Raspbian, I've tried all mounting options I know and my guess this is not a exiv2 specific issue but rather Raspbian CIF mount:-

*
pi@raspberrypiy:~/picamera$ exiv2 /mnt/USB1Tb/RPiY-Pictures/image19.jpg
/mnt/USB1Tb/RPiY-Pictures/image19.jpg: Failed to open the file

pi@raspberrypiy:~/picamera$ stat -L /mnt/USB1Tb/RPiY-Pictures/image19.jpg
File: /mnt/USB1Tb/RPiY-Pictures/image19.jpg
Size: 540256 Blocks: 2048 IO Block: 16384 regular file
Device: 13h/19d Inode: 158940790805 Links: 1
Access: (0775/-rwxrwxr-x) Uid: ( 1000/ pi) Gid: ( 1000/ pi)
Access: 2019-01-08 11:53:50.427742000 +0100
Modify: 2019-01-08 11:53:50.427742000 +0100
Change: 2019-01-08 11:53:50.427742000 +0100 *

yet this works on the local folder:

pi@raspberrypiy:~/picamera$ exiv2 -pv -g DateTime testpix/image19.jpg
0x0132 Image DateTime Ascii 20 2019:01:08 11:22:40
0x9003 Photo DateTimeOriginal Ascii 20 2019:01:08 11:22:40
0x9004 Photo DateTimeDigitized Ascii 20 2019:01:08 11:22:40

BUT on my Ubuntu 18.04 it all works as required (metadata extracted and inserted across CIF files):-

andrew@VESA-ubuntu:~$ exiv2 -pv --grep DateTime /nfs/all/USB1Tb/RPiY-Pictures/image1.jpg
andrew@VESA-ubuntu:~$ exiv2 -pv --grep DateTime /nfs/all/USB1Tb/RPiY-Pictures/image1.jpeg
0x0132 Image DateTime Ascii 20 2019:01:08 11:53:51
0x9003 Photo DateTimeOriginal Ascii 20 2019:01:08 11:53:51
0x9004 Photo DateTimeDigitized Ascii 20 2019:01:08 11:53:51
andrew@VESA-ubuntu:~$ exiv2 ex DateTime /nfs/all/USB1Tb/RPiY-Pictures/image1.jpeg
DateTime: Failed to open the file
andrew@VESA-ubuntu:~$ exiv2 ex /nfs/all/USB1Tb/RPiY-Pictures/image1.jpeg
exiv2: Overwrite `/nfs/all/USB1Tb/RPiY-Pictures/image1.exv'? y
andrew@VESA-ubuntu:~$ exiv2 in /nfs/all/USB1Tb/RPiY-Pictures/image1.jpg
andrew@VESA-ubuntu:~$ exiv2 -pv --grep DateTime /nfs/all/USB1Tb/RPiY-Pictures/image1.jpg
0x0132 Image DateTime Ascii 20 2019:01:08 11:53:51
0x9003 Photo DateTimeOriginal Ascii 20 2019:01:08 11:53:51
0x9004 Photo DateTimeDigitized Ascii 20 2019:01:08 11:53:51

Robin I'm at the limit of my knowledge how to test further on raspbian and I leave it to you to priorities as required.

In the meantime for the spring return of the hogs, I've re-written the picamera code to collect metadata time stamps etc on the fly to no more fiddling with CIFs

Andrew

Well some things are destined to remain a mystery.

By the way, exiv2 supports network I/O. The focus of that is http, however we also support SSH and FTP. So if you can have a webserver on the Raspery machine, you should be able to read the metadata via http. Just like this:

586 rmills@rmillsmbp:~ $ exiv2 -pa --grep datetime/i http://clanmills.com/Stonehenge.jpg
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
587 rmills@rmillsmbp:~ $ 

I'm pleased that Exiv2 v0.27 was released on 20 December 2018. We plan "dot" releases (quarterly security and minor fixes) of v0.27 over the next couple of years. Exiv2 v0.28 involves substantial rewrites and "modernisation" to C++11. I'll be 68 next week and look forward to retiring from Exiv2 after v0.28 is completed.

Thanks for the update and best wishes to you for 2019.