Bug #1189

Updating time changes Makernote

Added by Stan Kaminski about 1 year ago. Updated about 1 year ago.

Status:ClosedStart date:29 May 2016
Priority:NormalDue date:
Assignee:Robin Mills% Done:

100%

Category:makernoteEstimated time:6.00 hours
Target version:0.26

Description

I have a bunch of Nikon D90 photos (jpg+nef) that I need to update timezone for (actually, both timezone and date, since, you know...)

I used command:
exiv2 -a -07 adjust test.nef

Then I compared original tag with the updated one - unfortunately, Makernotes were changes as well as time/date tags!

Attaching extracted metadata files.

I compared the files manually, it seems that the last 28 bytes of original Makernote were truncated.

original.exif - Original metadata (435 KB) Stan Kaminski, 29 May 2016 20:22

adjusted.exif - Adjusted metadata (435 KB) Stan Kaminski, 29 May 2016 20:22


Related issues

Related to Exiv2 - Feature #992: Better raw file support and test Assigned 18 Sep 2014

Associated revisions

Revision 4316
Added by Robin Mills about 1 year ago

#1189 See issue report for discussion.

Revision 4321
Added by Robin Mills about 1 year ago

#1189 Use in-memory file to dump Nikon MakerNote tag 0x927c

Revision 4350
Added by Robin Mills about 1 year ago

#1189 Code change provided by private email from Taras. Thank You Taras.

History

#1 Updated by Robin Mills about 1 year ago

  • Category set to makernote
  • Status changed from New to Assigned
  • Assignee set to Robin Mills
  • Target version set to 0.26
  • % Done changed from 0 to 10
  • Estimated time set to 3.00

What version of exiv2 are you using?

$ exiv2 -vV   # verbose version
You're going to have to help me to get onto "the same page" as you. I have a D5300. I've run the following commands:
$ curl -O --silent http://clanmills.com/files/DSC_0001.NEF
$ exiv2 -pa DSC_0001.NEF | sort > before.txt
$ exiv2 -a -07 DSC_0001.NEF
$ exiv2 -pa DSC_0001.NEF | sort > after.txt
$ diff before.txt after.txt
The output seems fine to me! Sure the makernote has been rewritten and is a slightly different length. The location of the image has moved and so on. However the metadata seems preserved and OK (evidence below):
665 rmills@rmillsmbp:~/temp/foo $ diff before.txt after.txt 
4c4
< Exif.Image.DateTime                          Ascii      20  2016:04:20 08:00:13
---
> Exif.Image.DateTime                          Ascii      20  2016:04:20 01:00:13
6,7c6,7
< Exif.Image.ExifTag                           Long        1  2544
< Exif.Image.GPSTag                            Long        1  177664
---
> Exif.Image.ExifTag                           Long        1  3328
> Exif.Image.GPSTag                            Long        1  177846
22,23c22,23
< Exif.Image.StripOffsets                      Long        1  178176
< Exif.Image.SubIFDs                           Long        3  236032 236152 236380
---
> Exif.Image.StripOffsets                      Long        1  177864
> Exif.Image.SubIFDs                           Long        3  2866 2984 3210
25c25
< Exif.Image.XMLPacket                         Byte      2048  (Binary value suppressed)
---
> Exif.Image.XMLPacket                         Byte      2378  (Binary value suppressed)
29c29
< Exif.MakerNote.Offset                        Long        1  3088
---
> Exif.MakerNote.Offset                        Long        1  3810
55c55
< Exif.Nikon3.Preview                          Long        1  23766
---
> Exif.Nikon3.Preview                          Long        1  23740
122c122
< Exif.NikonPreview.JPEGInterchangeFormat      Long        1  24038
---
> Exif.NikonPreview.JPEGInterchangeFormat      Long        1  23846
145,146c145,146
< Exif.Photo.DateTimeDigitized                 Ascii      20  2016:04:20 08:00:13
< Exif.Photo.DateTimeOriginal                  Ascii      20  2016:04:20 08:00:13
---
> Exif.Photo.DateTimeDigitized                 Ascii      20  2016:04:20 01:00:13
> Exif.Photo.DateTimeOriginal                  Ascii      20  2016:04:20 01:00:13
160c160
< Exif.Photo.MakerNote                         Undefined 174576  (Binary value suppressed)
---
> Exif.Photo.MakerNote                         Undefined 173976  (Binary value suppressed)
176c176
< Exif.SubImage1.JPEGInterchangeFormat         Long        1  1239552
---
> Exif.SubImage1.JPEGInterchangeFormat         Long        1  235464
197c197
< Exif.SubImage2.StripOffsets                  Long        1  4171776
---
> Exif.SubImage2.StripOffsets                  Long        1  3167648
201c201
< Exif.SubImage3.JPEGInterchangeFormat         Long        1  236544
---
> Exif.SubImage3.JPEGInterchangeFormat         Long        1  21326300
666 rmills@rmillsmbp:~/temp/foo $ 
The maker note is decoded into tags Exif.NikonN.XXXX. There are 112 of them and they seem to have been transferred safely.
Exif.NikonFi.FileNumber                      Short       1  1
669 rmills@rmillsmbp:~/temp/foo $ exiv2 -pa --grep Nikon/i ~/Pictures/2016/Raw/DSC_0001.NEF > orig.txt
670 rmills@rmillsmbp:~/temp/foo $ exiv2 -pa --grep Nikon/i DSC_0001.NEF > adju.txt
671 rmills@rmillsmbp:~/temp/foo $ diff orig.txt adju.txt 
12c12
< Exif.Nikon3.Preview                          Long        1  23766
---
> Exif.Nikon3.Preview                          Long        1  23740
17c17
< Exif.NikonPreview.JPEGInterchangeFormat      Long        1  24038
---
> Exif.NikonPreview.JPEGInterchangeFormat      Long        1  23846
672 rmills@rmillsmbp:~/temp/foo $ wc orig.txt adju.txt 
     112     550    7688 orig.txt
     112     550    7688 adju.txt
     224    1100   15376 total
673 rmills@rmillsmbp:~/temp/foo $ 
It appears that Exiv2 has "shortened" the preview. My hunch is that Exiv2 enforces the rule that there should be no metadata in a preview. You can extract the previews on the original and adjusted files and inspect the extracted previews for metadata. I haven't bothered to do this because I'm on vacation at the moment!!

#2 Updated by Robin Mills about 1 year ago

  • % Done changed from 10 to 80

#3 Updated by Stan Kaminski about 1 year ago

Version is current Arch for ARMv5tel:

$ exiv2 -vV
exiv2 0.25 001900 (32 bit build)
Copyright (C) 2004-2015 Andreas Huggel.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public
License along with this program; if not, write to the Free
Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA
exiv2=0.25.0
platform=linux
compiler=G++
bits=32
dll=1
debug=0
version=5.3.0
date=Dec 11 2015
time=11:41:06
svn=0
ssh=0
curl=0
id=$Id: version.cpp 3800 2015-05-08 22:26:36Z robinwmills $
executable=/usr/bin/exiv2
library=/usr/lib/libstdc++.so.6
library=/usr/lib/libgcc_s.so.1
library=/usr/lib/libc.so.6
library=/lib/ld-linux.so.3
library=/usr/lib/libz.so.1
library=/usr/lib/libexpat.so.1
library=/usr/lib/libdl.so.2
library=/usr/lib/libm.so.6
have_regex=1
have_strerror_r=1
have_gmtime_r=1
have_inttypes=1
have_libintl=1
have_lensdata=1
have_iconv=1
have_memory=1
have_memset=1
have_lstat=1
have_stdbool=1
have_stdint=1
have_stdlib=1
have_strlib=0
have_strchr=1
have_strerror=1
have_strerror_r=1
have_strings_h=0
have_strtol=1
have_mmap=1
have_munmap=1
have_sys_stat=1
have_timegm=1
have_unistd_h=0
have_sys_mman=1
have_libz=0
have_xmptoolkit=1
have_bool=1
have_strings=1
have_sys_types=1
have_unistd=1
have_unicode_path=0
enable_video=1
enable_webready=0

It seems I missed that exiv2 decodes the "binary blob" as proper tags? I thought that exiv2 says "binary blob" because it's unable to decode proprietary "blobs", I'm happy that I was mistaken.

Can we somehow check what metadata is lost? But you are, of course, right - matter of a quick script to restore this.

Just a quick question before I file another unnecessary bug: exiv2 does not modify "Exif.Image.DateTimeOriginal" on purpose, so I'll have to update it manually, right?

PS. I'm very impressed with your speed - thanks for taking care of such a useful tool!

#4 Updated by Robin Mills about 1 year ago

  • % Done changed from 80 to 100

In all honesty, I don't believe any metadata is being lost. However, I invite you to prove me wrong!

In general, exiv2 will allow you to modify almost any item of metadata. However, it is not a metadata policeman. It doesn't enforce the standard. It gives you enough power to hang yourself. There are some items of metadata that cannot be changed. For example, you can't change the image sizes. However tags such as Exif.Image.DateTimeOriginal are no protected.

747 rmills@rmillsmbp:~/temp/foo $ exiv2 -pa --grep Dimen http://clanmills.com/Stonehenge.jpg
Exif.Photo.PixelXDimension                   Short       1  6000
Exif.Photo.PixelYDimension                   Short       1  4000

I believe when Andreas started writing exiv2, it was simply intended to modify some dates and options such as -t appeared at that time. So, -t might update more than one tag at a time. In general this approach is no longer part of our philosophy, although -t continues to be supported.

So, to answer your question, "Yes, you should update Exif.Image.DateTimeOriginal manually". Easy to sync this in bash:

783 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ file=Stonehenge.jpg
784 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ cp ~/$file .
785 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ DateTime=$(exiv2 -PEv -K Exif.Image.DateTime $file) 
786 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ echo $DateTime
2015:07:16 20:25:28
787 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ exiv2 -pa --grep DateTime $file
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
788 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ exiv2 -M"set Exif.Photo.DateTimeOriginal $DateTime" $file
789 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ exiv2 -pa --grep DateTime $file
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
790 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ 

#5 Updated by Robin Mills about 1 year ago

  • Status changed from Assigned to Closed
  • Estimated time changed from 3.00 to 2.00

Stan

I'm going to mark this "Closed". You are welcome to update this issue and I will respond. I will re-open the issue if we discover further work has to be done.

Robin

#6 Updated by Stan Kaminski about 1 year ago

Hi Robin,
I'm afraid this either necessitates more clarification, or it's something to be fixed...

I tried doing Makernote transplant between files, but it still loses about 100 bytes. Sequence:

$ exiv2 ad -a +7 test.nef
$ exiv2  -b -Pkv test2.nef -g Exif.Photo.MakerNote > origin.exif
$ exiv2  -b -Pkv test.nef -g Exif.Photo.MakerNote > timezone.exif
$ ls -la

-rw-r--r-- 1 stan stan   428344 May 31 12:30 origin.exif
-rw-r--r-- 1 stan stan 12913969 May 29 21:25 test2.nef
-rw-r--r-- 1 stan stan  6568533 May 29 22:09 test.jpg
-rw-r--r-- 1 stan stan 12913969 May 31 12:30 test.nef
-rw-r--r-- 1 stan stan   428243 May 31 12:30 timezone.exif

Interesting (perhaps?) note: sizes of test.nef and test2.nef match, but size of extracted metadata differs.
$ cat origin.exif | sed -E -e "s/^/set /" > foo.cmd
$ exiv2 -mfoo.cmd test.nef
$ exiv2  -b -Pkv test.nef -g Exif.Photo.MakerNote > timezone2.exif
$ ls -la

-rw-r--r-- 1 stan stan   428348 May 31 12:30 foo.cmd
-rw-r--r-- 1 stan stan   428344 May 31 12:30 origin.exif
-rw-r--r-- 1 stan stan 12913969 May 29 21:25 test2.nef
-rw-r--r-- 1 stan stan  6568533 May 29 22:09 test.jpg
-rw-r--r-- 1 stan stan 12913969 May 31 12:30 test.nef
-rw-r--r-- 1 stan stan   428243 May 31 12:30 timezone2.exif
-rw-r--r-- 1 stan stan   428243 May 31 12:30 timezone.exif

And here you can see that the size of "timezone.exif" and "timezone2.exif" is the same, and different from the size of "origin.exif" (which I tried to import).

Based on what you stated above, does exiv2 parser silently strips some data there? Can we do something about it? (import Makernote 1:1)

#7 Updated by Robin Mills about 1 year ago

I didn't write Exiv2, so I can't say off-hand exactly why we rewrite the file with a slightly different makernote lengths.

More important is to see if metadata has been lost in the rewrite. You'll see in the note#1 above, I compared orig.txt and adju.txt http://dev.exiv2.org/issues/1189#note-1

Can you repeat that investigation on your files. You can also extract the previews and compare them (pixel by pixel) before and after. You'll need solid evidence to convince me that there is something wrong here. Just saying "the lengths are different" isn't evidence that the data has been been corrupted. When the bytes of the graphics are written to file, the offsets will be different and that can/could easily explain the difference.

I'm on vacation, so my time is limited at present.

#8 Updated by Robin Mills about 1 year ago

I've had another little look at this using my file DSC_0001.NEF

Previews (Thumbnails)

The preview images have not been disturbed by the -t option. For sure the physical location of the preview/subImages has changed, however their content has not.

698 rmills@rmillsmbp:~/temp $ del thumbs *preview*
699 rmills@rmillsmbp:~/temp $ file=DSC_0001.NEF
700 rmills@rmillsmbp:~/temp $ cp ~/Pictures/2016/Raw/$file .
701 rmills@rmillsmbp:~/temp $ exiv2 -pa $file | sort > orig.txt
702 rmills@rmillsmbp:~/temp $ exiv2 --verbose -ep $file
File 1/1: DSC_0001.NEF
Writing preview 1 (image/tiff, 160x120 pixels, 57852 bytes) to file ./DSC_0001-preview1.tif
Writing preview 2 (image/jpeg, 640x424 pixels, 150119 bytes) to file ./DSC_0001-preview2.jpg
Writing preview 3 (image/jpeg, 1620x1080 pixels, 1002500 bytes) to file ./DSC_0001-preview3.jpg
Writing preview 4 (image/jpeg, 6000x4000 pixels, 2932184 bytes) to file ./DSC_0001-preview4.jpg
703 rmills@rmillsmbp:~/temp $ mkdir thumbs
704 rmills@rmillsmbp:~/temp $ mv *preview* thumbs
705 rmills@rmillsmbp:~/temp $ exiv2 -a -07 $file
706 rmills@rmillsmbp:~/temp $ exiv2 -pa $file | sort > adju.txt
707 rmills@rmillsmbp:~/temp $ exiv2 --verbose -ep $file
File 1/1: DSC_0001.NEF
Writing preview 1 (image/tiff, 160x120 pixels, 57852 bytes) to file ./DSC_0001-preview1.tif
Writing preview 2 (image/jpeg, 640x424 pixels, 150119 bytes) to file ./DSC_0001-preview2.jpg
Writing preview 3 (image/jpeg, 1620x1080 pixels, 1002500 bytes) to file ./DSC_0001-preview3.jpg
Writing preview 4 (image/jpeg, 6000x4000 pixels, 2932184 bytes) to file ./DSC_0001-preview4.jpg
708 rmills@rmillsmbp:~/temp $ ls -lt thumbs/*
-rw-r--r--+ 1 rmills  staff    57852 31 May 21:08 thumbs/DSC_0001-preview1.tif
-rw-r--r--+ 1 rmills  staff   150119 31 May 21:08 thumbs/DSC_0001-preview2.jpg
-rw-r--r--+ 1 rmills  staff  1002500 31 May 21:08 thumbs/DSC_0001-preview3.jpg
-rw-r--r--+ 1 rmills  staff  2932184 31 May 21:08 thumbs/DSC_0001-preview4.jpg
709 rmills@rmillsmbp:~/temp $ ls -lt *preview*
-rw-r--r--+ 1 rmills  staff    57852 31 May 21:10 DSC_0001-preview1.tif
-rw-r--r--+ 1 rmills  staff   150119 31 May 21:10 DSC_0001-preview2.jpg
-rw-r--r--+ 1 rmills  staff  1002500 31 May 21:10 DSC_0001-preview3.jpg
-rw-r--r--+ 1 rmills  staff  2932184 31 May 21:10 DSC_0001-preview4.jpg
710 rmills@rmillsmbp:~/temp $ diff orig.txt adju.txt
4c4
< Exif.Image.DateTime                          Ascii      20  2016:04:20 08:00:13
---
> Exif.Image.DateTime                          Ascii      20  2016:04:20 01:00:13
6,7c6,7
< Exif.Image.ExifTag                           Long        1  2544
< Exif.Image.GPSTag                            Long        1  177664
---
> Exif.Image.ExifTag                           Long        1  3328
> Exif.Image.GPSTag                            Long        1  177846
22,23c22,23
< Exif.Image.StripOffsets                      Long        1  178176
< Exif.Image.SubIFDs                           Long        3  236032 236152 236380
---
> Exif.Image.StripOffsets                      Long        1  177864
> Exif.Image.SubIFDs                           Long        3  2866 2984 3210
25c25
< Exif.Image.XMLPacket                         Byte      2048  (Binary value suppressed)
---
> Exif.Image.XMLPacket                         Byte      2378  (Binary value suppressed)
29c29
< Exif.MakerNote.Offset                        Long        1  3088
---
> Exif.MakerNote.Offset                        Long        1  3810
55c55
< Exif.Nikon3.Preview                          Long        1  23766
---
> Exif.Nikon3.Preview                          Long        1  23740
122c122
< Exif.NikonPreview.JPEGInterchangeFormat      Long        1  24038
---
> Exif.NikonPreview.JPEGInterchangeFormat      Long        1  23846
145,146c145,146
< Exif.Photo.DateTimeDigitized                 Ascii      20  2016:04:20 08:00:13
< Exif.Photo.DateTimeOriginal                  Ascii      20  2016:04:20 08:00:13
---
> Exif.Photo.DateTimeDigitized                 Ascii      20  2016:04:20 01:00:13
> Exif.Photo.DateTimeOriginal                  Ascii      20  2016:04:20 01:00:13
160c160
< Exif.Photo.MakerNote                         Undefined 174576  (Binary value suppressed)
---
> Exif.Photo.MakerNote                         Undefined 173976  (Binary value suppressed)
176c176
< Exif.SubImage1.JPEGInterchangeFormat         Long        1  1239552
---
> Exif.SubImage1.JPEGInterchangeFormat         Long        1  235464
197c197
< Exif.SubImage2.StripOffsets                  Long        1  4171776
---
> Exif.SubImage2.StripOffsets                  Long        1  3167648
201c201
< Exif.SubImage3.JPEGInterchangeFormat         Long        1  236544
---
> Exif.SubImage3.JPEGInterchangeFormat         Long        1  21326300
711 rmills@rmillsmbp:~/temp $ 

200 Keys are unchanged. No keys lost

734 rmills@rmillsmbp:~/temp $ wc orig.txt adju.txt
     208     991   14386 orig.txt
     208     991   14381 adju.txt
     416    1982   28767 total
735 rmills@rmillsmbp:~/temp $ 

XMLPacket

The tag Exif.Image.XMLPacket has a different length before and after the use of -t. However, the XML (xmp) is identical. It's formatted differently for reasons I have not investigated.

720 rmills@rmillsmbp:~/temp $ orig=orig.nef
721 rmills@rmillsmbp:~/temp $ adju=adju.nef
722 rmills@rmillsmbp:~/temp $ file=DSC_0001.NEF
723 rmills@rmillsmbp:~/temp $ cp ~/Pictures/2016/Raw/$file $orig
724 rmills@rmillsmbp:~/temp $ cp ~/Pictures/2016/Raw/$file $adju
725 rmills@rmillsmbp:~/temp $ exiv2 -a -07 $adju
726 rmills@rmillsmbp:~/temp $ exiv2 -pa --grep XMLPacket $orig $adju
orig.nef              Exif.Image.XMLPacket                         Byte      2048  (Binary value suppressed)
adju.nef              Exif.Image.XMLPacket                         Byte      2378  (Binary value suppressed)
727 rmills@rmillsmbp:~/temp $ exiv2 -pX $orig | xmllint --pretty 1 - > orig.xmp
728 rmills@rmillsmbp:~/temp $ exiv2 -pX $adju | xmllint --pretty 1 - > adju.xmp
729 rmills@rmillsmbp:~/temp $ ls -alt *.xmp
-rw-r--r--+ 1 rmills  staff   350 31 May 21:23 adju.xmp
-rw-r--r--+ 1 rmills  staff   359 31 May 21:22 orig.xmp
-rw-r--r--+ 1 rmills  staff  3867 18 Jan 07:32 ttt.xmp
-rw-r--r--+ 1 rmills  staff  3867 18 Jan 07:32 trunk.xmp
-rw-r--r--+ 1 rmills  staff  3332  8 Jan 11:15 exv.xmp
-rw-r--r--+ 1 rmills  staff  4854  8 Jan 11:09 eX.xmp
-rw-r--r--+ 1 rmills  staff  4178  8 Jan 10:56 raw.xmp
730 rmills@rmillsmbp:~/temp $ del ttt.xmp trunk.xmp exv.xmp eX.xmp raw.xmp 
731 rmills@rmillsmbp:~/temp $ cat adju.xmp
<?xml version="1.0"?>
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 4.4.0-Exiv2">
  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:Description xmlns:xmp="http://ns.adobe.com/xap/1.0/" rdf:about="" xmp:Rating="0"/>
  </rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>
732 rmills@rmillsmbp:~/temp $ 

Exif.Image.Exif

Don't know what this is. Changed? Yes. Why? Don't know.


< Exif.Image.ExifTag                           Long        1  2544
---
> Exif.Image.ExifTag                           Long        1  3328 

Exif.Image.GPS

Don't know what this is. Changed? Yes. Why? Don't know.


< Exif.Image.GPSTag                            Long        1  177664
---
> Exif.Image.GPSTag                            Long        1  177846

Right. I'm in Largs in Scotland. I was born here. I haven't been "home" for 3 years. The weather is perfect. About 24ºC today and not a cloud in the sky.

#9 Updated by Robin Mills about 1 year ago

Stan

Another beautiful sunny morning in Scotland. I've taken a careful look at your note #6 above: http://dev.exiv2.org/issues/1189#note-6

I don't know that copying a makernote from one file to another is too wise, however I believe it will work. I'm a little lost about the difference between test.nef and test2.nef. Perhaps you can use Dropbox or something to send me the files for investigation.

#10 Updated by Stan Kaminski about 1 year ago

You know, this issue is probably not important enough to warrant looking at the screen instead of looking at this incredible vista :-)

I've been doing some digging myself, but it's taking me time to get up-to-speed with how EXIF information works.

I did some tests with exiftool, just to compare results, and man, this one changes even more stuff. At least I could confirm that the preview has not changed at all. So back to exiv2 :-)

I found that in the Makernote, the 100-byte difference is just 28 characters (28 bytes), in my case: "71 14 168 208 168 96 49 156 66 149 217 195 50 159 182 148 37 210 59 252 209 131 130 82 185 255 217 0". If that'd be all 0's then I don't care, but clearly here is some information that is lost. Original size is 120624 bytes, after modification it's 120597 bytes.

Now the interesting part is that apart from losing 28 bytes at the end, the rest is identical between versions. Is it possible that exiv2 does not know the tags at the end and quietly deletes them? Looking at the code, it should report even unknown tags (that's also something I checked, there's no difference in unknown tags between versions).

Also, there's no xmp data in my NEFs, exiv2 even throws relevant warning.

When I do "adjust" it actually changes only times, does nothing to Exif.Image.GPSTag and Exif.Image.ExifTag.

Here's the photo I'm doing my tests on (12 MB): https://www.dropbox.com/s/whpww59cc1kjkns/2016-05-20%2008.28.54.nef?dl=0 - let me know when you get it, so I can remove it from public folder.

#11 Updated by Stan Kaminski about 1 year ago

Can't edit previous update: I just wanted to state the obvious, it looks like exiv2 is prematurely finishing writing Makernote. Either that, or it's reading too much? (I mean, dump = read -> too long, or update = write -> too short).

#12 Updated by Robin Mills about 1 year ago

  • Status changed from Closed to Assigned
  • % Done changed from 100 to 50
  • Estimated time changed from 2.00 to 4.00

Thanks, Stan. I've got you file.

I don't know what's in those 28 bytes. Will have to step the code in the debugger to find out what's going on.

You might be interested to see the output from exiv2 -pS (v0.25) and even more interested to see exiv2 -pR (trunk). Here's what I see in your file:

506 rmills@rmillsmbp:~ $ exiv2 -pR ~/Downloads/2016-05-20\ 08.28.54.nef 
STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
 address |    tag                           |      type |    count |    offset | value
      10 | 0x00fe NewSubfileType            |      LONG |        1 |         1 | 1
      22 | 0x0100 ImageWidth                |      LONG |        1 |       160 | 160
      34 | 0x0101 ImageLength               |      LONG |        1 |       120 | 120
      46 | 0x0102 BitsPerSample             |     SHORT |        3 |       316 | 8 8 8
      58 | 0x0103 Compression               |     SHORT |        1 |     65536 | 1
      70 | 0x0106 PhotometricInterpretation |     SHORT |        1 |    131072 | 2
      82 | 0x010f Make                      |     ASCII |       18 |       324 | NIKON CORPORATION
      94 | 0x0110 Model                     |     ASCII |       10 |       344 | NIKON D90
     106 | 0x0111 StripOffsets              |      LONG |        1 |    121656 | 121656
     118 | 0x0112 Orientation               |     SHORT |        1 |    524288 | 8
     130 | 0x0115 SamplesPerPixel           |     SHORT |        1 |    196608 | 3
     142 | 0x0116 RowsPerStrip              |      LONG |        1 |       120 | 120
     154 | 0x0117 StripByteCounts           |      LONG |        1 |     57600 | 57600
     166 | 0x011a XResolution               |  RATIONAL |        1 |       356 | 356/0
     178 | 0x011b YResolution               |  RATIONAL |        1 |       364 | 364/0
     190 | 0x011c PlanarConfiguration       |     SHORT |        1 |     65536 | 1
     202 | 0x0128 ResolutionUnit            |     SHORT |        1 |    131072 | 2
     214 | 0x0131 Software                  |     ASCII |       10 |       372 | Ver.1.00 
     226 | 0x0132 DateTime                  |     ASCII |       20 |       384 | 2016:05:20 08:28:54
     238 | 0x014a SubIFDs                   |      LONG |        2 |       404 | 179256 179376
  STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
   address |    tag                           |      type |    count |    offset | value
    179258 | 0x00fe NewSubfileType            |      LONG |        1 |         1 | 1
    179270 | 0x0103 Compression               |     SHORT |        1 |    393216 | 6
    179282 | 0x011a XResolution               |  RATIONAL |        1 |    179360 | 48288/2
    179294 | 0x011b YResolution               |  RATIONAL |        1 |    179368 | 48296/2
    179306 | 0x0128 ResolutionUnit            |     SHORT |        1 |    131072 | 2
    179318 | 0x0201 JPEGInterchangeFormat     |      LONG |        1 |    179648 | 179648
    179330 | 0x0202 JPEGInterchangeFormatLeng |      LONG |        1 |   1547046 | 1547046
    179342 | 0x0213 YCbCrPositioning          |     SHORT |        1 |    131072 | 2
  END /Users/rmills/Downloads/2016-05-20 08.28.54.nef
     250 | 0x0214 ReferenceBlackWhite       |  RATIONAL |        6 |       412 | 0/0 1/0 255/0 1/0 0/0 ...
     262 | 0x8769 ExifTag                   |      LONG |        1 |       480 | 480
  STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
   address |    tag                           |      type |    count |    offset | value
       482 | 0x829a ExposureTime              |  RATIONAL |        1 |       872 | 872/0
       494 | 0x829d FNumber                   |  RATIONAL |        1 |       880 | 880/0
       506 | 0x8822 ExposureProgram           |     SHORT |        1 |         0 | 0
       518 | 0x8827 ISOSpeedRatings           |     SHORT |        1 |  18350080 | 280
       530 | 0x9003 DateTimeOriginal          |     ASCII |       20 |       888 | 2016:05:20 08:28:54
       542 | 0x9004 DateTimeDigitized         |     ASCII |       20 |       908 | 2016:05:20 08:28:54
       554 | 0x9204 ExposureBiasValue         | SRATIONAL |        1 |       928 | 928/0
       566 | 0x9205 MaxApertureValue          |  RATIONAL |        1 |       936 | 936/0
       578 | 0x9207 MeteringMode              |     SHORT |        1 |    327680 | 5
       590 | 0x9208 LightSource               |     SHORT |        1 |         0 | 0
       602 | 0x9209 Flash                     |     SHORT |        1 |   1572864 | 24
       614 | 0x920a FocalLength               |  RATIONAL |        1 |       944 | 944/0
       626 | 0x927c MakerNote                 | UNDEFINED |   120624 |      1012 | Nikon.....MM.*.....6........0210 ...
       638 | 0x9286 UserComment               | UNDEFINED |       44 |       952 | ASCII...                         ...
       650 | 0x9290 SubSecTime                |     ASCII |        3 | 808452096 | 00
       662 | 0x9291 SubSecTimeOriginal        |     ASCII |        3 | 808452096 | 00
       674 | 0x9292 SubSecTimeDigitized       |     ASCII |        3 | 808452096 | 00
       686 | 0xa217 SensingMethod             |     SHORT |        1 |    131072 | 2
       698 | 0xa300 FileSource                | UNDEFINED |        1 |  50331648 | .
       710 | 0xa301 SceneType                 | UNDEFINED |        1 |  16777216 | .
       722 | 0xa302 CFAPattern                | UNDEFINED |        8 |       996 | ........
       734 | 0xa401 CustomRendered            |     SHORT |        1 |         0 | 0
       746 | 0xa402 ExposureMode              |     SHORT |        1 |         0 | 0
       758 | 0xa403 WhiteBalance              |     SHORT |        1 |         0 | 0
       770 | 0xa404 DigitalZoomRatio          |  RATIONAL |        1 |      1004 | 1004/0
       782 | 0xa405 FocalLengthIn35mmFilm     |     SHORT |        1 |   2424832 | 37
       794 | 0xa406 SceneCaptureType          |     SHORT |        1 |         0 | 0
       806 | 0xa407 GainControl               |     SHORT |        1 |         0 | 0
       818 | 0xa408 Contrast                  |     SHORT |        1 |         0 | 0
       830 | 0xa409 Saturation                |     SHORT |        1 |         0 | 0
       842 | 0xa40a Sharpness                 |     SHORT |        1 |         0 | 0
       854 | 0xa40c SubjectDistanceRange      |     SHORT |        1 |         0 | 0
  END /Users/rmills/Downloads/2016-05-20 08.28.54.nef
     274 | 0x8825 GPSTag                    |      LONG |        1 |    121636 | 121636
     286 | 0x9003 DateTimeOriginal          |     ASCII |       20 |       460 | 2016:05:20 08:28:54
     298 | 0x9216 TIFFEPStandardID          |      BYTE |        4 |  16777216 | ...
END /Users/rmills/Downloads/2016-05-20 08.28.54.nef
507 rmills@rmillsmbp:~ $ 
The interesting tag is:
       626 | 0x927c MakerNote                 | UNDEFINED |   120624 |      1012 | Nikon.....MM.*.....6........0210 ...
Clearly the MakerNote is a TIFF. I thought I had code to recursively dump that tag, maybe not. There's fun ahead with this. But not today. The sun's shining and I'm "home" for the first time in 3 years.

#13 Updated by Robin Mills about 1 year ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

r4316. I've updated -pR to recursively dump the TIFF that's embedded in the MakerNote. I don't really believe the output. This will require more investigation which we'll factor into #992 (which is the top priority item for v0.27). I have no more time available to spend on this if I'm going to make the schedule to be code-complete on v0.26 in June 2016.

#14 Updated by Robin Mills about 1 year ago

  • Estimated time changed from 4.00 to 6.00

I've spent another couple of hours on this. I've used an in-memory file (using the bytes in the 0x927c tag) to perform the recursive dump. The output of -pR now seems sensible.

STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
 address |    tag                           |      type |    count |    offset | value
      10 | 0x00fe NewSubfileType            |      LONG |        1 |         1 | 1
      22 | 0x0100 ImageWidth                |      LONG |        1 |       160 | 160
      34 | 0x0101 ImageLength               |      LONG |        1 |       120 | 120
      46 | 0x0102 BitsPerSample             |     SHORT |        3 |       316 | 8 8 8
      58 | 0x0103 Compression               |     SHORT |        1 |     65536 | 1
      70 | 0x0106 PhotometricInterpretation |     SHORT |        1 |    131072 | 2
      82 | 0x010f Make                      |     ASCII |       18 |       324 | NIKON CORPORATION
      94 | 0x0110 Model                     |     ASCII |       10 |       344 | NIKON D90
     106 | 0x0111 StripOffsets              |      LONG |        1 |    121656 | 121656
     118 | 0x0112 Orientation               |     SHORT |        1 |    524288 | 8
     130 | 0x0115 SamplesPerPixel           |     SHORT |        1 |    196608 | 3
     142 | 0x0116 RowsPerStrip              |      LONG |        1 |       120 | 120
     154 | 0x0117 StripByteCounts           |      LONG |        1 |     57600 | 57600
     166 | 0x011a XResolution               |  RATIONAL |        1 |       356 | 356/0
     178 | 0x011b YResolution               |  RATIONAL |        1 |       364 | 364/0
     190 | 0x011c PlanarConfiguration       |     SHORT |        1 |     65536 | 1
     202 | 0x0128 ResolutionUnit            |     SHORT |        1 |    131072 | 2
     214 | 0x0131 Software                  |     ASCII |       10 |       372 | Ver.1.00 
     226 | 0x0132 DateTime                  |     ASCII |       20 |       384 | 2016:05:20 08:28:54
     238 | 0x014a SubIFDs                   |      LONG |        2 |       404 | 179256 179376
  STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
   address |    tag                           |      type |    count |    offset | value
    179258 | 0x00fe NewSubfileType            |      LONG |        1 |         1 | 1
    179270 | 0x0103 Compression               |     SHORT |        1 |    393216 | 6
    179282 | 0x011a XResolution               |  RATIONAL |        1 |    179360 | 48288/2
    179294 | 0x011b YResolution               |  RATIONAL |        1 |    179368 | 48296/2
    179306 | 0x0128 ResolutionUnit            |     SHORT |        1 |    131072 | 2
    179318 | 0x0201 JPEGInterchangeFormat     |      LONG |        1 |    179648 | 179648
    179330 | 0x0202 JPEGInterchangeFormatLeng |      LONG |        1 |   1547046 | 1547046
    179342 | 0x0213 YCbCrPositioning          |     SHORT |        1 |    131072 | 2
  END /Users/rmills/Downloads/2016-05-20 08.28.54.nef
     250 | 0x0214 ReferenceBlackWhite       |  RATIONAL |        6 |       412 | 0/0 1/0 255/0 1/0 0/0 ...
     262 | 0x8769 ExifTag                   |      LONG |        1 |       480 | 480
  STRUCTURE OF TIFF FILE (MM): /Users/rmills/Downloads/2016-05-20 08.28.54.nef
   address |    tag                           |      type |    count |    offset | value
       482 | 0x829a ExposureTime              |  RATIONAL |        1 |       872 | 872/0
       494 | 0x829d FNumber                   |  RATIONAL |        1 |       880 | 880/0
       506 | 0x8822 ExposureProgram           |     SHORT |        1 |         0 | 0
       518 | 0x8827 ISOSpeedRatings           |     SHORT |        1 |  18350080 | 280
       530 | 0x9003 DateTimeOriginal          |     ASCII |       20 |       888 | 2016:05:20 08:28:54
       542 | 0x9004 DateTimeDigitized         |     ASCII |       20 |       908 | 2016:05:20 08:28:54
       554 | 0x9204 ExposureBiasValue         | SRATIONAL |        1 |       928 | 928/0
       566 | 0x9205 MaxApertureValue          |  RATIONAL |        1 |       936 | 936/0
       578 | 0x9207 MeteringMode              |     SHORT |        1 |    327680 | 5
       590 | 0x9208 LightSource               |     SHORT |        1 |         0 | 0
       602 | 0x9209 Flash                     |     SHORT |        1 |   1572864 | 24
       614 | 0x920a FocalLength               |  RATIONAL |        1 |       944 | 944/0
       626 | 0x927c MakerNote                 | UNDEFINED |   120624 |      1012 | Nikon.....MM.*.....6........0210 ...
    STRUCTURE OF TIFF FILE (MM): MemIo
     address |    tag                           |      type |    count |    offset | value
          10 | 0x0001 Version                   | UNDEFINED |        4 | 808595760 | 0210
          22 | 0x0002 ISOSpeed                  |     SHORT |        2 |       280 | 0 280
          34 | 0x0004 Quality                   |     ASCII |        8 |       662 | RAW    
          46 | 0x0005 WhiteBalance              |     ASCII |       13 |       670 | AUTO        
          58 | 0x0007 Focus                     |     ASCII |        7 |       686 | AF-A  
          70 | 0x0008 FlashSetting              |     ASCII |       13 |       694 | NORMAL      
          82 | 0x0009 GPSStatus                 |     ASCII |       20 |       710 |                    
          94 | 0x000b ProcessingSoftware        |    SSHORT |        2 |         0 | 0 0
         106 | 0x000c GPSSpeedRef               |  RATIONAL |        4 |       730 | 459/0 256/0 371/0 256/0
         118 | 0x000d GPSSpeed                  | UNDEFINED |        4 |     67072 | ...
         130 | 0x000e GPSTrackRef               | UNDEFINED |        4 |     68608 | ...
         142 | 0x0011 GPSImgDirection           |      LONG |        1 |     13954 | 13954
         154 | 0x0012 GPSMapDatum               | UNDEFINED |        4 |     67072 | ...
         166 | 0x0013 GPSDestLatitudeRef        |     SHORT |        2 |       280 | 0 280
         178 | 0x0017 GPSDestBearingRef         | UNDEFINED |        4 |     67072 | ...
         190 | 0x0018 GPSDestBearing            | UNDEFINED |        4 |     67072 | ...
         202 | 0x0019 GPSDestDistanceRef        | SRATIONAL |        1 |       762 | 762/0
         214 | 0x001b GPSProcessingMethod       |     SHORT |        7 |       770 | 0 4352 2868 4352 2868 ...
         226 | 0x001c GPSAreaInformation        | UNDEFINED |        3 |     67072 | ...
         238 | 0x001d GPSDateStamp              |     ASCII |        8 |       786 | 7034901
         250 | 0x001e GPSDifferential           |     SHORT |        1 |    131072 | 2
         262 | 0x001f                           | UNDEFINED |        8 |       794 | 0100...
         274 | 0x0022                           |     SHORT |        1 |4294901760 | 65535
         286 | 0x0023                           | UNDEFINED |       58 |       802 | 0100STANDARD............STANDARD ...
         298 | 0x0024                           | UNDEFINED |        4 |   3932160 | .<.
         310 | 0x0025                           | UNDEFINED |       14 |       862 | N.....N.......
         322 | 0x002c                           | UNDEFINED |       94 |       878 | 0100...d....................... ...
         334 | 0x0083                           |      BYTE |        1 | 234881024 | .
         346 | 0x0084                           |  RATIONAL |        4 |       974 | 180/0 10/0 1050/0 10/0
         358 | 0x0087                           |      BYTE |        1 |         0 | 
         370 | 0x0089                           |     SHORT |        1 |         0 | 0
         382 | 0x008a                           |     SHORT |        1 |    131072 | 2
         394 | 0x008b                           | UNDEFINED |        4 |1073810432 | @..
         406 | 0x008c                           | UNDEFINED |      578 |      1006 | I0............................. ...
         418 | 0x0091                           | UNDEFINED |     6600 |      1586 | 0213..........y"..+.....2.W2..*. ...
         430 | 0x0093                           |     SHORT |        1 |    262144 | 4
         442 | 0x0095                           |     ASCII |        5 |      8186 | OFF 
         454 | 0x0096                           | UNDEFINED |      624 |      8194 | D .H.H.H.H....... .0.@.P.`.p.... ...
         466 | 0x0097                           | UNDEFINED |     1302 |      8818 | 0211........................... ...
         478 | 0x0098                           | UNDEFINED |       33 |     10122 | 02041...E..^GPH..........W10o.k ...
         490 | 0x0099                           |     SHORT |        2 | 142607770 | 2176 1434
         502 | 0x009e                           |     SHORT |       10 |     10158 | 0 0 0 0 0 ...
         514 | 0x00a3                           |      BYTE |        1 |         0 | 
         526 | 0x00a4                           | UNDEFINED |        4 | 808595504 | 0200
         538 | 0x00a7                           |      LONG |        1 |      4736 | 4736
         550 | 0x00a8                           | UNDEFINED |       22 |     10178 | 0103.................
         562 | 0x00ab                           |     ASCII |       16 |     10202 | AUTO           
         574 | 0x00b0                           | UNDEFINED |       16 |     10218 | 0100...........
         586 | 0x00b1                           |     SHORT |        1 |         0 | 0
         598 | 0x00b6                           | UNDEFINED |        8 |     10234 | .......
         610 | 0x00b7                           | UNDEFINED |       30 |     10242 | 0100.........................
         622 | 0x00b8                           | UNDEFINED |      172 |     10274 | 0100...d....................... ...
         634 | 0x00bb                           | UNDEFINED |        6 |     10446 | 0100.
         646 | 0x00bc                           | UNDEFINED |     3500 |     10454 | 0100.D."...........P..p..@...A.. ...
    END MemIo
       638 | 0x9286 UserComment               | UNDEFINED |       44 |       952 | ASCII...                         ...
       650 | 0x9290 SubSecTime                |     ASCII |        3 | 808452096 | 00
       662 | 0x9291 SubSecTimeOriginal        |     ASCII |        3 | 808452096 | 00
       674 | 0x9292 SubSecTimeDigitized       |     ASCII |        3 | 808452096 | 00
       686 | 0xa217 SensingMethod             |     SHORT |        1 |    131072 | 2
       698 | 0xa300 FileSource                | UNDEFINED |        1 |  50331648 | .
       710 | 0xa301 SceneType                 | UNDEFINED |        1 |  16777216 | .
       722 | 0xa302 CFAPattern                | UNDEFINED |        8 |       996 | ........
       734 | 0xa401 CustomRendered            |     SHORT |        1 |         0 | 0
       746 | 0xa402 ExposureMode              |     SHORT |        1 |         0 | 0
       758 | 0xa403 WhiteBalance              |     SHORT |        1 |         0 | 0
       770 | 0xa404 DigitalZoomRatio          |  RATIONAL |        1 |      1004 | 1004/0
       782 | 0xa405 FocalLengthIn35mmFilm     |     SHORT |        1 |   2424832 | 37
       794 | 0xa406 SceneCaptureType          |     SHORT |        1 |         0 | 0
       806 | 0xa407 GainControl               |     SHORT |        1 |         0 | 0
       818 | 0xa408 Contrast                  |     SHORT |        1 |         0 | 0
       830 | 0xa409 Saturation                |     SHORT |        1 |         0 | 0
       842 | 0xa40a Sharpness                 |     SHORT |        1 |         0 | 0
       854 | 0xa40c SubjectDistanceRange      |     SHORT |        1 |         0 | 0
  END /Users/rmills/Downloads/2016-05-20 08.28.54.nef
     274 | 0x8825 GPSTag                    |      LONG |        1 |    121636 | 121636
     286 | 0x9003 DateTimeOriginal          |     ASCII |       20 |       460 | 2016:05:20 08:28:54
     298 | 0x9216 TIFFEPStandardID          |      BYTE |        4 |  16777216 | ...
END /Users/rmills/Downloads/2016-05-20 08.28.54.nef

Many of the tags are not being identified by name. For example 0x0084 is Lens:

689 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ grep -H 0084 src/*.cpp
src/nikonmn.cpp:        TagInfo(0x0084, "Lens", N_("Lens"), N_("Lens"), nikon3Id, makerTags, unsignedRational, -1, print0x0084),
src/nikonmn.cpp:    std::ostream& Nikon3MakerNote::print0x0084(std::ostream& os,
src/quicktimevideo.cpp:        {    0x2000084, "Xmp.video.LensModel" },
690 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ exiv2 -pR ~/Downloads/2016-05-20\ 08.28.54.nef | grep 0084
         346 | 0x0084                           |  RATIONAL |        4 |       974 | 180/0 10/0 1050/0 10/0
691 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ exiv2 -pv --grep Lens$ ~/Downloads/2016-05-20\ 08.28.54.nef 
0x0084 Nikon3       Lens                        Rational    4  180/10 1050/10 35/10 56/10
692 rmills@rmillsmbp:~/gnu/exiv2/bigtiff $ 
I don't have time to investigate further in v0.26.

Also available in: Atom PDF

Redmine Appliance - Powered by TurnKey Linux