Project

General

Profile

Bug #1253

After setting LensModel, Lightroom no longer recognizes image

Added by Martin Stolle about 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Robin Mills
Category:
makernote
Target version:
0.26
Start date:
13 Nov 2016
Due date:
% Done:

100%

Estimated time:
20.00 h

Description

Hi,

I have a manual lens and am using exiv to set the LensModel afterwards:

exiv2 -M "set Exif.Photo.FocalLength Rational 75/10" -M "set Exif.Photo.LensModel Ascii Samyang 7.5mm F/3.5 UMC Fish-eye MFT"  mo "$image"

While geeqie and exiftool show the new lens information, Lightroom refuses to import such images. Looking at a diff of the tags, there is a suspicious change of size of MakerNoteOlympus2 and of datatypes for a couple of IFD sections.

I have attached before and after raw files.

Files

20161113-114050_pb132546.orf (13.5 MB) 20161113-114050_pb132546.orf before modifying metadata Martin Stolle, 13 Nov 2016 15:07
20161113-114050_pb132546_tagged.orf (13.2 MB) 20161113-114050_pb132546_tagged.orf after modifying metadata Martin Stolle, 13 Nov 2016 15:07
20161113-114050_pb132546_exiftool.orf (13.2 MB) 20161113-114050_pb132546_exiftool.orf Martin Stolle, 13 Nov 2016 17:58

Related issues

Related to Exiv2 - Bug #1255: ORF MakerNote incorrectly re-writtenNew17 Nov 2016

Actions

History

#1

Updated by Martin Stolle about 5 years ago

version of exiv2 used: exiv2 0.25 001900 (64 bit build) (Ubuntu 16.04)

#2

Updated by Robin Mills about 5 years ago

  • Category set to makernote
  • Status changed from New to Assigned
  • Assignee set to Robin Mills
  • Target version set to 0.26

This sounds painful for you and for me. Thanks for attaching the images. A couple of questions:

1) Which version of LR are you using?
2) Have you been able to successfully assign the Lens with any other metadata tool such as ExifTool or Adobe Bridge?

I'll look at this later today. It's nice sunny afternoon in England and I'm dealing with the leaves in the garden at the moment.

#3

Updated by Martin Stolle about 5 years ago

I was able to add the same information with the following exiftool command:

exiftool -FocalLength=75/10 -LensModel="Samyang 7.5mm F/3.5 UMC Fish-eye MFT" 20161113-114050_pb132546_exiftool.orf

This was importable (with recognizable lens model) into Lightroom.

Lightroom is the latest Lightroom CC (2015.7 Release, Camera Raw 9.7)

Surprisingly, darktable still didn't recognize the lens, but I guess that's a darktable problem.

#4

Updated by Martin Stolle about 5 years ago

Just for completeness, this is darktable 2.0.3 (from Ubuntu 16.04)

#5

Updated by Robin Mills about 5 years ago

I've reproduced the LR import issue with LR5.0/Camera Raw 8.1 (2013 Edition of LR)

Some success, and some unhappiness! I'm not bothered about the IFD observation. The file was rewritten and some of the data has been relocated.

Happyness:

The original file has 32 bytes allocate for LensModel. 

525 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546.orf | grep -i Maker
       460 | 0x927c MakerNote                 | UNDEFINED |  1447744 |      3264 | OLYMPUS.II........\....@....... ...
526 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546_exiftool.orf | grep -i Maker
      1258 | 0x927c MakerNote                 | UNDEFINED |  1212494 |      1546 | OLYMPUS.II........\...r........ ...
527 rmills@rmillsmm:~/Downloads $

Keep the new lens name to less than 32 bytes and it will import:

  511  18:02:14 cp 20161113-114050_pb132546.orf modify.orf
  513  18:03:06 exiv2 -M "set Exif.Photo.LensModel Ascii Samyang 7.5mm lens" modify.orf

Inspecting the "raw" metadata with -pR (print Recursive), your exiftool file has achieved almost an identical result to exiv2: 

516 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546_exiftool.orf | grep -i lens
      1438 | 0xa432 LensSpecification         |  RATIONAL |        4 |   1214182 | 0/0 0/0 0/0 0/0
      1450 | 0xa434 LensModel                 |     ASCII |       37 |   1214214 | Samyang 7.5mm F/3.5 UMC Fish-eye ...
517 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546_tagged.orf | grep -i lens
      1438 | 0xa432 LensSpecification         |  RATIONAL |        4 |   1214176 | 0/0 0/0 0/0 0/0
      1450 | 0xa434 LensModel                 |     ASCII |       37 |   1214208 | Samyang 7.5mm F/3.5 UMC Fish-eye ...
518 rmills@rmillsmm:~/Downloads $

Unhappiness:

LensMode greater than 32 bytes will not import.

There's a substantial difference in the MakerNote:

525 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546.orf | grep -i Maker
       460 | 0x927c MakerNote                 | UNDEFINED |  1447744 |      3264 | OLYMPUS.II........\....@....... ...
526 rmills@rmillsmm:~/Downloads $ exiv2 -pR 20161113-114050_pb132546_exiftool.orf | grep -i Maker
      1258 | 0x927c MakerNote                 | UNDEFINED |  1212494 |      1546 | OLYMPUS.II........\...r........ ...
527 rmills@rmillsmm:~/Downloads $

So what's wrong?

I think the MakerNote has been corrupted by Exiv2 when LensModel is greater than 32 bytes

What can you and I do about this?

I don't have time at the moment to dig through all the makernote encoder/decoder for Olympus Cameras as I'm about to ship v0.26.

Can you ask Adobe to give you an explanation for why Camera Raw dislikes the _tagged.orf file. Don't give them a hint about my suspicion of our MakerNote encoder. If you tell them that, they'll instantly say that's the fault. Ask them to check their logs to determine the reason they can't import it.

#6

Updated by Robin Mills about 5 years ago

  • Status changed from Assigned to New
  • Target version changed from 0.26 to 0.28
#7

Updated by Robin Mills about 5 years ago

  • Status changed from New to Assigned
  • Target version changed from 0.28 to 0.26
  • % Done changed from 0 to 40
  • Estimated time set to 10.00 h

I've download the latest DNGConvertor for Mac (9.7.0). I've set the (1) Select the images to convert: Select Folder... to my ~/Downloads. Everything converts except tagged.orf who is totally ignored. I've monitored this operation with Console.app and filtered with "Adobe". Lots of messages appearing (including assertions) from the process "Adobe DNG Convertor". Sadly, none of them mean anything much to me (copy below).

Curiously the messages all involve dancing with Apple Software and AppKit in particular. I've tried to open our friend _tagged.orf with Preview.app and got this:

PVImageContainer initWithURL:file:///Users/rmills/Downloads/20161113-114050_pb132546_tagged.orf failed, error = Error Domain=NSCocoaErrorDomain Code=259 "The file “20161113-114050_pb132546_tagged.orf” could not be opened." UserInfo={NSURL=file:///Users/rmills/Downloads/20161113-114050_pb132546_tagged.orf, NSLocalizedDescription=The file “20161113-114050_pb132546_tagged.orf” could not be opened., NSLocalizedRecoverySuggestion=Preview currently does not support this raw file format.}
We're getting somewhere. The Apple Raw Image code hates the _tagged.orf file. However, we don't know why!

You told me you're working on Ubuntu, so none of the Apple code is implicated. However there's little doubt that _tagged.orf is a sorry file. And LightRoom dislikes the file as well. This puzzle shows up in LightRoom, however it is the file _tagged.orf that is not good.

For the moment, there is a work-around. You'll have to take care to restrict your lens name to the original 32 bytes allocated in the original file. Clearly we can rewrite that safely, however when we write a name longer that 32 bytes, the file has been damaged in some way.

I've inspected the decoded MakerNote - especially the Exif.OlympusEq.XXXX tags. All are unchanged between your good.orf and tagged.orf. So suspicion now lands on the tag _Exif.Photo.LensModel. I don't believe any tag in Exif.Photo.XXX could damage a file because that family of tags occurs in every image with Exif metadata (almost every single image). However, unlike the MakerNote decoder/encoder, it will be easy to step that code with the debugger. It's late now (23:30 in England). I'll have another look at this tomorrow.

default    22:50:55.839812 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:57.298552 +0000    Adobe DNG Converter    <private> is not MNT_LOCAL!
default    22:50:57.298596 +0000    Adobe DNG Converter    <private>
default    22:50:57.299870 +0000    Adobe DNG Converter    /Network/Servers is not MNT_LOCAL!
default    22:50:57.299910 +0000    Adobe DNG Converter    <private>
default    22:50:57.300584 +0000    Adobe DNG Converter    <private> is not MNT_LOCAL!
default    22:50:57.300619 +0000    Adobe DNG Converter    <private>
default    22:50:57.300926 +0000    Adobe DNG Converter    discovered extensions
default    22:50:57.301116 +0000    Adobe DNG Converter    <private> is not MNT_LOCAL!
default    22:50:57.301155 +0000    Adobe DNG Converter    <private>
default    22:50:57.301797 +0000    Adobe DNG Converter    got a connection for: com.apple.cache_delete
default    22:50:57.301950 +0000    Adobe DNG Converter    got proxy: <private> for service: com.apple.cache_delete
default    22:50:57.303094 +0000    Adobe DNG Converter    setting target uid to: 501 for service: com.apple.cache_delete
default    22:50:57.303125 +0000    Adobe DNG Converter    connection invalidated
default    22:50:57.542175 +0000    Adobe DNG Converter    calling plugIn beginUsing:
default    22:50:57.569760 +0000    Adobe DNG Converter    SFBrowserCallBack (node = <SFNode 0x6180002e0480>{domain = Network})
default    22:50:57.644040 +0000    Adobe DNG Converter    completed calling plugIn beginUsing: for pid: 67369
default    22:50:59.693317 +0000    Adobe DNG Converter    Connection to sharingd became invalid
default    22:50:59.742727 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.743086 +0000    Adobe DNG Converter    errors encountered while discovering extensions
default    22:50:59.747423 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.747562 +0000    Adobe DNG Converter    tearing down extension request for pid 67369
default    22:50:59.747655 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.747817 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.747965 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.748093 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.751308 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.894974 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:50:59.988915 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.094786 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.234902 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.235245 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.235390 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.235537 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.235616 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.245576 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.245987 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.294794 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.495149 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.651280 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.694530 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:00.895385 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.094548 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.295332 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.494741 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.691023 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.694602 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:01.840220 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:04.823697 +0000    Adobe DNG Converter    discovered extensions
default    22:51:05.852762 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:25.677271 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:51:25.990201 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
default    22:52:57.652568 +0000    Adobe DNG Converter    *** Assertion failure in -[NSEvent subtype], /Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1504.60/AppKit.subproj/NSEvent.m:3014
#8

Updated by Robin Mills about 5 years ago

  • % Done changed from 40 to 50

I'm making steady progress with this. There's no question TiffParserWorker::encode is unconditionally taking the wmIntrusive branch. This is fine when it's overwriting the 32 bytes already allocated for LensModel. It's not OK then it must be totally rewritten (wmNonIntrusive). On my local copy, I am unconditionally setting wmIntrusive, this "sort of" works in that the file is not corrupted. However, LensModel is not updated either and retains the 32 bytes (which I think are all nulls).

I hardly know this part of the code, so I will have to read/step/think. I expect/hope to fix this today.

#9

Updated by Robin Mills about 5 years ago

  • % Done changed from 50 to 40
  • Estimated time changed from 10.00 h to 20.00 h

I'm giving up on this for today. Here's what I know.
1) tiffimage.cpp # 1934
The following lines seem in the wrong order:

           parsedTree->accept(encoder);
            if (!encoder.dirty()) writeMethod = wmNonIntrusive;
And should be:
           if (!encoder.dirty()) writeMethod = wmNonIntrusive;
            parsedTree->accept(encoder);
parsedTree->accept() captures encoder.exifdata_. So encoder.dirty() is never true (because it has lost its metadata). So the code always takes the Intrusive branch which works when the new value of LensModel is less than the allocated 32 bytes in the file.

2) I still don't know why Preview (and LR and others) reject the file. I have investigated the possibility that we have forgotten to write the 4byte ZERO at the end of an IFD. That is working well (and many other matters I have considered). 

$ cp ~/Downloads/pb.orf . ; build/bin/Debug/exiv2 -M'set Exif.Photo.LensModel 12345678901234567890123456789012345' pb.orf ; exiv2 -pR pb.orf
STRUCTURE OF TIFF FILE (II): pb.orf
 address |    tag                           |      type |    count |    offset | value
      10 | 0x0100 ImageWidth                |      LONG |        1 |      4640 | 4640
      22 | 0x0101 ImageLength               |      LONG |        1 |      3472 | 3472
      34 | 0x0102 BitsPerSample             |     SHORT |        1 |        16 | 16
      46 | 0x0103 Compression               |     SHORT |        1 |         1 | 1
      58 | 0x0106 PhotometricInterpretation |     SHORT |        1 |         1 | 1
      70 | 0x010e ImageDescription          |     ASCII |       32 |       290 | OLYMPUS DIGITAL CAMERA         
      82 | 0x010f Make                      |     ASCII |       24 |       322 | OLYMPUS IMAGING CORP.  
      94 | 0x0110 Model                     |     ASCII |       17 |       346 | E-M5            
     106 | 0x0111 StripOffsets              |      LONG |        1 |   1214244 | 1214244
     118 | 0x0112 Orientation               |     SHORT |        1 |         1 | 1
     130 | 0x0115 SamplesPerPixel           |     SHORT |        1 |         1 | 1
     142 | 0x0116 RowsPerStrip              |      LONG |        1 |      3472 | 3472
     154 | 0x0117 StripByteCounts           |      LONG |        1 |  12656269 | 12656269
     166 | 0x011a XResolution               |  RATIONAL |        1 |       364 | 350/1
     178 | 0x011b YResolution               |  RATIONAL |        1 |       372 | 350/1
     190 | 0x011c PlanarConfiguration       |     SHORT |        1 |         1 | 1
     202 | 0x0128 ResolutionUnit            |     SHORT |        1 |         2 | 2
     214 | 0x0131 Software                  |     ASCII |       32 |       380 | Version 2.2                    
     226 | 0x0132 DateTime                  |     ASCII |       20 |       412 | 2016:11:13 10:40:50
     238 | 0x013b Artist                    |     ASCII |       64 |       432 | Martin Stolle                    ...
     250 | 0x8298 Copyright                 |     ASCII |       64 |       496 | Martin Stolle                    ...
     262 | 0x8769 ExifTag                   |      LONG |        1 |      1088 | 1088
  STRUCTURE OF TIFF FILE (II): pb.orf
   address |    tag                           |      type |    count |    offset | value
      1090 | 0x829a ExposureTime              |  RATIONAL |        1 |      1466 | 1/100
      1102 | 0x829d FNumber                   |  RATIONAL |        1 |      1474 | 0/10
      1114 | 0x8822 ExposureProgram           |     SHORT |        1 |         2 | 2
      1126 | 0x8827 ISOSpeedRatings           |     SHORT |        1 |       640 | 640
      1138 | 0x8830 SensitivityType           |     SHORT |        1 |         1 | 1
      1150 | 0x9000 ExifVersion               | UNDEFINED |        4 | 808661552 | 0230
      1162 | 0x9003 DateTimeOriginal          |     ASCII |       20 |      1482 | 2016:11:13 10:40:50
      1174 | 0x9004 DateTimeDigitized         |     ASCII |       20 |      1502 | 2016:11:13 10:40:50
      1186 | 0x9204 ExposureBiasValue         | SRATIONAL |        1 |      1522 | 0/10
      1198 | 0x9205 MaxApertureValue          |  RATIONAL |        1 |      1530 | 0/256
      1210 | 0x9207 MeteringMode              |     SHORT |        1 |         5 | 5
      1222 | 0x9208 LightSource               |     SHORT |        1 |         0 | 0
      1234 | 0x9209 Flash                     |     SHORT |        1 |         8 | 8
      1246 | 0x920a FocalLength               |  RATIONAL |        1 |      1538 | 0/10
      1258 | 0x927c MakerNote                 | UNDEFINED |  1212488 |      1546 | OLYMPUS.II........\...r........ ...
      1270 | 0x9286 UserComment               | UNDEFINED |      125 |   1214034 | ........                         ...
      1282 | 0xa000 FlashpixVersion           | UNDEFINED |        4 | 808464688 | 0100
      1294 | 0xa001 ColorSpace                |     SHORT |        1 |         1 | 1
      1306 | 0xa300 FileSource                | UNDEFINED |        1 |         3 | .
      1318 | 0xa302 CFAPattern                | UNDEFINED |        8 |   1214160 | ........
      1330 | 0xa401 CustomRendered            |     SHORT |        1 |         0 | 0
      1342 | 0xa402 ExposureMode              |     SHORT |        1 |         0 | 0
      1354 | 0xa403 WhiteBalance              |     SHORT |        1 |         0 | 0
      1366 | 0xa404 DigitalZoomRatio          |  RATIONAL |        1 |   1214168 | 100/100
      1378 | 0xa406 SceneCaptureType          |     SHORT |        1 |         0 | 0
      1390 | 0xa407 GainControl               |     SHORT |        1 |         2 | 2
      1402 | 0xa408 Contrast                  |     SHORT |        1 |         0 | 0
      1414 | 0xa409 Saturation                |     SHORT |        1 |         0 | 0
      1426 | 0xa40a Sharpness                 |     SHORT |        1 |         1 | 1
      1438 | 0xa432 LensSpecification         |  RATIONAL |        4 |   1214176 | 0/0 0/0 0/0 0/0
      1450 | 0xa434 LensModel                 |     ASCII |       36 |   1214208 | 12345678901234567890123456789012 ...
  END pb.orf
     274 | 0xc4a5 PrintImageMatching        | UNDEFINED |      528 |       560 | PrintIM.0300..%................ ...
END pb.orf
$ dd skip=1088 bs=1 count=2 if=pb.orf 2>/dev/null | od -h -w12 | cut -d' ' -f 2- | head -1
001f
$ dd skip=1090 bs=1 count=400 if=pb.orf 2>/dev/null | od -h -w12 | cut -d' ' -f 2- | head -32
829a 0005 0001 0000 05ba 0000
829d 0005 0001 0000 05c2 0000
8822 0003 0001 0000 0002 0000
8827 0003 0001 0000 0280 0000
8830 0003 0001 0000 0001 0000
9000 0007 0004 0000 3230 3033
9003 0002 0014 0000 05ca 0000
9004 0002 0014 0000 05de 0000
9204 000a 0001 0000 05f2 0000
9205 0005 0001 0000 05fa 0000
9207 0003 0001 0000 0005 0000
9208 0003 0001 0000 0000 0000
9209 0003 0001 0000 0008 0000
920a 0005 0001 0000 0602 0000
927c 0007 8048 0012 060a 0000
9286 0007 007d 0000 8652 0012
a000 0007 0004 0000 3130 3030
a001 0003 0001 0000 0001 0000
a300 0007 0001 0000 0003 0000
a302 0007 0008 0000 86d0 0012
a401 0003 0001 0000 0000 0000
a402 0003 0001 0000 0000 0000
a403 0003 0001 0000 0000 0000
a404 0005 0001 0000 86d8 0012
a406 0003 0001 0000 0000 0000
a407 0003 0001 0000 0002 0000
a408 0003 0001 0000 0000 0000
a409 0003 0001 0000 0000 0000
a40a 0003 0001 0000 0001 0000
a432 0005 0004 0000 86e0 0012
a434 0002 000b 0000 8700 0012
0000 0000 0001 0000 0064 0000
$

I didn't write tiffcomposite.cpp and have never changed a line of code in that file. So, I've spent several hours in the debugger. The code is well written and clear. It has been very stable for years.

Tomorrow is another day.

#10

Updated by Robin Mills about 5 years ago

  • Category changed from makernote to miscellaneous
  • % Done changed from 40 to 80
  • Estimated time changed from 20.00 h to 12.00 h

I have run out of ideas about why LightRoom rejects the _tagged.orf file. Here are things that I have done today:

1) Modified the header of _tagged.orf to be a TIFF (bytes 2 and 3 are short 42. Header: 0x4949002a "II\0*").
libtiff utilities tiffinfo and tiffdump run as expected.
2) I've downloaded LibRaw-0.17.2 and run simple_dcraw. It produces at 47mb tiff image.
3) Picasa and darkroom do not complain about the _tagged.orf (neither use LibRaw)
4) I've analysed the difference in the metadata between the original and modified _tagged files:

$ exiv2 -pa ~/Downloads/pb.orf | sort > pb.txt
$ exiv2 -pa ~/Downloads/ta.orf | sort > ta.txt
$ diff pb.txt ta.txt 
6c6
< Exif.Image.ExifTag                           Long        1  290
---
> Exif.Image.ExifTag                           Long        1  1088 <---- Correct.  Larger ExifTag relocated
21c21
< Exif.Image.StripOffsets                      Long        1  1451008
---
> Exif.Image.StripOffsets                      Long        1  1214246
25c25
< Exif.MakerNote.Offset                        Long        1  3264
---
> Exif.MakerNote.Offset                        Long        1  1546
27,31c27,31
< Exif.Olympus2.CameraSettings                 Ifd         1  420
< Exif.Olympus2.Equipment                      Ifd         1  114
< Exif.Olympus2.FocusInfo                      Ifd         1  2994
< Exif.Olympus2.ImageProcessing                Ifd         1  1344
< Exif.Olympus2.RawDevelopment                 Ifd         1  1170
---
> Exif.Olympus2.CameraSettings                 Long        1  5190
> Exif.Olympus2.Equipment                      Long        1  4602
> Exif.Olympus2.FocusInfo                      Long        1  1207556
> Exif.Olympus2.ImageProcessing                Long        1  1076176
> Exif.Olympus2.RawDevelopment                 Long        1  1075970
75c75
< Exif.OlympusCs.PreviewImageStart             Long        1  48960
---
> Exif.OlympusCs.PreviewImageStart             Long        1  6660
191c191
< Exif.Photo.FocalLength                       Rational    1  0.0 mm
---
> Exif.Photo.FocalLength                       Rational    1  7.5 mm <--- Correct
194c194
< Exif.Photo.LensModel                         Ascii      32  
---
> Exif.Photo.LensModel                         Ascii      37  Samyang 7.5mm F/3.5 UMC Fish-eye MFT <--- Correct
197c197
< Exif.Photo.MakerNote                         Undefined 1447744  (Binary value suppressed)
---
> Exif.Photo.MakerNote                         Undefined 1212488  (Binary value suppressed)
505 rmills@rmillsmbp:~ $
Several observations about this:

1) I acknowledge that Exiv2 has modified this file. Adobe and Apple code refuses to open the modified file.
2) Other image viewers (Picasa, darktable, DCRaw) accept the modified file.
3) The work-around is to restrict Exif.Photo.LensModel to 32 bytes (including the null terminator).
4) I have spent 10+ hours on this matter and examined everything I believe is relevant. I have been unable to find any violation of the TIFF Specification in the _tagged.orf file.
5) There are differences in the files concerning the MakerNote. I doubt if any image viewer decodes the maker note, so although I can't explain the difference, I don't think this explains LightRoom and Preview's behaviour. exiftool is happy to process the file, so the metadata structures in the file are valid.

The only way we can make progress is to get information from an image viewer that refuses to recognise the file (LightRoom, DNG Convertor, or Preview).

I am thinking to close this issue as "Bug in Lightroom". This is a very unsatisfactory conclusion, however I don't know what else I can do.

So, Mr Stolle: Three questions:

1) Can you live with my workaround (restrict the LensModel value to 32 bytes)?
2) Can you contact Adobe and ask why LightRoom rejected the _tagged file and Picasa does not?
3) Can you suggest anything else that you or I can do to progress this issue?

#11

Updated by Martin Stolle about 5 years ago

Hi, thanks for looking into this issue.

To answer your question:

1) I will probably use ExifTool to set the LensModel, as setting it there does not exhibit the import problem.
2) I will try to contact them tomorrow or later this week. They unfortunately only offer live chat support during business hours, when I am not at home. (There is no email/form I could find. If you know of an asynchronous contact mechanism, I'd be more than happy to contact them.)
3) My exiv2 foo is not very strong, but could you try removing the MakerNote? Based on your investigation so far, it seems as if the assumption that Lightroom doesn't care about the contents of MakeNote may be wrong (as Sherlock Holmes once said: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.")

#12

Updated by Robin Mills about 5 years ago

Martin

Can you email me please I want to discuss something privately with you. (nothing sinister, simply something I would prefer not to discuss in public).

Robin

#13

Updated by Robin Mills about 5 years ago

  • % Done changed from 80 to 70
  • Estimated time changed from 12.00 h to 20.00 h

I'm glad you're happy with your work around to use ExifTool. Phil reads the Exiv2 forum and occasionally chips in very helpful comments. I might ask him about this, however there's no reason to disturb him while we're making progress.

There are three groups of applications crashing your file:

1) Adobe applications
2) Apple’s Core Graphics
3) Microsoft Windows/10 Photo app (very nice app incidentally).

The following open-source apps render your file: digiKam, GwenView, darktable, GIMP, RawTherepee, Krita.

I like you suggestion to remove the MakerNote. I’ll do that today. It’ll require some brain surgery in Exiv2 - however I’m confident of success.

Another possibility is that the graphics are incorrect! The following tags identify the location of the pixels: 

518 rmills@rmillsmbp:~/gnu/exiv2/trunk $ build/bin/Debug/exiv2 -pv --grep strip/i pb.orf
0x0111 Image        StripOffsets                Long        1  1451008
0x0116 Image        RowsPerStrip                Long        1  3472
0x0117 Image        StripByteCounts             Long        1  12656269
519 rmills@rmillsmbp:~/gnu/exiv2/trunk $

If there’s something wrong with those tags, a Tiff Renderer could detect a buffer overflow. Or it might not!

It’s something, that’s for certain. We’ll find and fix it. I’m confident.

#14

Updated by Robin Mills about 5 years ago

I've made steady progress with this AND feel very close to solving this puzzle. That file has a rather strange value of StripByteCounts of 12656269. This is an ODD number for Compression=1 BitsPerSample=16 SamplesPerPixel=1 Width=4640 Height=3472. I would expect the frame buffer to be 2*4640*3471 = 32210880 (32mb). The file is 12mb. I think the Olympus people have compressed the frame buffer! However both Exiv2 and exiftool have relocate the frame buffer and the exiftool version does not crash! Both exiv2 and exiftool have the framebuffer 4-byte aligned.

$ exiv2 -pR pb.orf  | grep -i -e strip -i -e image -i -e compres -i -e per
      10 | 0x0100 ImageWidth                |      LONG |        1 |      4640 | 4640
      22 | 0x0101 ImageLength               |      LONG |        1 |      3472 | 3472
      34 | 0x0102 BitsPerSample             |     SHORT |        1 |        16 | 16
      46 | 0x0103 Compression               |     SHORT |        1 |         1 | 1
      70 | 0x010e ImageDescription          |     ASCII |       32 |       290 | OLYMPUS DIGITAL CAMERA         
     106 | 0x0111 StripOffsets              |      LONG |        1 |      1744 | 1744
     130 | 0x0115 SamplesPerPixel           |     SHORT |        1 |         1 | 1
     142 | 0x0116 RowsPerStrip              |      LONG |        1 |      3472 | 3472
     154 | 0x0117 StripByteCounts           |      LONG |        1 |  12656269 | 12656269
      1198 | 0x9205 MaxApertureValue          |  RATIONAL |        1 |      1518 | 0/256
     274 | 0xc4a5 PrintImageMatching        | UNDEFINED |      528 |       560 | PrintIM.0300..%................ ...
$

In your 3 files, there's a significant difference between _exiftool.org and _tagged.orf in StripOffsets! 

$ exiv2 -pa --grep offset/i ~/Downloads/201*orf
20161113-114050_pb132546.orf  Exif.Image.StripOffsets                      Long        1  1451008
20161113-114050_pb132546_exiftool.orf  Exif.Image.StripOffsets                      Long        1  1214252
20161113-114050_pb132546_tagged.orf  Exif.Image.StripOffsets                      Long        1  1214246

And the file sizes are different by 8 bytes. 

$ ls -alt ~/Downloads/201*orf
-rw-r--r--+ 1 rmills staff 13870516 Nov 16 22:05 20161113-114050_pb132546_tagged.orf
-rw-r--r--+ 1 rmills staff 13870522 Nov 16 22:05 20161113-114050_pb132546_exiftool.orf
-rw-r--r--+ 1 rmills staff 14107277 Nov 16 21:37 20161113-114050_pb132546.orf
$

I've been able to nuke the MakerNote. 

628 rmills@rmillsmbp:~/gnu/exiv2/trunk $ svn diff .
Index: src/tiffcomposite.cpp
===================================================================
--- src/tiffcomposite.cpp    (revision 4684)
+++ src/tiffcomposite.cpp    (working copy)
@@ -916,7 +916,7 @@
         visitor.visitDirectory(this);
         for (Components::const_iterator i = components_.begin();
              visitor.go(TiffVisitor::geTraverse) && i != components_.end(); ++i) {
-            (*i)->accept(visitor);
+            if ( (*i)->tag() != 0x927c) (*i)->accept(visitor);
         }
         if (visitor.go(TiffVisitor::geTraverse)) visitor.visitDirectoryNext(this);
         if (pNext_) pNext_->accept(visitor);
$
File is much smaller:
$ ls -alt ~/Downloads/pb.orf 
$ cp ~/Downloads/pb.orf .
$ exiv2 -M'set Exif.Photo.LensModel 12345678901234567890123456789012345' pb.orf
$ ls -alt pb.orf
-rw-r--r--+ 1 rmills staff 14107277 Nov 13 17:16 /Users/rmills/Downloads/pb.orf
-rw-r--r--+ 1 rmills staff 12658014 Nov 16 21:27 pb.orf
$
The file is now so simple that I can easily inspect the two IFDs.

Tomorrow, I will investigate the 8 byte mystery between exiftool and exiv2. Tomorrow I will get to the bottom of this and declare victory!

#15

Updated by Robin Mills about 5 years ago

  • Estimated time changed from 20.00 h to 25.00 h
#16

Updated by Robin Mills about 5 years ago

  • % Done changed from 70 to 80

Sherlock Holmes Strikes Again! When I remove the MakerNote, Preview is very upset.

$ cp ~/Downloads/20161113-114050_pb132546.orf pb.orf
$ ls -alt pb.orf
-rw-r--r--+ 1 rmills staff 14107277 Nov 17 08:38 pb.orf
$ exiftool -LensModel=different pb.orf
    1 image files updated
$ ls -alt pb.orf
-rw-r--r--+ 1 rmills staff 13870466 Nov 17 08:38 pb.orf
$ open pb.orf     # Opens in Preview.app
$ exiftool -makernotes:all= pb.orf
    1 image files updated
$ ls -alt pb.orf
-rw-r--r--+ 1 rmills staff 12657988 Nov 17 08:39 pb.orf
$ open pb.orf      # Preview.app barfs
And I know where to look. exiftool has 4 more bytes in the makernote:
$ cp ~/Downloads/20161113-114050_pb132546.orf pb.orf
$ exiv2 -M'set Exif.Photo.LensModel 12345678901234567890123456789012345' pb.orf
$ exiv2 -pR pb.orf | grep -i maker
      1258 | 0x927c MakerNote                 | UNDEFINED |  1212488 |      1546 | OLYMPUS.II........\...r........ ...
$ 

$ cp ~/Downloads/20161113-114050_pb132546.orf pb.orf
$ exiftool -LensModel=12345678901234567890123456789012345 pb.orf
    1 image files updated
$ exiv2 -pR pb.orf | grep -i maker
      1258 | 0x927c MakerNote                 | UNDEFINED |  1212492 |      1546 | OLYMPUS.II........\...r........ ...
$

So we have now returned to my observation/guess on Sunday afternoon:

I think the MakerNote has been corrupted by Exiv2 when LensModel is greater than 32 bytes

#17

Updated by Robin Mills about 5 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 80 to 100
  • Estimated time changed from 25.00 h to 20.00 h

I'm closing this issue as we have identified the "true" fault for which I have opened a new issue: #1255. Martin and I have identified a couple of work-arounds.

The underlying fault (rewriting the file has corrupted the MakerNote) has never been reported. The code to write the MakerNote was added in r1992 in 2010. I think this has gone un-noticed because most open-source renderers do not decode the MakerNote. However it's clear that Apple, Adobe and Microsoft decode the MakerNote and do no forgive this error.

I've had a look at the code involved in rewriting the MakerNote. I've never had reason to explore this code in the past. It's quite abstract and uses arrays of structures to encode various IFDs. It's going to take a considerable effort to understand and debug this code. I've opened issue #1255 to undertake that task in v0.27.

#18

Updated by Robin Mills about 5 years ago

  • Category changed from miscellaneous to makernote

Also available in: Atom PDF