Bug #770
Exif.CanonFi.FileNumber value is always (0)
10%
Description
Exif.CanonFi.FileNumber is documented as a valid Canon File Info Tag, but exiv2 0.21.1 returns (0) when extracting this tag. On the same file, exiftool will return the correct metadata. I have tested this on a Canon 5D Mk II RAW file in which I know the correct value, so I can confirm the bug. Testing a Canon 1Ds Mk III RAW files also results in a value of 0 for File Number, but I cannot confirm if exiftool's output in this in this case or not because I did not create the file.
Files
Related issues
History
Updated by Alan Pater about 6 years ago
- Category set to makernote
- Target version changed from 53 to 0.26
Confirmed that issue remains with exiv2 0.25 using the sample from:
http://www.rawsamples.ch/raws/canon/5dm2/RAW_CANON_5DMARK2_PREPROD.CR2Updated by Robin Mills over 5 years ago
- Status changed from New to Resolved
- Assignee set to Robin Mills
- % Done changed from 0 to 100
- Estimated time set to 1.00 h
I think this has already been resolved. I couldn't reproduce this with RAW_CANON_5DMARK2_PREPROD.CR2, however it seems to be fixed with the file MG.CR2 which was supplied by a user to investigate #1081
Here's the evidence:
540 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiftool -all MG.CR2 | grep 123 File Number : 123-3694 541 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -pa --grep FileNumber MG.CR2 Exif.CanonFi.FileNumber Long 1 123-3694 542 rmills@rmillsmbp:~/gnu/exiv2/trunk $ exiv2 -vVg svn exiv2 0.25 001900 (64 bit build) svn=4268 543 rmills@rmillsmbp:~/gnu/exiv2/trunk $
Updated by Damon Lynch over 5 years ago
Robin Mills wrote:
I think this has already been resolved. I couldn't reproduce this with RAW_CANON_5DMARK2_PREPROD.CR2, however it seems to be fixed with the file MG.CR2 which was supplied by a user to investigate #1081
Here's the evidence:[...]
Unfortunately I don't think been fully resolved, at least with version 0.25 001900 (64 bit build) on a file from a Canon EOS-1D X:
$ exiftool 20160403-2031-1-iso400-f5.0-85mm-250.cr2 |grep Number
ExifTool Version Number : 10.10
F Number : 5.0
Shot Number In Continuous Burst : 0
Bracket Shot Number : 0
Internal Serial Number : X067442
Serial Number : 092015000200
Lens Serial Number : 0000013ebd
File Number : 100-1337
$ exiv2 -pt 20160403-2031-1-iso400-f5.0-85mm-250.cr2 |grep 1337
$ exiv2 -pt 20160403-2031-1-iso400-f5.0-85mm-250.cr2 |grep Number
Exif.Photo.FNumber Rational 1 F5
Exif.CanonFi.FileNumber Long 1 (0)
Exif.CanonFi.BracketShotNumber SShort 1 0
Exif.Canon.InternalSerialNumber Ascii 16 X067442
Exif.Photo.BodySerialNumber Ascii 13 092015000200
Exif.Photo.LensSerialNumber Ascii 11 0000013ebd
The same behaviour is exhibited on the file RAW_CANON_EOS_1DX.CR2 also found at www.rawsamples.ch
Updated by Robin Mills over 5 years ago
- Status changed from Resolved to Closed
Please submit a patch and update to the test harness.
Updated by Damon Lynch over 5 years ago
Robin Mills wrote:
Please submit a patch and update to the test harness.
I'm sorry, I don't understand your request. Nor do I understand why the bug is "closed" when the bug appears to still be present (although of course I'm not using the exiv2 from trunk).
Updated by Robin Mills over 5 years ago
I am not going to do anything about this. I intend to walk off this project. I have put a huge effort into Exiv2. People like you never say "Please" or "Thank You". I am totally pissed with the whole thing. You do understand that I am an unpaid volunteer? This is not a business, I don't earn a penny for the thousands of hours I put Exiv2.
I didn't write Exiv2. I don't care if this bug continues to exist of not. I have had enough.
Updated by Damon Lynch over 5 years ago
Robin Mills wrote:
I am not going to do anything about this. I intend to walk off this project. I have put a huge effort into Exiv2. People like you never say "Please" or "Thank You". I am totally pissed with the whole thing. You do understand that I am an unpaid volunteer? This is not a business, I don't earn a penny for the thousands of hours I put Exiv2.
I didn't write Exiv2. I don't care if this bug continues to exist of not. I have had enough.
I'm truly sorry that you feel this way. Thank you for your efforts through the years!
FWIW I maintain my own FOSS project which relies on exiv2. Without a tool like exiv2 my project could not exist. Likewise for me it is not a business and is something I do on a voluntary basis. I also have to deal with large numbers of bug reports and feature requests. I also need to file bugs with projects like exiv2 if there is any hope for them to be fixed. Sometimes the project acknowledges the bug (like in this case) or sometimes the bug report is completely ignored and consequently the bug can linger for years.
As an application developer, in the eyes of all but the most sophisticated users, my project bears the blame for all bugs even though the actual bug might be in a library that my applications calls.
Users almost never contribute code, nor do I expect them to.
Such is life when working with FOSS code. The joy of coding comes with the pain of hard work.
Again, thank you for your contributions Robin! Good luck to you, whatever you choose to do next.
Updated by Robin Mills over 5 years ago
- Status changed from Closed to New
- Assignee deleted (
Robin Mills)
Thanks. I'm having a very bad day for reasons that have nothing to do with software. I've had enough. If Exiv2 dies, so be it. Good Luck with your project.
I will reopen this bug report. Maybe somebody will service this issue.
Updated by Robin Mills over 5 years ago
- Target version changed from 0.26 to 1.0
Damon
I've found the file RAW_CANON_EOS_1DX.CR2 on www.rawsamples.ch and you are correct, the issue seems to be there on the trunk:
625 rmills@rmillsmbp:~/Downloads $ exiftool -all RAW_CANON_EOS_1DX.CR2 | grep -i number ExifTool Version Number : 10.13 F Number : 5.6 Shot Number In Continuous Burst : 0 Flash Guide Number : 0 Bracket Shot Number : 0 Internal Serial Number : X037087 Serial Number : 062011000450 Lens Serial Number : 0000000000 File Number : 100-0035 626 rmills@rmillsmbp:~/Downloads $ exiv2 -pa --grep number/i RAW_CANON_EOS_1DX.CR2 Exif.Photo.FNumber Rational 1 F5.6 Exif.CanonFi.FileNumber Long 1 (0) Exif.CanonFi.BracketShotNumber SShort 1 0 Exif.Canon.InternalSerialNumber Ascii 16 X037087 Exif.Photo.BodySerialNumber Ascii 13 062011000450 Exif.Photo.LensSerialNumber Ascii 11 0000000000 627 rmills@rmillsmbp:~/Downloads $I don't know how exiftool is able to recover 100-0035
We have an outstanding issue #992 to work on raw image support. The target for #992 is v1.0. A user once stated that "v1.0" means "it'll never get dealt with". It does not mean that. It means it is not scheduled for attention. We have 35 issues in target v1.0 and most will take weeks or months.
I apologise for "blowing up" on Wednesday. The trigger was neither you nor Exiv2. On Wednesday afternoon, I received 3 emails saying "Thanks for working on Exiv2". What a pleasant and motivating surprise. I suspect you triggered that encouragement. Thank You for doing that.
I have set the target for this issue to v1.0 and hope that one day Exiv2 will have an engineer available to work on that part of the code.
Updated by Robin Mills over 5 years ago
- % Done changed from 100 to 10
- Estimated time changed from 1.00 h to 10.00 h
Updated by Ari Exi almost 5 years ago
Issue also currently affects
Canon Rebel / T3
Exif.Canon.ImageType Ascii 16 Canon EOS 1100D
Exif.Canon.ModelID Long 1 EOS Rebel T3 / 1100D
exiftool 20161008-155525-Rach-2997.JPG |grep File
File Number : 100-2997
exiv2 -pt 20161008-155525-Rach-2997.JPG |grep File
Exif.CanonFi.FileNumber Long 1 (0)
From exiftool,
....#------------------------------------------------------------------------------
- File: CanonRaw.pm #
- Description: Read Canon RAW (CRW) meta information
....
- Canon raw file tag table
- Note: Tag ID's have upper 2 bits set to zero, since these 2 bits
- just specify the location of the information
%Image::ExifTool::CanonRaw::Main = (
GROUPS => { 0 => 'MakerNotes', 2 => 'Camera' },
PROCESS_PROC => \&ProcessCanonRaw,
WRITE_PROC => \&WriteCanonRaw,
CHECK_PROC => \&CheckCanonRaw,
WRITABLE => 1,
0x0000 => { Name => 'NullRecord', Writable => 'undef' }, #3
0x0001 => { #3
Name => 'FreeBytes',
Format => 'undef',
Binary => 1,
},
0x0032 => { Name => 'CanonColorInfo1', Writable => 0 },0x1817 => {
Name => 'FileNumber',
Writable => 'int32u',
Groups => { 2 => 'Image' },
PrintConv => '$_=$val;s/(\d+)(\d{4})/$1-$2/;$_',
PrintConvInv => '$_=$val;s/-//;$_',
},
Updated by Robin Mills almost 5 years ago
Thank You, Ari for the update. Can you attach your test file, please?
Updated by Ari Exi almost 5 years ago
- File 20161009-100-3999.JPG 20161009-100-3999.JPG added
- File 20161016-1128.CR2 20161016-1128.CR2 added
I attach two files for comparison, from two different Canons, one JPG from a Canon Rebel T3 (EOS1100D) (recent) JPG, and a CR2 from an S100 (3-4 years old).
I realized that it was the JPG giving probems.
The JPG has an exif tag: "Exif.CanonFi.FileNumber" that gives problems with Exiv2 but works OK with exiftool:
exiftool 20161009-100-3999.JPG |grep "File Num"
File Number : 100-3999
exiv2 20161009-100-3999.JPG -pt |grep File
Exif.CanonFi.FileNumber Long 1 (0)
All images from this camera give the same "0" File Number value with Exiv2 (and are OK for exiftool)
Note the "Fi" in the tag name.
Here's the same issue reported for images from Canon 400D, A85: https://bugs.launchpad.net/rapid/+bug/754531
I can also see the problem here from a digikam error report, Canon 50D image: https://bugs.kde.org/show_bug.cgi?id=257904
In contrast the CR2 from the S100 works in both Exiv2 and exiftool:
exiftool 20161016-1128.CR2 |grep "File Num"
File Number : 103-1128
exiv2 -pt 20161016-1128.CR2 |grep FileN
Exif.Canon.FileNumber Long 1 103-1128
So the non-"Fi" tag works well, but the "Fi" one doesn't.
Looking again at exiftool code, this time at "canon.pm", I see model-dependent code to handle file-number.
.................
#------------------------------------------------------------------------------
this seems to be the "CanonFi" part in canon.pm
....
0x93 => {
Name => 'CanonFileInfo', # (ShootInfoEx)
SubDirectory => {
Validate => 'Image::ExifTool::Canon::Validate($dirData,$subdirStart,$size)',
TagTable => 'Image::ExifTool::Canon::FileInfo',
},
......
- File number information (MakerNotes tag 0x93)
%Image::ExifTool::Canon::FileInfo = (
%binaryDataAttrs,
FORMAT => 'int16s',
FIRST_ENTRY => 1,
GROUPS => { 0 => 'MakerNotes', 2 => 'Image' },
DATAMEMBER => [ 20 ],
1 => [ { #5
Name => 'FileNumber',
Condition => '$$self{Model} =~ /\b(20D|350D|REBEL XT|Kiss Digital N)\b/',
Format => 'int32u', # Thanks to Juha Eskelinen for figuring this out: # [this is an odd bit mapping -- it looks like the file number exists as # a 16-bit integer containing the high bits, followed by an 8-bit integer # with the low bits. But it is more convenient to have this in a single # word, so some bit manipulations are necessary... - PH] # The bit pattern of the 32-bit word is: # 31....24 23....16 15.....8 7......0 # 00000000 ffffffff DDDDDDDD ddFFFFFF # 0 = zero bits (not part of the file number?) # f/F = low/high bits of file number # d/D = low/high bits of directory number # The directory and file number are then converted into decimal # and separated by a '-' to give the file number used in the 20D
ValueConv => '(($val&0xffc0)>>6)*10000+(($val>>16)&0xff)+(($val&0x3f)<<8)',
ValueConvInv => q{
my $d = int($val/10000);
my $f = $val - $d * 10000;
return (($d<<6) & 0xffc0) + (($f & 0xff)<<16) + (($f>>8) & 0x3f);
},
PrintConv => '$_=$val,s/(\d+)(\d{4})/$1-$2/,$_',
PrintConvInv => '$val=~s/-//g;$val',
}, { #16
Name => 'FileNumber',
Condition => '$$self{Model} =~ /\b(30D|400D|REBEL XTi|Kiss Digital X|K236)\b/',
Format => 'int32u',
Notes => q{
the location of the upper 4 bits of the directory number is a mystery for
the EOS 30D, so the reported directory number will be incorrect for original
images with a directory number of 164 or greater
}, # Thanks to Emil Sit for figuring this out: # [more insane bit maniplations like the 20D/350D above, but this time we # appear to have lost the upper 4 bits of the directory number (this was # verified through tests with directory numbers 100, 222, 801 and 999) - PH] # The bit pattern for the 30D is: (see 20D notes above for more information) # 31....24 23....16 15.....8 7......0 # 00000000 ffff0000 ddddddFF FFFFFFFF # [NOTE: the 4 high order directory bits don't appear in this record, but # I have chosen to write them into bits 16-19 since these 4 zero bits look # very suspicious, and are a convenient place to store this information - PH]
ValueConv => q{
my $d = ($val & 0xffc00) >> 10; # we know there are missing bits if directory number is < 100
$d = 0x40 while $d < 100; # (repair the damage as best we can)
return $d*10000 + (($val&0x3ff)<<4) + (($val>>20)&0x0f);
},
ValueConvInv => q{
my $d = int($val/10000);
my $f = $val - $d * 10000;
return ($d << 10) + (($f>>4)&0x3ff) + (($f&0x0f)<<20);
},
PrintConv => '$_=$val,s/(\d)(\d{4})/$1-$2/,$_',
PrintConvInv => '$val=~s/-//g;$val',
},
Updated by Robin Mills almost 5 years ago
- Target version changed from 1.0 to 0.28
Thanks very much for the test files. I confirm your findings:
$ exiv2 -pa --grep file/i http://dev.exiv2.org/attachments/download/1110/20161009-100-3999.JPG Exif.CanonFi.FileNumber Long 1 (0) $ exiv2 -pa --grep file/i http://dev.exiv2.org/attachments/download/1111/20161016-1128.CR2 Exif.Canon.FileNumber Long 1 103-1128 Exif.Photo.FileSource Undefined 1 Digital still camera $Congratulations on reading the exiftool's hiroglyphic perl.
The CR2 and JPG seem to have a different makernote:
$ exiv2 -pa --grep makernote/i --binary http://dev.exiv2.org/attachments/download/1111/20161016-1128.CR2 > cr2.bin $ exiv2 -pa --grep makernote/i --binary http://dev.exiv2.org/attachments/download/1110/20161009-100-3999.JPG > jpg.bin $ ls -alt *.bin -rw-r--r--+ 1 rmills staff 16054 Nov 21 08:12 jpg.bin -rw-r--r--+ 1 rmills staff 19812 Nov 21 08:12 cr2.bin $This issue has been in the code for a long time. Exiv2 is currently at "Release Candidate 1" for v0.26. I've assigned this for attention in Exiv2 v0.27 which I hope will get underway early in 2017. The project has had the good fortune to recruit 4 new engineers and I hope that at least one will become an expert in the code to encode/decode MakerNote.
I also hope that in Exiv2 v0.27 the team will be able to extend our image test coverage. For example, I've downloaded all the RAW images from https://www.rawsamples.ch/index.php/en/ and all of exiftool's test files (7000 of them). My thoughts are to extend the buildserver test code to monitor/compare exiftool and Exiv2 output on our daily build.
Thank you for updating this issue and providing additional test files. I hope this will be fixed in v0.27. Additionally I hope to extend our image test coverage in during the v0.27 project.