Magic Lantern Missing End of File/Corrupt File / by Conor Cunningham

I ran into a spot of bother whilst trying Magic Lantern on my Canon 5d MkIII whilst filming Jupiter for a project I was working on.

In this post, I'll look at the problem I ran into, and how I solved it.

The Problems

I attached my 5d mkIII to my telescope and began filming Jupiter using the Magic Lantern software. I noticed after approximately 60 seconds that it stopped filming ( I heard the mirror fall back into place). I tried again, but the same thing happened.

More interestingly however was the when I looked at my card, I had two RAW files for each film. The first was named M10-1433.RAW and the other M10-1433.R00. I tried to open them in RAWMagic, but I received the following error message.

The error I received when trying to open the .RAW file in RAWMagic

Problem 1

Turns out that Canon formats the memory card using FAT32 ( a filesystem ) which only allows 4GB maximum file size. Hence the film was split into two separate files as the first file had reached the 4GB max. Ok, simple solution I thought, simply join the two files together using a UNIX command ( I run MacOSX and am an Linux geek ). I confirmed this with a google search and see that this is common and the fix was simply to join the files together. Too easy, or so I thought.

Solution 1

Using a terminal on MAC or Linux, simply dump the contents of the second file at the end of the first. To do this, use the UNIX command cat.

cat M10-1433.RAW M10-1433.R00 > jupiter1600.raw

The cat command reads and outputs the contents of a file. The > inputs what is on the left of it into what is on the right of it. So, in the above case the contents of M10-1433.RAW is read and input in jupiter1600.raw then the same is done for M10-1433.R00.

The UNIX command run in a MAC OSX Terminal Window

Problem 2

Problem two was a little more daunting. After joining the files together I attempted to open them in RAWMagic 1.0 but I received the same error message as above. Hadn't I joined the files together? I checked, doubled check and triple checked. Yes, the files were joined together. What was the problem?

I decided to shoot a 3 second and see if I could open it in RAWMagic. Yes, no problems at all. I converted it and everything was hunky dory.

The next thing I decided to do was to compare the files in a HEX editor. If you're not technically minded, this may be a bit confusing, but bare with me. If you have had this problem hopefully this will help you.

For this task I used Hex Fiend; a great wee Hex editor for MAC.

Solution 2

The first thing I did was scroll to the end of each file. I suspected that the End of File (EOF) may be missing from my jupiter1600.raw file. Here's what I saw.

Jupite1600.RAW on the left and my 3 second film on the right. Notice the extra data on the right. This is the

Notice on the left there is nothing but zeros whereas on the right there is data. Hey presto I thought, simply cut and paste the data from the right to the left and we're on our way. I then went to convert the file in RAWMagic, but there was a problem.

Look at the number of frames! The first file is 3 seconds long, the other 60 seconds!

My new 60 second joined file apparently had only 57 frames. (ok, so maybe my first file is only 2 seconds long). This meant that the number of frames in the file was stored in the last part of the data I had copied. 57 in decimal (our every day numbering system) is 39 in Hex. In the fourth column in the top row of data in the right image above is the number 39. This is telling the file how many frames are contained within. I need to change this. My film was 60 seconds long and shooting at 29 frames a second. 60 x 29 = 1740. (i'll use 1600 frames to be safe). 1600 in HEX = 640 (or more precisley 0x0640).

Ok, so here is where it gets a little bit tricky. Without getting all computer sciency, the RAW file isn't as straightforward as you might think. It uses Little Endian encoding which means to interpret the numbers we need to read them back to front. So, 0x0640 will actually be 0x4006 (we've reversed the positions of the bytes).

Have a look below. In red is the number of frames. Edit this to put in your number of frames. Be sure to note that the yellow area I've highlighted. If you have moved the 01 over a few positions to the right, simply delete the zeros until the 01 is back in the position as displayed below. The yellow tells the file what its frame rate is.

My finished file with 1600 frames.

Looking at RAWMagic below, we can now see that both my files are being read and with the correct number of frames.

RAWMagic can now open my file

For the curious, nerdy and geeky

For those of you wanting some more information, by using the source code we can work out the following from the first line of the HEX header.

Each 00 represents 1 Byte (1B). A Char equals 1 Byte, a Short equals 2 Bytes and an Int represents 4 Bytes.

Looking at the source code we see the following;

#ifndef _lv_rec_h_
#define _lv_rec_h_

#include "raw.h"

/* file footer data */
typedef struct
{
unsigned char magic[4];
unsigned short xRes;
unsigned short yRes;
unsigned int frameSize;
unsigned int frameCount;
unsigned int frameSkip;
unsigned int sourceFpsx1000;
unsigned int reserved3;
unsigned int reserved4;
struct raw_info raw_info;
} lv_rec_file_footer_t;

#endif

Source code from https://bitbucket.org/hudson/magic-lantern/src/622f3c2c43847df7779fae0f3a100625302a1051/modules/lv_rec/lv_rec.h?at=5D3-12

Our HEX data can now be broke down as follows

  1. 52 41 57 4D  = char magic[4]; ( Char = 1 Byte, but we use 4 chars here, so 4 Bytes)
  2. 40 06 = short xRes;
  3. 84 03 = short yRes;
  4. 00 80 26 00 = int frameSize;
  5. 40 06 00 00 = int frameCount;
  6. 01 00 00 00 = int frameSkip;
  7. 12 75 00 00 = sourceFpsx1000;

So on and so on.

Ok, so I hoped this helped you.