Chronos 1.4 RAW conversion tool - pyraw2dng

[tesla500]

For those of you itching to shoot RAW on your Chronos, you can now use the raw saving feature in the latest beta software v0.2.5 along with this new conversion tool to make DNG sequences. This method of saving raw is only recommended for legacy workflows, the CinemaDNG saving option should be uses in most cases.

pyraw2dng converts the .raw files saved by Chronos to standard Adobe DNG files.

The pyraw2dng utility is open source, you can view the source code on Github:
https://github.com/krontech/chronos-utils/tree/master/python_raw2dng

Requirements

Python 2.7, available here: https://www.python.org/downloads/release/python-278/
This might work in Python 3 as well, but it has not been tested.

During the Windows Python 2.7 installation, ?install? "Add Python.exe to Path" as shown below:

Usage

Save your video using the RAW16bit format on the camera. Raw16RJ works, but the images are dark and need to be fixed in post.

Download the script (see link above)

Copy the script file, pyraw2dng.py and the raw 16 bit format video into the same directory.

Open a terminal (command prompt), then navigate to the folder with your videos and script

Execute the script as shown in the example below. You will need to update the -w and -l switches with your actual frame size.
pyraw2dng.py -w (width) -l (length) (filename)

Where:
(width) is the horizontal resolution of your recording
(length) is the vertical resolution of your recording
(filename) is the .raw file you want to convert

An example of a complete command is shown below:
pyraw2dng.py -w 1280 -l 1024 vid.raw

The raw video will be converted and saved as a DNG sequence in a folder of the same name as the .raw file.

Let us know how this works for you!

 

[john delonghi]

It might be useful if you could post here the advantages of shooting RAW with the Chronos...

 

[tesla500]

It might be useful if you could post here the advantages of shooting RAW with the Chronos...

I'll let this comparison speak for itself:

Advantages of RAW are far greater control of the image processing pipeline, better color correction, better demosaic, etc. This ultimately results in far sharper and more accurate images than the current in-camera processing provides.

Better in-camera processing is on the way, eventually you should expect in-camera to be almost as good as the raw samples.

You can also find the full scenes, and dng attached.

 

[john delonghi]

Well that's very dramatic. Much more so that I'd see with a jpg/RAW comparison from a DSLR.
Needless to say, I'll be shooting RAW from now on!

Is there any chance of posting a road map of the various key firmware improvements you're working on so we'd get an idea of which will be available when?

 

[JamesB]

Wonder if it would be possible to save both RAW data and H.264 of the same clip automatically in a future firmware. Without having to change settings and save each one separately. That way it will be like RAW + JPG on still cameras.  The detail improvement with RAW is like getting a new camera.  Will test in the next few days.  ;D

 

[Jeffism]

Wow, the difference is much greater than I expected. The details on the stick of RAM and the colours of the cables 

 

[gyppor]

Wow, that's dramatic. Really dramatic. I've never shot raw before but I'm going to take the plunge, 100%.

 

[patrickrebstock]

wow those differences are dramatic!

 

[john delonghi]

This is going to help a lot when lighting conditions are not as good as we'd like them to be.

I've done a couple of quick tests and this is the workflow I've used for them as it may help you if you want to try recording in RAW mode.

1. Generate .raw file as David outlines above
2. In Adobe Bridge, open the first .DNG file as a Camera Raw file and tweak exposure, sharpness, colour etc. As the DNG is a raw file format you can do a lot here to improve the overall look and quality of each frame
3. Apply changes to all the DNG files and save them. I save them as .JPGs for the next step
4. In Photoshop, Open As... Image sequence and pull in all the JPGs
5. Export... Render Video and save whatever video format and quality you want to use. I've used Adobe H264 max. quality for these test.

The light was really dull when I took these, so the processed RAW version does have quite a lot of noise, but it's a massive improvement from the original and a much better result than I achieved tweaking the brightness, contrast and colour in my edit software.

Here's an original untouched frame from the Chronos H264 version of the clip I saved:

Here's a frame from the final video I rendered from PS:

Here are the links to the videos:

Original from Chronos:
https://youtu.be/6JMfVIYYci0

Original tweaked in edit software:
https://youtu.be/upStvyZh9U0

RAW version tweaked in PS:
https://youtu.be/Dk1ShX0J8WI

 

[patrickrebstock]

so i tried my first save a raw file today and it froze and stoped pretty early on i looked at the file on the card and its 4.29 gb do you know why it may have stoped saving and froze? it saved  the h264 just fine before that

 

[tesla500]

so i tried my first save a raw file today and it froze and stoped pretty early on i looked at the file on the card and its 4.29 gb do you know why it may have stoped saving and froze? it saved  the h264 just fine before that

How long did it take to freeze approximately? There's currently a limitation with FAT32 formatted cards, they can't handle files over 4GB, if the file you saved was larger that would explain the problem. You can try an EXT3 formatted card if you want to save larger files. exFAT and NTFS support is in the works so this will be solved.

 

[jefish]

Here are the links to the videos:

Original from Chronos:
https://youtu.be/6JMfVIYYci0

Original tweaked in edit software:
https://youtu.be/upStvyZh9U0

RAW version tweaked in PS:
https://youtu.be/Dk1ShX0J8WI

That's just extraordinary.

 

[john delonghi]

The 4GB limit is going to be an issue when saving in RAW mode, so I'm hoping you can solve it soon!

I tried EXT3 formatting, but I couldn't get it to work.

 

[patrickrebstock]

so i tried my first save a raw file today and it froze and stoped pretty early on i looked at the file on the card and its 4.29 gb do you know why it may have stoped saving and froze? it saved  the h264 just fine before that

How long did it take to freeze approximately? There's currently a limitation with FAT32 formatted cards, they can't handle files over 4GB, if the file you saved was larger that would explain the problem. You can try an EXT3 formatted card if you want to save larger files. exFAT and NTFS support is in the works so this will be solved.

At least 5-10minutes it was saving at 3-5fps I think

 

[tesla500]

At least 5-10minutes it was saving at 3-5fps I think

That lines up with the 4GB limit. The camera should warn you about this, this was added between v0.2.3 and 0.2.5, which version are you on?

 

[john delonghi]

I have made some raw recordings at different resolutions but didn't make a note of the settings.

How can I find out what length and width to put into the script just from the RAW file?

 

[JamesB]

Well, I tested the heck out of the RAW recording this weekend and I am very impressed with the quality gains. So much so that H.264 seems to be an easy way to destroy your image information.  I believe there should be a way of improving the h.264 output to offer better de-bayering even at the cost of encoding speed.  If DSLRs and Mirrorless cameras can offer h.264 free of most Bayer artifacts the Chronos should be able to do so as well. You will still have less dynamic range and much less color information but at least the color border artifacts should be gone.

I ran into problems with file size as well so the 4GB limit warning was good but it came up even when the file to be saved was just 1.8GB So there is no way of telling.  Your best bet is to have ample card speed and do attempts.  My average RAW file size was 1.67GB by just saving a small part of the action which is always trying to capture what is best.  I had to fill 2 32GB cards in my tests.  Sadly I read that the 12 bit compact RAW is not yet supported but will be so I'll have to wait for that to materialize to see the output.  I also recorded an h.264 version of most clips so I could see the gains in quality.

Much to my surprise H.264 in camera is a horrible conversion compared to the RAW data debayer in Camera RAW, much more so than I expected based on the Krontech frames that were shared.  Noise is still a problem for the RAW files but the extra information and ability to reduce color and luminance noise in Camera RAW before developing is very nice.  The Amount of detail preserved is night and day. Dynamic range is also fantastic by comparison, no contest... the ability to white balance after the fact can also not be understated.  The DNG Converter does a great job with no quality degradation well-done team!

Things that can improve: 

1) Ability to save H.264 and RAW at the same time so you can leave the camera unattended as it takes sometimes 10+min to save a RAW file. Always good to have the H.264 just in case.

2) Ability to eject the SD Card from the save clip dialogue as I ran out of space before saving one time, it would be nice to be able to swap from the saving dialogue box without corruption possibility if force ejected.

3) Camera locked up when card filled and had to do a hard reset. There was a dialogue that said something about card space and I said ok and camera locked up.  Abort button did not respond.

4) When shooting in 720p or lower resolution it would be great to have the button / scrollbar interface on the bottom to amplify the screen real estate and see a larger preview. Only full sensor size seems to require interface to the right. A simple orientation switch would be great.

5) Of course DNG direct from camera which is coming but also a chance to take a second look at H.264 and try to improve the output, the encoder is not doing a proper job of preserving detail and could really be a near RAW detail image without all of these artifacts on edges jumping out.  Most cameras from big manufacturers can do extremely clean H.264 requiring RAW only in specialized situations.  Even if the speed on saving is sacrificed H.264 high-quality encoding should be on the timeline for a software update.  Fingers Crossed X

I am including a frame from RAW and H.264 from one of the videos side by side. Look at the detail and water patterns for artifacts on the H.264, it is night and day difference.  Chronos 1.4 comes alive with RAW. 

 

[john delonghi]

Absolutely agree with all of that. RAW is the only way to go IMO but to make it usable we need:

Remove 4GB filesize limit
Speed up write to the SD/USB
Include metadata somehow with each recording

I'm happy to do a lot of post processing as that can be largely automated and I am about to order a nice new i7 8700-based PC
RAW+jpg would be nice, but not if it impacts on save time too much.

Of course, if all of this can be achieved once we have tethered ethernet working, that will probably be the better way to go anyway.

I still need some way to find out what resolution is of some of the raw files I recorded yesterday...

 

[tesla500]

I have made some raw recordings at different resolutions but didn't make a note of the settings.

How can I find out what length and width to put into the script just from the RAW file?

You'll need to use trial and error for now, unfortunately. Try different values for horizontal resolution, leaving the vertical resolution set to 1024, until you get an image that looks clear. Once you have horizontal resolution set correctly, it's much easier to figure out the proper vertical resolution just by looking at the output image.

This conversion tool is very much a stopgap measure until we have this functionality built into the camera.

 

[tesla500]

Well, I tested the heck out of the RAW recording this weekend and I am very impressed with the quality gains. So much so that H.264 seems to be an easy way to destroy your image information.  I believe there should be a way of improving the h.264 output to offer better de-bayering even at the cost of encoding speed.  If DSLRs and Mirrorless cameras can offer h.264 free of most Bayer artifacts the Chronos should be able to do so as well. You will still have less dynamic range and much less color information but at least the color border artifacts should be gone.

I ran into problems with file size as well so the 4GB limit warning was good but it came up even when the file to be saved was just 1.8GB So there is no way of telling.  Your best bet is to have ample card speed and do attempts.  My average RAW file size was 1.67GB by just saving a small part of the action which is always trying to capture what is best.  I had to fill 2 32GB cards in my tests.  Sadly I read that the 12 bit compact RAW is not yet supported but will be so I'll have to wait for that to materialize to see the output.  I also recorded an h.264 version of most clips so I could see the gains in quality.

Much to my surprise H.264 in camera is a horrible conversion compared to the RAW data debayer in Camera RAW, much more so than I expected based on the Krontech frames that were shared.  Noise is still a problem for the RAW files but the extra information and ability to reduce color and luminance noise in Camera RAW before developing is very nice.  The Amount of detail preserved is night and day. Dynamic range is also fantastic by comparison, no contest... the ability to white balance after the fact can also not be understated.  The DNG Converter does a great job with no quality degradation well-done team!

Things that can improve: 

1) Ability to save H.264 and RAW at the same time so you can leave the camera unattended as it takes sometimes 10+min to save a RAW file. Always good to have the H.264 just in case.

2) Ability to eject the SD Card from the save clip dialogue as I ran out of space before saving one time, it would be nice to be able to swap from the saving dialogue box without corruption possibility if force ejected.

3) Camera locked up when card filled and had to do a hard reset. There was a dialogue that said something about card space and I said ok and camera locked up.  Abort button did not respond.

4) When shooting in 720p or lower resolution it would be great to have the button / scrollbar interface on the bottom to amplify the screen real estate and see a larger preview. Only full sensor size seems to require interface to the right. A simple orientation switch would be great.

5) Of course DNG direct from camera which is coming but also a chance to take a second look at H.264 and try to improve the output, the encoder is not doing a proper job of preserving detail and could really be a near RAW detail image without all of these artifacts on edges jumping out.  Most cameras from big manufacturers can do extremely clean H.264 requiring RAW only in specialized situations.  Even if the speed on saving is sacrificed H.264 high-quality encoding should be on the timeline for a software update.  Fingers Crossed X

I am including a frame from RAW and H.264 from one of the videos side by side. Look at the detail and water patterns for artifacts on the H.264, it is night and day difference.  Chronos 1.4 comes alive with RAW. 

I absolutely agree with you there, the in-camera processing needs a lot of improvement. The current project is still Ethernet control, which is turning out to be a much larger project that anticipated. Once that's done we'll be working on a better in-camera video pipeline, the primary goals are better democratic (Likely AHD, same as used by Adobe Camera RAW last time I checked), as well as better color processing. I believe this will fix the h264 quality issues.

Metadata, large file support and write speed increases are also in the works. Right now, you can work around these issues by using an EXT3 formatted eSATA or USB hard drive. Write speeds up to 30MB/s should be achievable currently.

Combined h264 and RAW saving is something we can do shortly as well.