Author Topic: Chronos 1.4 RAW conversion tool - pyraw2dng  (Read 89936 times)

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #15 on: February 25, 2018, 09:27:56 AM »
I have made some raw recordings at different resolutions but didn't make a note of the settings.

How can I find out what length and width to put into the script just from the RAW file?

JamesB

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #16 on: February 25, 2018, 06:30:49 PM »
Well, I tested the heck out of the RAW recording this weekend and I am very impressed with the quality gains. So much so that H.264 seems to be an easy way to destroy your image information.  I believe there should be a way of improving the h.264 output to offer better de-bayering even at the cost of encoding speed.  If DSLRs and Mirrorless cameras can offer h.264 free of most Bayer artifacts the Chronos should be able to do so as well. You will still have less dynamic range and much less color information but at least the color border artifacts should be gone.

I ran into problems with file size as well so the 4GB limit warning was good but it came up even when the file to be saved was just 1.8GB So there is no way of telling.  Your best bet is to have ample card speed and do attempts.   My average RAW file size was 1.67GB by just saving a small part of the action which is always trying to capture what is best.  I had to fill 2 32GB cards in my tests.  Sadly I read that the 12 bit compact RAW is not yet supported but will be so I'll have to wait for that to materialize to see the output.  I also recorded an h.264 version of most clips so I could see the gains in quality.

Much to my surprise H.264 in camera is a horrible conversion compared to the RAW data debayer in Camera RAW, much more so than I expected based on the Krontech frames that were shared.   Noise is still a problem for the RAW files but the extra information and ability to reduce color and luminance noise in Camera RAW before developing is very nice.  The Amount of detail preserved is night and day. Dynamic range is also fantastic by comparison, no contest... the ability to white balance after the fact can also not be understated.  The DNG Converter does a great job with no quality degradation well-done team!

Things that can improve: 

1) Ability to save H.264 and RAW at the same time so you can leave the camera unattended as it takes sometimes 10+min to save a RAW file. Always good to have the H.264 just in case.

2) Ability to eject the SD Card from the save clip dialogue as I ran out of space before saving one time, it would be nice to be able to swap from the saving dialogue box without corruption possibility if force ejected.

3) Camera locked up when card filled and had to do a hard reset. There was a dialogue that said something about card space and I said ok and camera locked up.  Abort button did not respond.

4) When shooting in 720p or lower resolution it would be great to have the button / scrollbar interface on the bottom to amplify the screen real estate and see a larger preview. Only full sensor size seems to require interface to the right. A simple orientation switch would be great.

5) Of course DNG direct from camera which is coming but also a chance to take a second look at H.264 and try to improve the output, the encoder is not doing a proper job of preserving detail and could really be a near RAW detail image without all of these artifacts on edges jumping out.  Most cameras from big manufacturers can do extremely clean H.264 requiring RAW only in specialized situations.  Even if the speed on saving is sacrificed H.264 high-quality encoding should be on the timeline for a software update.  Fingers Crossed X

I am including a frame from RAW and H.264 from one of the videos side by side. Look at the detail and water patterns for artifacts on the H.264, it is night and day difference.  Chronos 1.4 comes alive with RAW.  ;)


John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #17 on: February 26, 2018, 01:58:59 AM »
Absolutely agree with all of that. RAW is the only way to go IMO but to make it usable we need:

Remove 4GB filesize limit
Speed up write to the SD/USB
Include metadata somehow with each recording

I'm happy to do a lot of post processing as that can be largely automated and I am about to order a nice new i7 8700-based PC :)
RAW+jpg would be nice, but not if it impacts on save time too much.

Of course, if all of this can be achieved once we have tethered ethernet working, that will probably be the better way to go anyway.

I still need some way to find out what resolution is of some of the raw files I recorded yesterday...


tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #18 on: February 26, 2018, 01:07:42 PM »
I have made some raw recordings at different resolutions but didn't make a note of the settings.

How can I find out what length and width to put into the script just from the RAW file?

You'll need to use trial and error for now, unfortunately. Try different values for horizontal resolution, leaving the vertical resolution set to 1024, until you get an image that looks clear. Once you have horizontal resolution set correctly, it's much easier to figure out the proper vertical resolution just by looking at the output image.

This conversion tool is very much a stopgap measure until we have this functionality built into the camera.

tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #19 on: February 26, 2018, 01:16:24 PM »
Well, I tested the heck out of the RAW recording this weekend and I am very impressed with the quality gains. So much so that H.264 seems to be an easy way to destroy your image information.  I believe there should be a way of improving the h.264 output to offer better de-bayering even at the cost of encoding speed.  If DSLRs and Mirrorless cameras can offer h.264 free of most Bayer artifacts the Chronos should be able to do so as well. You will still have less dynamic range and much less color information but at least the color border artifacts should be gone.

I ran into problems with file size as well so the 4GB limit warning was good but it came up even when the file to be saved was just 1.8GB So there is no way of telling.  Your best bet is to have ample card speed and do attempts.   My average RAW file size was 1.67GB by just saving a small part of the action which is always trying to capture what is best.  I had to fill 2 32GB cards in my tests.  Sadly I read that the 12 bit compact RAW is not yet supported but will be so I'll have to wait for that to materialize to see the output.  I also recorded an h.264 version of most clips so I could see the gains in quality.

Much to my surprise H.264 in camera is a horrible conversion compared to the RAW data debayer in Camera RAW, much more so than I expected based on the Krontech frames that were shared.   Noise is still a problem for the RAW files but the extra information and ability to reduce color and luminance noise in Camera RAW before developing is very nice.  The Amount of detail preserved is night and day. Dynamic range is also fantastic by comparison, no contest... the ability to white balance after the fact can also not be understated.  The DNG Converter does a great job with no quality degradation well-done team!

Things that can improve: 

1) Ability to save H.264 and RAW at the same time so you can leave the camera unattended as it takes sometimes 10+min to save a RAW file. Always good to have the H.264 just in case.

2) Ability to eject the SD Card from the save clip dialogue as I ran out of space before saving one time, it would be nice to be able to swap from the saving dialogue box without corruption possibility if force ejected.

3) Camera locked up when card filled and had to do a hard reset. There was a dialogue that said something about card space and I said ok and camera locked up.  Abort button did not respond.

4) When shooting in 720p or lower resolution it would be great to have the button / scrollbar interface on the bottom to amplify the screen real estate and see a larger preview. Only full sensor size seems to require interface to the right. A simple orientation switch would be great.

5) Of course DNG direct from camera which is coming but also a chance to take a second look at H.264 and try to improve the output, the encoder is not doing a proper job of preserving detail and could really be a near RAW detail image without all of these artifacts on edges jumping out.  Most cameras from big manufacturers can do extremely clean H.264 requiring RAW only in specialized situations.  Even if the speed on saving is sacrificed H.264 high-quality encoding should be on the timeline for a software update.  Fingers Crossed X

I am including a frame from RAW and H.264 from one of the videos side by side. Look at the detail and water patterns for artifacts on the H.264, it is night and day difference.  Chronos 1.4 comes alive with RAW.  ;)

I absolutely agree with you there, the in-camera processing needs a lot of improvement. The current project is still Ethernet control, which is turning out to be a much larger project that anticipated. Once that's done we'll be working on a better in-camera video pipeline, the primary goals are better democratic (Likely AHD, same as used by Adobe Camera RAW last time I checked), as well as better color processing. I believe this will fix the h264 quality issues.

Metadata, large file support and write speed increases are also in the works. Right now, you can work around these issues by using an EXT3 formatted eSATA or USB hard drive. Write speeds up to 30MB/s should be achievable currently.

Combined h264 and RAW saving is something we can do shortly as well.

JamesB

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #20 on: February 26, 2018, 01:38:59 PM »
Thanks, David, it is really encouraging to see the Chronos starting to mature into a dependable professional tool.   Excited by the upcoming possibilities from you and the development team.  :D

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #21 on: February 27, 2018, 12:46:20 AM »
I have made some raw recordings at different resolutions but didn't make a note of the settings.

How can I find out what length and width to put into the script just from the RAW file?

You'll need to use trial and error for now, unfortunately. Try different values for horizontal resolution, leaving the vertical resolution set to 1024, until you get an image that looks clear. Once you have horizontal resolution set correctly, it's much easier to figure out the proper vertical resolution just by looking at the output image.

This conversion tool is very much a stopgap measure until we have this functionality built into the camera.

Ok I guessed the right number, so I can process those raw files now! Thanks.
« Last Edit: February 27, 2018, 04:32:58 AM by John DeLonghi »

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #22 on: February 28, 2018, 08:13:25 AM »
I've now managed to get over the 4GB limit using the method Dan D found, but on a Windows 10 PC.

If you want to save RAW files larger than 4GB, here's what you do:

Download and install the free utilities 'MiniTool Partition Wizard Free' and 'Ext2Fsd'.
Format a USB stick or SD card as EXT3 using MiniTool Partition Wizard. I've used a 128GB USB stick successfully.
Plug it into the Chronos and select it in the Save Location dropdown. If it doesn't appear, something's gone wrong with the formatting of the drive.
Record and save to the EXT3 drive (when you get the 4GB warning message just tell it to save anyway.)
When finished saving, plug the card/stick into your PC and use Ext2Fsd to copy the contents to an NTFS drive. 

I just did this with a 21GB recording. It took 47 minutes to save to the stick, but we know it's going to be slow at the moment.
The main thing is it worked I don't have the 4GB limit any more.

I think I better free up some hard drive space now  ;D

patrickrebstock

  • Full Member
  • ***
  • Posts: 114
  • playing camera
    • View Profile
    • patrickrebstock.com
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #23 on: March 06, 2018, 12:48:26 PM »
For those of you itching to shoot RAW on your Chronos, you can now use the raw saving feature in the latest beta software v0.2.5 along with this new conversion tool to make DNG sequences. This is a stopgap solution until we have DNG saving built into the camera.

pyraw2dng converts the .raw files saved by Chronos to standard Adobe DNG files.

The pyraw2dng utility is open source, you can view the source code on Github:
https://github.com/krontech/chronos-utils/tree/master/python_raw2dng

Requirements

Python 2.7, available here: https://www.python.org/downloads/release/python-278/
This might work in Python 3 as well, but it has not been tested.

During the Windows Python 2.7 installation, ‘install’ "Add Python.exe to Path" as shown below:



Usage

Save your video using the RAW16bit format on the camera. Raw16RJ works, but the images are dark and need to be fixed in post. RAW12bit packed mode is not yet supported, but is coming soon.

Download the script (see attachment)

Copy the script file, pyraw2dng.py and the raw 16 bit format video into the same directory.

Open a terminal (command prompt), then navigate to the folder with your videos and script

Execute the script as shown in the example below. You will need to update the -w and -l switches with your actual frame size.
pyraw2dng.py -w (width) -l (length) (filename)

Where:
(width) is the horizontal resolution of your recording
(length) is the vertical resolution of your recording
(filename) is the .raw file you want to convert

An example of a complete command is shown below:
pyraw2dng.py -w 1280 -l 1024 vid.raw

The raw video will be converted and saved as a DNG sequence in a folder of the same name as the .raw file.

Let us know how this works for you!
   how do you execute the script can you post screenshots for us not familiar with terminal?

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #24 on: March 07, 2018, 03:15:36 AM »
Quote from: patrickrebstock
how do you execute the script can you post screenshots for us not familiar with terminal?


Are you PC-based?

patrickrebstock

  • Full Member
  • ***
  • Posts: 114
  • playing camera
    • View Profile
    • patrickrebstock.com
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #25 on: March 07, 2018, 05:20:41 PM »
Quote from: patrickrebstock
how do you execute the script can you post screenshots for us not familiar with terminal?


Are you PC-based?
i have a pc and a mac

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #26 on: March 08, 2018, 01:37:52 AM »
As long as you've installed Python as David details, the easiest way is to create a folder on your PC, copy the pyraw2dng.py file and your .raw file from the Chronos into it, then create a batch file to run it.

I've attached an example .bat file you can use.

So, in the folder you should have test.bat, test.raw and pyraw2dng.py. Edit your .bat file with a text editor to get the resolution and raw filename correct then just double click the test.bat file and it should run.

Sorchey

  • Newbie
  • *
  • Posts: 8
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #27 on: April 04, 2018, 01:59:26 PM »
Once that's done we'll be working on a better in-camera video pipeline, the primary goals are better democratic (Likely AHD, same as used by Adobe Camera RAW last time I checked), as well as better color processing. I believe this will fix the h264 quality issues.

I've just seen this thread and I'm symply blown away by the comparison between RAW and h264.

Although it is a very unpopular question, could you predict a rough time schedule for this update? Can we expect something in that direction this year, maybe this summer?


tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #28 on: April 06, 2018, 07:03:09 PM »
I've just seen this thread and I'm symply blown away by the comparison between RAW and h264.

Although it is a very unpopular question, could you predict a rough time schedule for this update? Can we expect something in that direction this year, maybe this summer?

We're going to get the color improvements into v0.3. The release candidate was going to go out this week but we delayed to get the color changes in, check the Software Dev forum over the next week or two for the release candidate.

Better demosaic is going to be in V0.4 or V0.5 sometime late spring or early summer.

JamesB

  • Jr. Member
  • **
  • Posts: 68
    • View Profile
Re: Chronos 1.4 RAW conversion tool - pyraw2dng
« Reply #29 on: April 10, 2018, 01:18:02 PM »
Great news, also would love to get the compressed RAW converter option soon as I have a list of clips that are just sitting there. It also makes for faster storage in the field, regular RAW just eats up cards for lunch. 

I am very happy with the RAW quality and after using it, H.264, as it is right now, is not worth it even for the speed.  Also with proper and often black calibration, most banding is taken care of.   

A 10-bit version of H.265 or a 4:2:2 H.264 variant with better demosaic algo would make this amazing for speed and quality. 

It would also be so cool if we could have a daylight balance preset for the WB. Maybe be able to save a previous WB in a file as balancing every time is an extra step.  Most of us either shoot in our studio or outside needing only 2 WB presets.  For a later build of course.   Great job so far David!