Author Topic: Software V0.3 Beta - Updated May 3 2018  (Read 66678 times)

jasonfish

  • Jr. Member
  • **
  • Posts: 81
    • View Profile
Re: Software V0.3 Beta
« Reply #15 on: December 20, 2017, 11:30:48 AM »
I'm getting best save performance to a high-end USB 3.0 compatible thumb drive formatted fat32, at about 13 MB/s. It's twice as fast as a '30 MB/s' rated SD card, which seems to only give me about 7 MB/s? Maybe it's not a good card.

...

Partitioned memory mode is cool. I don't see support for being able to switch memory segments yet though, did I miss something?

I get about 10% faster saves when using a USB>SD adapter vs the same card in the SD slot. I think the USB interface is just slightly faster, for one.

---

Same question here.

Loial

  • Krontech
  • Newbie
  • *****
  • Posts: 24
    • View Profile
Re: Software V0.3 Beta
« Reply #16 on: December 20, 2017, 03:59:28 PM »
First of all; Thanks for the fantastic feedback. This is a huge help!

Hey guys, thanks for releasing the beta software early! Sorry for long post. Bug reports and tips below.

I'm using the camera in an optics lab at a university. RAW is essential; 12 bit packed mode is great! Our lab has some high end scientific high speed cameras that cost at LEAST 30x more than the Chronos. I've done side by side comparisons and the Chronos is really holding its own - great work Dave & co. Next year, I'll write up and share the data. The only thing that lets the camera down right now is the lack of a remote control API via ethernet and a working HDMI output so we can control & view remotely. I'm sure it will come! :-)

If anyone else is planning to put the Chronos 1.4 to scientific use, perhaps we should start a forum page for sharing code, tips, and example videos of cool stuff. I'm hoping to write an ImageJ/Fiji plugin for reading in the raw data next year. For now, I've got a Python program successfully extracting frames from the 12 bit packed format with no trouble. Once I've optimised the code (it's rather slow and lacking comments), I'll put it on Github. Planning to add support for directly reading gzipped raw files so we can have true lossless compression.

I've done a whole lot of playing about with the 0.2.3 beta in the last few days on my monochrome 8 GB model and I've found some interesting bugs and workarounds. Info will follow for those who care. I'm happy to field-test stuff and follow up on bug fixes if it helps.

Awesome to hear you got that working. I only just made the 16bpp translation and was going to tackle the 12bpp packed format in the next bit.

Dan D.

WRITE FILES BIGGER THAN 4GB!
When trying to dump the whole RAM to RAW, the fat32 file size limit of 4GB is a REAL pain. And almost as bad, I can't write past the first 8 GB of fat32 media. I'm going to assume NTFS is out of the question, and exFAT is not supported yet. Since the Chronos is running *nix, I thought I'd take a punt and format my flash drive to ext3 - and it works!! I can now save the FULL ram of the camera (8gb) in a single hit to one file, and to boot I can use the whole size of the drive rather than just the first 8GB! The "above 4GB" warning message will still appear - just acknowledge & continue saving. This is super cool, with two drawbacks. One, mounting ext2/3/4 in windows & mac is a pain - and two, the save speed is roughly half that of fat32.

I'm getting best save performance to a high-end USB 3.0 compatible thumb drive formatted fat32, at about 13 MB/s. It's twice as fast as a '30 MB/s' rated SD card, which seems to only give me about 7 MB/s? Maybe it's not a good card.

The save performance should improve in the future - we've found a rather annoying "feature" of the video pipeline in use in that it does a memcpy to move the frame into kernel space from OMX. This is where all the extra speed that saving to USB or eSATA should be given is being eaten up; the CPU can only move so much data.

It's in the pipeline of things that need to be fixed but probably won't be fixed before the network control features are at least somewhat working.

EXTERNAL SYNC
I've found that if you want to gate the shutter from a TTL signal or at least sync the start of exposure from an external clock, you need to set the internal framerate to be just slightly higher than the incoming signal, otherwise frames can drop. i.e. 1 kHz square wave on BNC cable, camera set at 1000 fps, some frames dropped, but if camera is expecting 1001 fps, no dropped frames.  Setting the internal frame rate too high above the external signal can also sometimes cause unreliable sync though.

Good to know detail. We generally use the gating from another camera for testing and of course the timing would always be close enough that it won't cause problems. Another side of this is it's generally a very good idea to change the frame time down on a slave camera before changing the master timing - this would also be true for external signals as we've seen the application crash once every now and then and haven't tracked down why yet - something timing-wise in the FPGA.


BUG REPORTS
i) The eject SD / eject USB in the Util menu are throwing "failed to eject" errors for me every time in v0.2.3.  The "safely eject" button in the Play:Settings screen works.

ii) Aborting save of RAW can cause the camera to crash on occasion (it goes to a shutting down screen and hangs)

iii) 12 bit packed RAW files are often truncated a couple frames short of the requested mark out point. Sometimes the files end halfway through a pixel! i.e. record 1000 frames, save all of it to 12 bit packed RAW, saving UI counts up to 1000 and done, and I end up with 991 frames and a bit of the next frame in my file. This is not a file size limit issue as far as I can tell.

iv) When saving, the counter over-runs the mark out point by 5-6 frames. If mark out is at the end of the memory, it will overflow and count back up to 5 or 6. Maybe connected to (iii) or could be aesthetic.

v) Warning about file size being too large for fat32 storage (above 4GB) is wrong when using 12 bit packed format - it shows the error once the file gets above about 3 GB. I think the calculation for the warning is assuming that the file being saved will be 16 bpp?

vi) When the total data saved to a fat32 drive reaches 8 GB, a save error occurs (sure...) but then the playback & live framebuffer displays go green and distorted on the left, and black on the right side, as if the software suddenly thinks my mono camera is a color model and is misinterpreting the data. Putting in a new empty media and redoing the Save causes the problem to go away.

vii) Sometimes when I boot, the camera has suddenly gone into Gated mode (1 frame!). I think this is related to the next bug.

viii) I use the external shutter gate option a lot. If I try and change resolutions and frame rates when this is enabled, bad things happen, which can be any of the following:
  • the display stops updating or becomes partially obscured by a block of static
  • the camera goes into Gated mode (1 frame) and refuses to accept a change of setting
  • camera can get stuck in a permanent recording loop (red led stuck on, record button does nothing)
  • camera can refuse to shut down and will still be misbehaving when rebooted!
This did not happen with software v0.2 and is a new bug.
Workaround: turn off external shutter gating before changing frame rates and resolutions. Once settings are good, then switch back to external gating.
When the bad things above happen, turn off external gating and hard reset the camera. I had to actually pull the power cord and battery and hard reboot for the problem to resolve.

Thanks - will check into these details. I'll make note on any that are specific.

iii) I haven't been able to read 12bpp data yet so didn't catch this. Will test and figure out what's causing that. For now it would be a good idea to always record a dozen extra frames. Please let me know if there are any frames missing from what you can tell. One of the other features in the pipe is overlay text and/or embedded data so it'll be easier to catch dropped frames. There should be none at this time based on the tests we've done of the other save modes.

iv) cosmetic if it's not dropping frames - the pipeline is async so may play a few more frames into the cpu-side buffer before everything's shut down. I'll note that to make a quick fix to hold the frame number and position if it's past the end but for now it's useful for seeing if there are dropped frames.

vi) raw recording still uses the 24bit interface by encoding two pixels side by side. In verilog this would be Pixel1={Blue[7:0],Green[7:4]}, Pixel2 = {Green[3:0],Red[7:0]}... Already fixed for next beta update, will get that out when there are a few more bugs fixed.

vii/viii) Gated mode, as noted above with the changing of resolution, has some edge cases in the FPGA side that can cause crashing of the pipeline in there. If you find anything that can reproduce this easily Please let us know, this has been a tricky one to track down.

Would it be acceptable to turn off live preview if not currently recording when in gated mode?


FEATURE REQUESTS
  • Partitioned memory mode is cool. I don't see support for being able to switch memory segments yet though, did I miss something?
  • Saving the whole RAM to RAW can be a slow affair. It would be super cool to have an option to turn on an auditory alert or screen flash when saving is done so I know to look up from my laptop!
  • I connected IO2 with frame sync output to an oscilloscope at 1Meg impedance and the signal was very funky until I pulled it down to 50 Ohms. In a future version of the camera, can you include a 50R pulldown resistor option for either IO2 or IO3?
  • Does the hardware support binning mode? i.e. read out each block of 2x2 pixels as a single virtual pixel with 4x the effective sensitivity, image is now 1/4 the size.
  • Can you modify the RAW formats to add a few bytes of header data? Ie pixels width and height, bits per pixel (for discriminating RGB and mono data files), and may be camera serial no? Most scientific cameras write data out this way. ImageJ / FIJI RAW reader by default assumes the first 1024 bytes of scientific camera RAW are header anyway.
  • I see the specs list two 1 MS/s ADCs. Is there any way we could use these for recording an arbitrary signal like a pressure or temperature sensor in sync with the movie? Are they wired to the mic jack?
  • Is Krontech willing to share who they buy their CMOS arrays from? I'm not going to go into competition  ;D , but often these details need to go into the methods sections of scientific journal papers.

Will work on those. The indicator when save complete is really good. As soon as audio is working I'll make sure that's one of the options.

The sensor does include a binning mode but we haven't set it up or tested it yet.

The ADC inputs are on the terminal connector along with trigger 2 and 3. Once we overlay or embedded data we can save the analog values there. It will be a bit as we haven't started implementing this part.

We are using Luxima sensors.

Dan D

  • Newbie
  • *
  • Posts: 19
    • View Profile
Re: Software V0.3 Beta
« Reply #17 on: December 21, 2017, 12:12:24 AM »
Cheers!
  • I'll see if I can reliably reproduce the external sync bugs when I get back to the lab in early Jan. I'm generating my TTL sync signal from a Beaglebone Black's PRU so I can pretty much make it do whatever I want. If you want me to test anything with external triggering, let me know.
  • I've found by looking at the sync output vs the gating input on a scope that there's a random delay between the shutter gate input rising edge and the output sync rising edge, it can be anywhere from about 200ns to 1.1us. It's not jitter, just a bias that changes depending on the framerate. The jitter is much smaller, more like 11 ns (I guess that's the clock rate). I suppose the delay is an artifact of the way the FPGA's handling the timing. When the light source is a microsecond strobe pulse, we have to delay it an extra microsecond or two to avoid the problem. For some science experiments, the time between subsequent exposures is super important and so knowing this delay is necessary. If we could reliably predict the exposure delay as a function of the camera's settings, that would be super cool.
  • The live preview is handy in externally gated mode as it immediately lets me know that my strobe light and exposure timing are aligned - as well as making sure my image is in focus. If it was disabled to improve stability, I'd have to rely on an oscilloscope signal from the frame sync output to tell the timing, and getting things focused up would be a lot more annoying - I'd have to change modes back and forth.
  • If audio support is a way off, flashing the screen for completed saving or putting a brightly coloured alert message on the screen would work.
  • Understood re. the speed of saving issue, binning, ADC signals etc.

Dan

Fyodor

  • Jr. Member
  • **
  • Posts: 60
    • View Profile
Re: Software V0.3 Beta
« Reply #18 on: December 21, 2017, 12:15:10 AM »
@ Dan D:
I'm really missing a "like" button in this forum!

NiNeff

  • Full Member
  • ***
  • Posts: 169
    • View Profile
Re: Software V0.3 Beta
« Reply #19 on: December 21, 2017, 11:37:30 AM »
Hey!
I just tried to update my camera and it did not seem to work, however I'm not sure what failed. As far as I can tell I'm still on the old 0.2. Version because I cant find any of the new features and the version dialog still claims to be 0.2 (and not 0.2.3 or 0.3 or whatever)

The upgrade process itself went exactly like described in the first post, no errors appeared. I'm using an 8GB USBstick, maybe it's too big? But the update was recognized by the camera and it did find and apply the update on the stick.

I know this is a really bad "bug" report as it contains virtually no info, so please tell me what to try and which information you need so I can report back. If needed, i will record the whole upgrade process using another camera ;)
« Last Edit: December 21, 2017, 11:39:36 AM by NiNeff »

BiduleOhm

  • Jr. Member
  • **
  • Posts: 85
    • View Profile
Re: Software V0.3 Beta
« Reply #20 on: December 21, 2017, 11:41:54 AM »
You've probably already tried but did you try to reboot the camera?

NiNeff

  • Full Member
  • ***
  • Posts: 169
    • View Profile
Re: Software V0.3 Beta
« Reply #21 on: December 21, 2017, 02:04:15 PM »
You've probably already tried but did you try to reboot the camera?
yes, it had no effect. I'm still on the old version.

patrickrebstock

  • Full Member
  • ***
  • Posts: 114
  • playing camera
    • View Profile
    • patrickrebstock.com
Re: Software V0.3 Beta
« Reply #22 on: December 21, 2017, 06:10:43 PM »
i am on the new firmware and the automaticly record and auto save after trigger are working!  stoked
i have run into a few issues where it wont save to card because it says there is not enough space but i checked the card and there was 100gigs of space, once removed sd card i could not get it to recognize again in the playback and settings menu with the refresh button, just had to loose the clip and reboot

Feature request, is there a way to set WB with Kelvin temperature rather than hitting a WB button with it facing at a white card
i would love to set it at 5,500 kelvin, there are so manny times that i shoot where i cant get a WB card to work, i shoot on very telephoto lenses 300mm and 800mm and its not practical to find something white 50 yards away, also the similar problem with my fisheye lens hard to get a WB

question does it save black balance settings across reboots?

Dan D

  • Newbie
  • *
  • Posts: 19
    • View Profile
Re: Software V0.3 Beta
« Reply #23 on: December 21, 2017, 06:23:04 PM »
i have run into a few issues where it wont save to card because it says there is not enough space but i checked the card and there was 100gigs of space, once removed sd card i could not get it to recognize again in the playback and settings menu with the refresh button, just had to loose the clip and reboot

Yeah, I've had the same issue with losing access to the SD card slot after an out of space error until I reboot. I think the SD drive fails to be properly unmounted so inserting a new SD card won't work because the mountpoint is 'busy'. The USB ports don't seem to have this problem - if you use an SD card reader in the USB port you may have better luck.

I think the out of space error is happening because (a) the software can't seem to write past the first 8 gb of a fat32 partition and (b) fat32 has a 4gb file size limit - you can hit that easily when saving RAW. I found a temporary workaround to be to format your drive for Linux ie. ext3.  You just have to install some 3rd party software on mac/windows to read the files off. Another way around might be to partition to the drive into lots of 8gb fat32 segments, which would probably give faster saving, although you'll still get hit with the 4gb file size limit and it'll be a pain to use.  Devs can correct this if I am mistaken.

patrickrebstock

  • Full Member
  • ***
  • Posts: 114
  • playing camera
    • View Profile
    • patrickrebstock.com
Re: Software V0.3 Beta
« Reply #24 on: December 22, 2017, 04:30:45 AM »
i have run into a few issues where it wont save to card because it says there is not enough space but i checked the card and there was 100gigs of space, once removed sd card i could not get it to recognize again in the playback and settings menu with the refresh button, just had to loose the clip and reboot

Yeah, I've had the same issue with losing access to the SD card slot after an out of space error until I reboot. I think the SD drive fails to be properly unmounted so inserting a new SD card won't work because the mountpoint is 'busy'. The USB ports don't seem to have this problem - if you use an SD card reader in the USB port you may have better luck.
hmm im not saving raw just h264

I think the out of space error is happening because (a) the software can't seem to write past the first 8 gb of a fat32 partition and (b) fat32 has a 4gb file size limit - you can hit that easily when saving RAW. I found a temporary workaround to be to format your drive for Linux ie. ext3.  You just have to install some 3rd party software on mac/windows to read the files off. Another way around might be to partition to the drive into lots of 8gb fat32 segments, which would probably give faster saving, although you'll still get hit with the 4gb file size limit and it'll be a pain to use.  Devs can correct this if I am mistaken.

tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: Software V0.3 Beta
« Reply #25 on: December 23, 2017, 01:31:51 AM »
So what does the "Partial Ethernet" mean in the change log?

That just means you can connect the camera to a network, and with some scripts, you can scp saved videos over the network to a remote server. Far from full Ethernet control but it's something for now.

John Delonghi

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: Software V0.3 Beta
« Reply #26 on: December 23, 2017, 08:38:41 AM »
I've had the out of space error when using the same SD card I usually put in the slot but put in a card reader in the USB interface. One reboot didn't fix it but a second did.

gahang

  • Newbie
  • *
  • Posts: 6
    • View Profile
Re: Software V0.3 Beta
« Reply #27 on: December 23, 2017, 11:41:01 PM »
So what does the "Partial Ethernet" mean in the change log?

That just means you can connect the camera to a network, and with some scripts, you can scp saved videos over the network to a remote server. Far from full Ethernet control but it's something for now.

That's still a usefull function. We have to control the camera remotely due to the X-ray radiation in our workshop. So can you please send me the scripts that is already in use?

gyppor

  • Newbie
  • *
  • Posts: 49
    • View Profile
Re: Software V0.3 Beta
« Reply #28 on: December 24, 2017, 03:43:11 PM »
I'm having a minor issue after the update - when I use very low resolutions, especially 336x96, I get one flashing horizontal brown line near the top of the frame and one flashing blue line near the middle. These show after a black calibration. I'm assuming this has to do with the RAM temperature - I sometimes had the flashing line issue in the past running previous firmware versions but it was always resolved by performing a black calibration. Now black cal doesn't eliminate the lines.

My camera is serial #18 running software version 0.3. I do have an edited resolutions file, I'm not sure if this would cause any issues?

Loial

  • Krontech
  • Newbie
  • *****
  • Posts: 24
    • View Profile
Re: Software V0.3 Beta
« Reply #29 on: December 24, 2017, 11:10:24 PM »
Due to forum problems please remove all spaces after slashes in the following before use.

Right now you can connect a usb-mini cable to the OTG port and connect it to a computer. It'll show up as a network device and offer DHCP for 192.168.12.x. It'll be at 192.168.12.1. You can then ssh into the camera with root and any password.

If you kill the camera app "killall camApp" you can then edit the settings file "/ Settings/ KronTech/ amApp.conf" and restart the camera application "/ etc/ init.d/ camera start".

By using Auto Record and Auto Save modes you can get the camera to automatically save a buffer to the SD card using an external trigger signal. When combined with the methods above you can get very rudimentary control and operation of the camera. Any settings that are not possible will default to another value so it's better to configure the camera in advance. Also there is no provision for blackcal or other functions which means that if you're on an arbitrary resolution that has not been calibrated yet, image quality will be poor.

The saved files will be under "/ media/ mmcblk1p1/ " and timestamps or file naming can be used to determine which one is the latest.

I don't have any scripts that do this directly, just one that can manipulate the files and download the latest and the like (we use this for final calibration, filesystem and os testing). When I'm back from Christmas I can get some of the source up.