Author Topic: "Missing" the out point when saving.  (Read 40274 times)

nik282000

  • Full Member
  • ***
  • Posts: 104
    • View Profile
"Missing" the out point when saving.
« on: June 23, 2017, 01:08:45 PM »
When saving files I have had the camera miss the Mark Out point a couple of times.

I set my in and out, set 30fps for the output file frame rate and then hit save, the camera goes through its saving sequence but it passes the Mark Out point and if I am not paying attention it will wrap around back to the beginning of the recording (see image below). Aborting after the camera has passed the out point causes it to become unresponsive (I haven't caught it before wrapping around yet), powering off then on gets it going again. The saved file is unplayable in VLC  (I haven't tried any creative methods of extracting frames).

Has any one else seen this behavior?

Passed the Out and wrapped around: http://i.imgur.com/rjJjoKZ.jpg
Note the Current frame, Mark In and Mark Out.

tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: "Missing" the out point when saving.
« Reply #1 on: June 23, 2017, 11:41:00 PM »
That is known to happen when the card becomes completely full during save, and potentially if the card doesn't have enough write speed. It seems quite rare though, if you find a way to reliably duplicate it, let me know!

nik282000

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: "Missing" the out point when saving.
« Reply #2 on: June 24, 2017, 05:08:17 AM »
Awesome, thanks for the info. I am using a "SanDisk Ultra Fit 128GB USB 3.0 Flash Drive (SDCZ43-128G-G46)" which has been working well until now but I do have a bunch of non-Chonos crap on the disk as well. I'll see if clearing it out makes a difference.

tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca
Re: "Missing" the out point when saving.
« Reply #3 on: June 24, 2017, 11:50:58 AM »
I would try an SD card, I've experienced far more problems with USB drives than SD cards.

Camoit

  • Newbie
  • *
  • Posts: 46
    • View Profile
Re: "Missing" the out point when saving.
« Reply #4 on: June 24, 2017, 06:34:18 PM »
I would try an SD card, I've experienced far more problems with USB drives than SD cards.

Do you have a list of compatible SD cards yet, that you recommend?

tesla500

  • Krontech
  • Full Member
  • *****
  • Posts: 218
    • View Profile
    • krontech.ca

oakwhiz

  • Krontech
  • Newbie
  • *****
  • Posts: 11
    • View Profile
Re: "Missing" the out point when saving.
« Reply #6 on: September 07, 2017, 12:25:29 PM »
These cards have been working very well:

http://www.lexar.com/products/memory-cards/sd/Lexar-Professional-633x-SDHC-SDXC-UHS-I-cards.html#SKU=LSD32GCB1NL633

I'd like to second this, I have had very good experiences with these cards.

That is known to happen when the card becomes completely full during save, and potentially if the card doesn't have enough write speed. It seems quite rare though, if you find a way to reliably duplicate it, let me know!

I'd also like to elaborate on this, the technical reason for this issue seems to stem from the disk not being able to answer requests in time. Good quality SD cards are able to maintain a certain minimum constant I/O bandwidth, but lower quality SD cards operate in fast bursts followed by periods of inactivity (they probably have internal SRAM cache to make up for the fact that they have poor flash block erase performance.) If the disk blocks on I/O then two things happen: the encoder buffers start to become full, and a certain piece of code feeding frames to the encoder tends to get stuck in an infinite loop when frames are dropped. The encoder is synchronous, but we are working on a way to add flow control. This would enable the encoder is able to pause the flow of frames from RAM when the encoder is unable to push frames onto the disk.

nik282000

  • Full Member
  • ***
  • Posts: 104
    • View Profile
Re: "Missing" the out point when saving.
« Reply #7 on: September 07, 2017, 09:55:39 PM »
Thanks for the explanation, it covered my next question. The USB key I am using is USB 3 which I would have thought would be fast enough but it seems to only be 99 times out of 100 (still not bad). Is there a bottleneck in USB that the SD card interface lacks? I'm glad to hear that you guys are working on flow control for slower storage devices, thanks!

Electra

  • Jr. Member
  • **
  • Posts: 55
    • View Profile
Re: "Missing" the out point when saving.
« Reply #8 on: September 07, 2017, 11:02:36 PM »
As far as I know, the camera only has a USB 2.0 port. Without knowing internals of the processor, USB should have better throughput than SD,  However the big limitation seems to be the encoder and I seem to recall much over about 12Mb/s(I'm likely wrong.) won't make any difference. My 80Mb/s cards write no faster than 40Mb/s ones. 

USB drives have hugely different write speeds and it's not as well documented as on SD cards.  I've got old USB 2 drives that write faster than huge but slow USB 3 drives(They /read/ fast, but write is bad). There should be utilities/benchmarks out there to test them, but you'll need to find one for your platform.

oakwhiz

  • Krontech
  • Newbie
  • *****
  • Posts: 11
    • View Profile
Re: "Missing" the out point when saving.
« Reply #9 on: September 08, 2017, 01:56:55 AM »
Another manifestation of this issue: if your saved footage has a random or semi periodic stutter in it, your disk is too slow, and the encoder is skipping frames as a result.

My 80Mb/s cards write no faster than 40Mb/s ones. 

It's true that this is a limitation of the encoder. It may be technically possible for a lossless compression algorithm to surpass the encoder's speed, but at the cost of SD card free space. We are looking into various lossless compression options such as linear predictive coding over YUV color space data, or raw bayer data.

Electra

  • Jr. Member
  • **
  • Posts: 55
    • View Profile
Re: "Missing" the out point when saving.
« Reply #10 on: September 08, 2017, 02:21:33 AM »
In a previous post:  http://forum.krontech.ca/index.php?topic=26.msg164#msg164
Tesla_500: "The SD interface is limited to 48MHz @ 4-bit, so cards above 24MB/s are of no benefit. I've had good luck with normal class 10 cards (10MB/s). Higher speeds may be beneficial later when RAW saving is available, but there doesn't appear to be any benefit for compressed saving, except maybe if you turn the bitrate up very high."

So I made sure I got 40Mb/s + cards. (The common speed up from that point).

What I'm /really/ curious about when it comes to writing out raw data.. Is this RJ45 connector on the side that has some kind of high speed twisted pair interface that can handle rather a lot of data? :)  (Or even the esata interface in a pinch) Maybe directly off the sensor, trade off resolution/fps for external storage? :)   1000fps,  800x600(or lower).. But writing directly to disk, rather than the RAM buffer? :)
But that is just me dreaming for now.

Edit: I've said it to David and I'll say it here. It's very nice knowing /why/ each limitation exists.  In this case, it's 'because the interface maxes out at that  speed'. Not 'Oh, we thought you'd buy the next version if we capped it there'.  That it's a hardware limitation, not a software one. Makes each one, while sometimes frustrating, understandable.
« Last Edit: September 08, 2017, 02:40:09 AM by Electra »

oakwhiz

  • Krontech
  • Newbie
  • *****
  • Posts: 11
    • View Profile
Re: "Missing" the out point when saving.
« Reply #11 on: September 08, 2017, 02:41:46 AM »
In a previous post:  http://forum.krontech.ca/index.php?topic=26.msg164#msg164
Tesla_500: "The SD interface is limited to 48MHz @ 4-bit, so cards above 24MB/s are of no benefit. I've had good luck with normal class 10 cards (10MB/s). Higher speeds may be beneficial later when RAW saving is available, but there doesn't appear to be any benefit for compressed saving, except maybe if you turn the bitrate up very high."

So I made sure I got 40Mb/s + cards. (The common speed up from that point).

What I'm /really/ curious about when it comes to writing out raw data.. Is this RJ45 connector on the side that has some kind of high speed twisted pair interface that can handle rather a lot of data? :)  (Or even the esata interface in a pinch) Maybe directly off the sensor, trade off resolution/fps for external storage? :)   1000fps,  800x600(or lower).. But writing directly to disk, rather than the RAM buffer? :)
But that is just me dreaming for now.

I forgot about the speed limitation on the SD interface. I'd hazard a guess that the eSATA interface would be the fastest way to exfiltrate data from the camera, since it runs at 3 Gbps.