-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure files from stream match local write #20
Comments
This hopefully won't happen because its a fast and stable SSD, or a relatively fast and stable USB flash drive (depending on whether the latter is inserted), but its not impossible.
Yeah, I'd talked about this with David for a while but we never actually got around to implementing it. The idea is to run every N minutes and delete the oldest max(ceiling(2 * N * 60 s/minute * 0.022 GB/s * 1 GB/file - GB_FREE)) files, such that the internal drive serves as a circular buffer in case something happens to the Pi, its software or its drives. I suggest implement this on the Pi as a PipelineStep in a Brokkr pipeline that's set to run every N minutes, You could use Paramiko to execute the commands remotely on the sensor, but for a quick and dirty use case its simpler to just run the SSH commands directly with Checking if the sensor and Brokkr have the same files is not trivial; either implementation or performance cost wise, but you could read the trigger GPS times by striding every 22 000 000 bytes of the sensor's data files, reading that chunk, processing it into a header and comparing it with those captured by Brokkr. Or, you could do the same with CRC checksums, but that would require a lot more disk bandwith on Brokkr's end, either to store them or to retrieve them. If its missing, you could send an error message and/or copy just the missing data chunks into a special directory. However, I'm not sure that's worth it, since you can check the |
Brokkr watches the stream of data from the sensor and writes to a Pi-attached drive. The sensor also writes data. So, we have two places where data can be found. Typically:
We want to:
This will catch any case where
brokkr
fails to write data. Note that we don't need to worry about the converse.If it fills, the sensor goes into a reboot cycle. This can happen near daily, depending on the size of the USB drive.
The text was updated successfully, but these errors were encountered: