Feature Request: Test Config with Sample Data

[StarGeezerPhil]

Can't remember if I suggested this previously, but it'd be great if we could use a small set of sample data to test and tweak configuration changes - as it's often a case of days/weeks of tweaking to find the right settings (and often wasting those nights where the settings are wrong).

I don't think this is possible using the stored images as they've already been processed?

So, possible approaches?

• Save the first night's RAW data on first run
• Caching RAW data from the previous X day/night to give us the opportunity to save this for testing (Config Option or in TImelapse Viewer: "Retain RAW data for testing")
• Select an small period's (10 minutes/hour) worth of this data to use in "Test Config" option - a preview area where we can process a small subset of RAW data with a new configuration.

I'm just investigating the new stretching functionality, but had terrible results initially - and know it'll be a week/two (weather-dependent) of tweaking to find the right settings... having even 10 minutes of (good) RAW data to test against would make this process much more pleasant and effective.

[StarGeezerPhil]

@aaronwmorris is it better to submit these as feature requests in issues or as general discussions here.

TL;dr: idea of saving a small subset of raw data (e.g. 30 minutes or an hour) for use in config optimisation - e.g. running different ADU settings as this is the part that I find takes days to get right whenever I make small changes to the equipment or position.

The {ip address}/indi-allsky/adu is useful, however - is it possible to add a column that shows the Average ADU within the mask/active detection area?

[aaronwmorris]

Either way of requesting a feature is fine.

Looking at the code, I believe the ADU and SQM reported are the values within the detection area (not globally for the image).

Even if a subset of data were saved, you would not be able to test different ADU settings because the ADU is used to calculate the exposure values.

[StarGeezerPhil]

I'm thinking more about the timelapse/startrail generation - as the generation is time-consuming.

E.g. with the sample data from your own location and conditions (personally I'd pick a 30/60 minute interval that goes from clear & dark to overcast, and possibly a neighbourhood floodlight light spill) then tweak the processing > Star Trails Max ADU, Star Trails Mask Threshold and Star Trails Pixel Threshold values.

I spent 4 days doing this the other week as you're looking at very fine adjustments and then ~20+ minutes for each generation to complete.

On this: I've also found that the task queue seems to act quite strange at times - not always adding the generation task, or seeming to add them in duplicate (but that may be the new aurora and smoke data)... in fact - just checking now (no generation should be running as it's middle of the day) and I have all these:

[aaronwmorris]

For the tasks, those are probably aurora/smoke related items. They may be generating an exception which is why the result is not being updated (or I forgot to add a result in some code path). If you see an uncaught exception in your logs, let me know.

For the startrails, you can use my test script ./testing/starTrails_max.py to test out different settings. I believe it takes most of the same settings. This is how I test out new code/options before integrating them into the main program.

[StarGeezerPhil]

Thanks Aaron - I'll take a look. Any rough details on how the script works?

I'm not spotting anything obvious in the logs, but will take a closer look.

[aaronwmorris]

The scripts have built in help. I just merged an update that includes the default values (but these should be roughly the same as the web defaults).

Basically, you just provide an output file name (jpg) and an input directory. The script will find all image files recursively and build a star trail. The input folder can be a single hours worth of images to make things run faster.

The test script is nice because it tells you if an image will be included or the specific reason an image was excluded.

[StarGeezerPhil]

Hi @aaronwmorris

Just getting round to testing this, can you tell me where I'm going wrong here:

EDIT: ignore me - sorry inputdir is just required at the end of the command 🤣

(indi-allsky) Pi@Pi:~/indi-allsky $ ./testing/starTrails_max.py --inputdir /var/www/html/allsky/images/startrail_test/ --output /var/www/html/allsky/images/startrail_test/startrail/ --latitude 50 --longitude -1
usage: starTrails_max.py [-h] --output OUTPUT [--max_adu MAX_ADU] [--mask_threshold MASK_THRESHOLD]
                         [--pixel_cutoff_threshold PIXEL_CUTOFF_THRESHOLD] --latitude LATITUDE --longitude LONGITUDE
                         [--sun_alt_threshold SUN_ALT_THRESHOLD] [--moon_alt_threshold MOON_ALT_THRESHOLD]
                         [--moon_phase_threshold MOON_PHASE_THRESHOLD]
                         inputdir
starTrails_max.py: error: unrecognized arguments: --inputdir

Also, having studied the image data from the SQL, just some ideas/thoughts:

• Due to the exposure changing, I'm finding that I have prime quality images that have far too high an ADU value (e.g. a jump to 60s exposure moves the ADU from the 80 range up to 110+, but the images are perfect)
• Could you combine star count AND ADU? e.g. some kind of customisable logic so we can specify to use images who have an ADU value < 80 OR (starcount > 200 AND ADU < 120), possibly with an absolute minimum starcount - the starcount actually looks like a more reliable measure here than the ADU value.

For example, sample data from last night:

• 363 images with a star count > 200
• only 181 images with an ADU < 81
• combining these into a SELECT, e.g. WHERE stars > 150 AND ((ADU < 78) OR (stars > 200 AND ADU < 90)) gives me a result of 361 images

[aaronwmorris]

To me, it seems that using the star count alone might just be a superior metric than ADU or even ADU+Stars.

[StarGeezerPhil]

Hi @aaronwmorris did you give this any further thought (using starcount for star trails)?

With bright skies here I don't seem to be able to find reliable ADU levels.

[aaronwmorris]

It is on my todo list.

[aaronwmorris]

With #1223 you are now able to use star counts for star trail generation.

[StarGeezerPhil]

I missed this!!! Can't wait to try it mate - it'll be night-and-day for my setup 🔥🔥🔥

Really appreciate your attentiveness to the community feedback @aaronwmorris - thank you!!

[aaronwmorris]

With #965 you can now test processing a FITS file saved in indi-allsky. Right now things are pretty basic, but you can test different stretching, contrast enhancement, and color adjustments.

You can test the latest FITS from the tools menu or you can process individual FITS from the image viewer page.

[StarGeezerPhil]

Sorry for the delay in coming back to you @aaronwmorris...

I updated, but can't access the new page with Internal Server Error:

Back to the script, I thought I had this all working, but with the update I'm also finding that the starTrails_max.py script isn't working as before. It's erroring out as below and also seems to be detecting my location incorrectly (I've changed my lat/long below a little to protect my location), but showing sun as up when it shouldn't - am I formatting one/both of these incorrectly?

(indi-allsky) pi@pi:~/indi-allsky $ ./testing/starTrails_max.py --output /var/www/html/allsky/images/startrail_test/test.jpg --latitude 54 --longitude -1.5 --max_adu 82 --mask_threshold 100 --pixel_cutoff_threshold 0.09 --moon_alt_threshold 91 /var/www/html/allsky/images/startrail_test
WARNING:root:Max ADU: 82
WARNING:root:Mask threshold: 100
WARNING:root:Mask threshold %: 0.1
WARNING:root:Latitude configured for 54
WARNING:root:Longitude configured for -1.5
WARNING:root:Sun altitude threshold: -15.0
WARNING:root:Moon altitude threshold: 91.0
WARNING:root:Moon phase threshold: 33.0
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test/10_22
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test/10_20
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test/10_23
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test/10_21
INFO:root:Searching for image files in /var/www/html/allsky/images/startrail_test/10_19
INFO:root:Reading file: /var/www/html/allsky/images/startrail_test/test.jpg
INFO:root:Generating mask based on SQM_ROI
WARNING:root:Using central ROI for ADU mask

Example 1:

INFO:root:Reading file: /var/www/html/pi/images/startrail_test/10_19/ccd1_20231010_195651.jpg
WARNING:root: Excluding image due to sun altitude: -14.9
INFO:root:Reading file: /var/www/html/pi/images/startrail_test/10_19/ccd1_20231010_195755.jpg
Traceback (most recent call last):
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 375, in <module>
    sg.main(args.output, args.inputdir)
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 151, in main
    self.processImage(filename_p, image)
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 215, in processImage
    m_avg = cv2.mean(image_gray, mask=self._sqm_mask)[0]
cv2.error: OpenCV(4.6.0) /io/opencv/modules/core/src/matrix_iterator.cpp:71: error: (-215:Assertion failed) A.size == arrays[i0]->size in function 'init'

Example 2:
Is there a flag to ignore the lat/long and just process all files fed to it as if they're eligible?

WARNING:root: Excluding image due to sun altitude: 35.1
INFO:root:Reading file: /var/www/html/allsky/images/startrail_test/10_20/ccd1_20231010_200005.jpg
Traceback (most recent call last):
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 375, in <module>
    sg.main(args.output, args.inputdir)
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 151, in main
    self.processImage(filename_p, image)
  File "/home/pi/indi-allsky/./testing/starTrails_max.py", line 215, in processImage
    m_avg = cv2.mean(image_gray, mask=self._sqm_mask)[0]
cv2.error: OpenCV(4.6.0) /io/opencv/modules/core/src/matrix_iterator.cpp:71: error: (-215:Assertion failed) A.size == arrays[i0]->size in function 'init'

[aaronwmorris]

The error you are seeing is probably due to having images or a mask with different dimensions.

For the FITS processing, you will have to configure your instances to save some FITS files (that are recorded in the DB).

[StarGeezerPhil]

There shouldn't be any mask in use - this is the manual script and in initialisation it says INFO:root:Generating mask based on SQM_ROI. But, the images all match correctly and the masks/darks etc are fully functional in the daily scheduled tasks.

So, does the new page only load if there are FITS files? Be useful to have a message displayed to say "no FITS files available to process" or similar.

Thanks Aaron

[aaronwmorris]

I think that image has different dimensions from the rest.

I have fixed the 500 error for the FITS processing.