Questions about snappergps-backend processing time

[yuyeguiji]

I deployed snappergps-app and snappergps-backend on my server. The hardware configuration of the server is 16 cores, the memory is 128 GiB, and the system is ubuntu 20.04. snappergps-backend installed mkl-fft, the test data I uploaded has 880 points, and it took about 25 minutes to process, with an average of 1.7 seconds per point. I want to know if this processing speed is normal, if it is normal processing speed, is there any way to increase the processing speed? (P.S. I increased the parallel processing setting option, --max-batch-size 100, the processing speed did not increase, but the memory usage became higher)

[JonasBchrt]

TL;DR: The magnitude of the processing time seems correct.

In practice, I'd expect processing times of roughly 1 second per fix with this software configuration. However, this depends on the uploaded dataset:

• A time consuming step is initially estimating the frequency offset of the RF front-end. This is done for every dataset once. So, the overall average processing time is higher for short dataset and shorter for long datasets since the relative importance of this offset decreases.
• If sky visibility is good at the start of the dataset, then this frequency-offset estimation concludes in as little as 5 steps. If sky visibility is bad, it can take up to 100 steps, potentially increasing the overall processing time significantly. (In theory, you could reduce the time-out in backend.py here.)
• For the position estimation, the code uses two algorithms: first, it tries least-squares. If that does not give a good result for a snapshot, then it will try the more expensive MLE for the same snapshot. (See our ACM SenSys paper for algorithm details.) The means that a simple dataset will run faster (because only least-squares is used) while a more challenging dataset with less sky visibility / cheaper antenna ... will run slower (because MLE will run often).
• For the very first upload after starting the program, additional time is needed to load the Earth elevation model. The second, third etc. upload will, therefore, run faster.

In theory, you could do a few parameter changes if you are very keen on speeding up the processing:

• Turn off MLE here. Effect on processing time is likely gonna be small.
• Use less GNSS. By default, GPS (G), Galileo (E), and BeiDou (C) are used. Removing BeiDou will speed up things a lot, removing Galileo a little. Obviously, performance will go down, see our ACM SenSys paper for a comparison. You would need to remove the systems maybe here and definitely here.
• Decrease the frequency search space or the frequency resolution during satellite acquisition, either in the offset estimation or in the actual acquisition. Impact of the latter on speed can be big, but you really need to evaluate if the changes work for your hardware and deployment durations and temperatures. If all your deployments will be relatively short with small-ish temperature changes, this could be a route.

In practice, parallelisation is probably a more successful strategy if you do not want to harm positioning accuracy/reliability. You have a very low CPU usage since your are running only a single processing node. With 16 cores you could, in theory, process 16 uploads in parallel. If 25 min for one dataset does not sound great, maybe 25 min for 16 datasets sounds better. (RAM is likely be an issue if you really wanna run close to 16 processing nodes, though. You may want to play with the --max-batch-size flag or set use_local_interpolator = True here).

Side note: There's a release-candidate branch that I'm likely gonna merge soon. Shouldn't have a big impact on processing time, though, but improve performance slightly for longer deployments.

[JonasBchrt]

Oh, and putting the FFTs of satellite acquisition on a GPU also speeds up things. However, I discontinued support for this in the code. So you'd need to drag in your favourite GPU library (e.g., cupy) instead of the FFTs from numpy, scipy, mkl etc. yourself. And need a server with a GPU.

[yuyeguiji]

Thank you for such a detailed reply, this may be a good idea, I am going to rebuild the hardware environment for testing. I'm not very familiar with the functions that need to be replaced. I'm going to just replace the Numpy functions with the CuPy functions for testing, which may be easier.

[JonasBchrt]

Hi @yuyeguiji
I am currently running a survey among SnapperGPS users to figure out what exactly people are interested in and how I could improve. Maybe you would be willing to give me some structured feedback? https://forms.office.com/e/qkLgDVXdmM Shouldn't take long (5-8 min). And almost all questions are optional - any amount of input is fine and potentially helpful.