extend GNSSNTRIPClient to return source table to the main app

[SebKuzminsky]

Hi there, I'm in the process of writing a program with the following behavior:

1. Read the list of available NTRIP Servers from an NTRIP Caster.
2. Read the current position from a GNSS receiver.
3. Automatically select the nearest NTRIP Server.
4. Subscribe to RTCM3 correction data from the selected NTRIP Server & feed it to the GNSS receiver, read corrected position data from the GNSS receiver.

From my reading of the source code I think there is currently no way for a program to get the source table using GNSSNTRIPClient: if no mountpoint is supplied it fetches the source table and prints it to stdout, then exits. I can imagine getting around this problem by capturing and parsing stdout, but that seems clumsy and awkward.

I would like to discuss ways of extending GNSSNTRIPClient to enable the above use case.

I can think of two possibilities:

1. The GNSSNTRIPClient could fetch the NTRIP Server list (sourcetable) like it currently does, but then instead of printing it, parse it into some kind of pythonic data structure and place it in the output Queue for the main application to read. This is a relatively small change, but it has the downside that after fetching, parsing, and adding the sourcetable to the queue, the GNSSNTRIPClient has no more work to do - since it doesn't have a mountpoint, it can't subscribe to any RTCM3 correction data.
2. Alternatively, because fetching the NTRIP Server table from the NTRIP Caster is a pretty different kind of operation than subscribing to a continuous stream of RTCM3 data from an NTRIP Server, maybe we could add a new, non-threaded, blocking get_ntrip_server_list() function to the GNSSNTRIPClient API. When called, this function would make the HTTP request with the NTRIP Caster, parse the data as above, and return it to the calling function.

My preference is option 2. I welcome thoughts and feedback on these ideas. I'm happy to implement whatever is deemed most desirable.

[semuadmin]

Hi @SebKuzminsky,

So, as you've summised, retrieving the sourcetable and streaming the live RTCM3 correction data will always need to be separate operations, as the caster closes the HTTP connection after returning the sourcetable. Also, bear in mind that sourcetable retrieval doesn't generally require authentication, whereas streaming RTCM3 data generally does.

For example, the PyGPSCllient GUI utility uses a threaded GNSSNTRIPClient operation to populate a sourcetable Listbox from which the user can select the required mountpoint, but then needs to re-run that operation with a non-blank mountpoint to actually stream the correction data. It was intentionally designed this way.

To achieve the functionality your're after, you would - as you say - need to run the two separate operations consecutively, retaining the sourcetable list between operations and invoking the existing find_mp_distance helper function (with a blank mountpoint name) to locate the nearest mountpoint.

Of the options you list, 'option 2' would also be my preference, though I would be inclined to make both calls non-blocking to facilitate programmatic use by external applications.

However, I'm not 100% convinced a fully automated selection would be that useful in practice - some observations off the top of my head:

• I would see this as application code rather than utility library code.
• Not all mountpoints provide a usable lat/lon in the sourcetable, and those that do only generally provide one to 2dp (i.e. +/- a km or so). The exact location or 'Antenna Reference Point (ARP)' is contained in the RTCM 1005/1006 message in ECEF format.
• If the nearest mountpoint happens to be, say, 300km away, the correction data would be all but useless. You might want to include a configurable 'maximum range' parameter that would only automatically select a mountpoint if it was within a useful range (typically <= 30km).
• The sourcetable entries contain data that affects the subsequent streaming operation (e.g. whether a GGA position sentence is required; whether authentication and/or decryption/decompression is required, whether a fee is required, etc.) So, again, you would need to include some conditional processing to configure the streaming operation based on the automatically selected mountpoint entry. (FYI PyGPSClient implements a get_mp_info() helper function in its helpers.py which might be useful in this context).

But happy to discuss further - contributions always welcome :-).

[semuadmin]

Hi @SebKuzminsky ,

FYI I've been thinking about this and, as you say, "option 1" has the advantage of requiring just simple changes to the existing GNSSNTRIPClient._do_header() method, e.g:

    def _do_header(self, sock: socket, stopevent: Event, output: object) -> str:
        """
        THREADED
        Parse response header lines.

        :param socket sock: socket
        :param Event stopevent: stop event
        :return: return status or error message
        :rtype: str
        """

        stable = []
        data = True

        while data and not stopevent.is_set():
            try:
                data = sock.recv(DEFAULT_BUFSIZE)
                header_lines = data.decode(encoding="utf-8").split("\r\n")
                for line in header_lines:
                    # if sourcetable request, populate list
                    if True in [line.find(cd) > 0 for cd in HTTPERR]:  # HTTP 40x
                        self._do_log(line, VERBOSITY_MEDIUM, False)
                        return line
                    elif line.find("STR;") >= 0:  # sourcetable entry
                        strbits = line.split(";")
                        if strbits[0] == "STR":
                            strbits.pop(0)
                            stable.append(strbits)
                    elif line.find("ENDSOURCETABLE") >= 0:  # end of sourcetable
                        self._settings["sourcetable"] = stable
                        mp, dist = self._get_closest_mountpoint()
                        self._do_write(output, stable, (mp, dist)) # ************ NEW LINE HERE ***********
                        self._do_log("Complete sourcetable follows...\n")
                        for lines in self._settings["sourcetable"]:
                            self._do_log(lines, VERBOSITY_MEDIUM, False)
                        return "1"

            except UnicodeDecodeError:
                data = False

        return "0"

It then becomes a trivial matter for the calling application to read the sourcetable and closest mountpoint from the output queue and then invoke the GNSSNTRIPClient again using this mountpoint, e.g. something like this quick adaptation of the existing rtk_example.py example:

"""
pygnssutils - rtk_example_auto.py

Adaptation of rtk_example which 'auto-detects' the closest mountpoint.

Created on 27 Aug 2023

:author: semuadmin
:copyright: SEMU Consulting © 2022
:license: BSD 3-Clause
"""
# pylint: disable=invalid-name

from queue import Queue
from threading import Event
from time import sleep

from pygnssutils import VERBOSITY_LOW, GNSSNTRIPClient
from gnssapp import GNSSSkeletonApp

CONNECTED = 1

if __name__ == "__main__":
    # GNSS receiver serial port parameters - AMEND AS REQUIRED:
    SERIAL_PORT = "/dev/tty.usbmodem1301"
    BAUDRATE = 38400
    TIMEOUT = 10

    # NTRIP caster parameters - AMEND AS REQUIRED:
    # Ideally, mountpoint should be <30 km from location.
    IPPROT = "IPv4"  # or "IPv6"
    NTRIP_SERVER = "your ntrip server"
    NTRIP_PORT = 2101
    FLOWINFO = 0  # for IPv6
    SCOPEID = 0  # for IPv6
    MOUNTPOINT = "" # leave blank to retrieve sourcetable
    NTRIP_USER = "[email protected]"
    NTRIP_PASSWORD = "password"

    # NMEA GGA sentence status - AMEND AS REQUIRED:
    GGAMODE = 0  # use fixed reference position (0 = use live position)
    GGAINT = 60  # interval in seconds (-1 = do not send NMEA GGA sentences)
    # Fixed reference coordinates (only used when GGAMODE = 1) - AMEND AS REQUIRED:
    REFLAT = 51.176534
    REFLON = -2.15453
    REFALT = 40.8542
    REFSEP = 26.1743

    send_queue = Queue()
    sourcetable_queue = Queue()
    stop_event = Event()

    try:
        print(f"Starting GNSS reader/writer on {SERIAL_PORT} @ {BAUDRATE}...\n")
        with GNSSSkeletonApp(
            SERIAL_PORT,
            BAUDRATE,
            TIMEOUT,
            stopevent=stop_event,
            sendqueue=send_queue,
            idonly=True,
            enableubx=True,
            showhacc=True,
        ) as gna:
            gna.run()
            sleep(2)  # wait for receiver to output at least 1 navigation solution

            # ************* NEW LINES BELOW **************
            mountpoint = ""
            print(
                f"Retrieving closest mountpoint from {NTRIP_SERVER}:{NTRIP_PORT}...\n"
            )
            with GNSSNTRIPClient(gna, verbosity=VERBOSITY_LOW) as gnc:
                streaming = gnc.run(
                    server=NTRIP_SERVER,
                    port=NTRIP_PORT,
                    mountpoint=mountpoint,
                    ntripuser=NTRIP_USER,
                    ntrippassword=NTRIP_PASSWORD,
                    output=sourcetable_queue,
                )

                srt, (mountpoint, dist) = sourcetable_queue.get()
                print(f"\nClosest mountpoint is {mountpoint} which is {dist} km away\n")
            # ************* NEW LINES ABOVE **************

            print(
                f"Streaming RTCM3 data from {NTRIP_SERVER}:{NTRIP_PORT}/{mountpoint}...\n"
            )
            with GNSSNTRIPClient(gna, verbosity=VERBOSITY_LOW) as gnc:
                streaming = gnc.run(
                    server=NTRIP_SERVER,
                    port=NTRIP_PORT,
                    mountpoint=mountpoint,
                    ntripuser=NTRIP_USER,
                    ntrippassword=NTRIP_PASSWORD,
                    output=send_queue,
                )

                while (
                    streaming and not stop_event.is_set()
                ):  # run until user presses CTRL-C
                    sleep(1)
                sleep(1)

    except KeyboardInterrupt:
        stop_event.set()
        print("Terminated by user")

This would be a relatively trivial change to make and minimises duplication of code - any thoughts?

See for example https://github.com/semuconsulting/pygnssutils/tree/auto-mountpoint-detection

[SebKuzminsky]

rtk_example_auto does exactly what I want, thank you very much! Any chance of merging that into main?

[semuadmin]

It'll merge in release 1.0.14 shortly - just checking no negative impact on PyGPSClient.

FYI, some NTRIP Casters offer an 'out of the box' NEAR or VRS (Virtual Reference Station) capability which will automatically return a stream from the mountpoint closest to a $GPGGA location provided by the client. The find-closest-mountpoint functionality in GNSSNTRIPClient effectively replicates this capability for any Caster.

[SebKuzminsky]

Thanks @semuadmin, i was not aware of that feature. I'm very new to all this and I've only ever used rtk2go.com, after seeing them mentioned in someone's project online.

[semuadmin]

Version 1.0.14 now released. If you're happy with this response, could you flag it as 'Answered' please - much appreciated.

[SebKuzminsky]

I don't see how to mark a comment in this discussion as "Answered". I see a "Close discussion as resolved" option at the bottom, but i dont think that's what you're asking me to do...

Here I see that there's supposed to be a "Mark as Answer" button below each comment, but none of the comments in this thread have that button in my view.

[semuadmin]

No worries - I think you can only flag the top level reply, or something like that, but I'll close it as resolved.