Make a python version of the remote control API that works over rs232 #288
Replies: 21 comments 10 replies
-
|
Issues to be thought about, how to deal with the connection loops that are threads in Java. What test application should be used etc.. |
Beta Was this translation helpful? Give feedback.
-
|
If any python user of the library wants to have a go at implementing the protocol for python, I'd be very glad to help you understand the protocol and help out with Serial and Socket understanding.. |
Beta Was this translation helpful? Give feedback.
-
|
I will need to build a library for Dart (because of Flutter) and some time later also for Python (because I will most likely build a Home Assistant integration). So perhaps I could work on both. Is the remote control API identical when using RS232 vs TagValue? I will need TagValue support but ideally I could support both. |
Beta Was this translation helpful? Give feedback.
-
|
That would be great, I would provide as much help as I possibly could with that as it would be generally useful to many, there are four API implementations at the moment that you can look at:
All of the implementations use the same protocol and messaging, ethernet, RS232 and even the websocket version is just a thin websocket wrapper around the same protocol. It is best to assume that other protocols could be added later, and only process messages that have protocol 1 (TagVal). Probably the best starting point is: https://www.thecoderscorner.com/products/apps/embed-control/embed-control-tagval-wire-protocol/ and then take a look through the current implementations. I'd recommend maybe starting with some unit tests and a protocol converter in one of the languages that can serialize the messages. If you create a repo for it, or if you prefer to put it here I'll do my best to help out with it as well. |
Beta Was this translation helpful? Give feedback.
-
|
I think the easiest approach will be if I create the repos and later I will transfer ownership to you. From the start, there will be a lot of commits and I don't want to waste anyone's time on approving the work-in-progress code. Allright, I will start by looking at the existing code and we can discuss in this thread. Later, if it will be necessary, we can also chat through some other clients. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
How is it going on the library front, feel free to add me on any repo you create, I've done a few different ports of the API now, and I could probably help get something workable quite quickly. You can ping me using the contact form on thecoderscorner or PM me in the forum. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @davetcc I had busy month working on non-coding stuff. I have this in my mind and will definitely ask you when I hit on some block. |
Beta Was this translation helpful? Give feedback.
-
|
@davetcc what do you use for testing besides writing unit tests for the parser? I will need to test realtime communication between the library and the embedded server. Ideally I would use something like an embedded simulator on my local PC if there's anything like that. |
Beta Was this translation helpful? Give feedback.
-
|
For the embedded simulator I just use a menu generated from tcMenu for Raspberry Pi/Java and add Ethernet support. Runs fine on most desktops. There is even one ready built with instructions for running it in this repo see the embedded Java example. Just needs a JDK with JavaFX. I tend to use Liberica JDK. |
Beta Was this translation helpful? Give feedback.
-
|
Let me know if you have trouble building it and I’ll jpackage it to somewhere for now. |
Beta Was this translation helpful? Give feedback.
-
|
It struck me recently that if all you want is a UI for iOS and Android, another starting point may be embedCONTROL.js With a React.native front end, you could target both mobile devices very easily and the API is pretty much already written and working, at least for WebSockets. A full socket and Bluetooth implementation could follow for the native version. There is already a web front end working that works well with mobile, If you run the embeddedJavaExample in the tcMenu project the web UI is served from port 8080, if native bindings were used instead of webpage bindings it may do most of what you need. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Dave, I came to a similar idea to you. Just instead of embedCONTROL.js I would use Flutter. There is a library to embed JS code. I already started my work on the app, except it is not bound to actual data. Now I start in paralell working on the library. My guess is that it won't be that complex once I get the gist of it. https://www.youtube.com/watch?v=T_nFXBiCyZ4 By the way, does this protocol support embedding custom data/messages? Because I'd like to send additional information, like status messages. I have a status screen and I want users to know just by looking at the display what is the state of the device. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
From your description I see the implementations vary in supported functionality. In ideal case, what should the library support? I came to this so far:
It looks like the JS version supports just the WebSockets, while Java supports IP and serial connections, right? Also, it looks like Java supports both server and client. I assume it's used in the Raspberry Pi build, correct? How do you implement WebSockets on Arduino side? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
It's by far safe to say that the Java API is the most complete and most tested, if you were copying either into Dart, I would use the Java one, as you'll come out with more or less everything (WebSocket support, IP support, Serial support). The JS implementation at the moment only supports running in a browser, so only supports web sockets as that is all it can support in that context. However, I was thinking that a react native UI may not be that difficult from where it is. At the moment, the websocket support requires a Java server to run that talks the regular protocol to Arduino and runs a jetty web server. It was originally envisaged to run on a RPI or similar, and probably run as if it were a regular IoT component. See the menu in menu docs on the main site if you wanted more background on that. I tried an Arduino implementation that could serve up a react app and then run a websocket, but the HTTP requests needed were just too large, initially running into problems with AsyncWebServer, even trying writing my own, and whatever way required amounts of memory that were demanding even for ESP32 or STM32. I decided to park that idea for now, but you can see it's all checked into another experiment repo as it were. https://github.com/davetcc/TcMenuNetLayer |
Beta Was this translation helpful? Give feedback.
-
|
Oh, it makes sense. So the WebSockets are just the bridge to the Java API server. I plan to build my version on the Java one. From what I can already tell:
Having all those pieces, the Python/Dart libraries could support both client and server. At this point it probably won't make much sense as we're talking to Arduino/RPi, but perhaps there will be a usecase in the future. Also, it would help with testing having all bits and pieces together. I for sure could use a Python server. Something serving a simple menu with two entries. Python library could optionally support websockets as well. There is a library to provide support. But I'll start with what I need the most and then we'll go from there - a client supporting IP connections. I will keep updates in this thread. And once there will be something useful, we can discuss transfering the repository under you. |
Beta Was this translation helpful? Give feedback.
-
|
By the way, didn't you consider separating the API part from the codebase? Like a separate repository called for example |
Beta Was this translation helpful? Give feedback.
-
|
It kind of is still separate, even though it's in the main package at the moment, it builds into maven central just as the API: https://search.maven.org/search?q=g:com.thecoderscorner.tcmenu There are pluses and minuses to separating it, but the dart and other implementations would need their own repo anyway. Probably the best thing to do is drop me an email (from my profile there's an email option) and we can maybe catch up on the best way forward, for both this and i18n support. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good. By the way, I have almost complete data model ready. MenuTree is TBD. https://github.com/Lutemi/tcmenu-core-python/tree/main/tcmenu/domain/state |
Beta Was this translation helpful? Give feedback.
-
|
Should we move the conversation into discussions on this repo for now? Discord doesn't send notifications well, and anything relatively private that is not suited to the discussion, just email me. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, we can do that. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @davetcc , The Java version of embedCONTROL supports connections via serial and IP sockets. Do both use TagVal under the hood? Because when I look at the code of |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good. Basically now I have basic data structures working and tested and what's left is making it work with Python-specific modules for sockets and serial communication. Unlike your java code, Python version will be single-threaded but it will be using asyncio and coroutines. For usecases like ours where the code waits for network it's performant. The code is currently private but I can make it available to you. Perhaps we could move the code here as private for now. |
Beta Was this translation helpful? Give feedback.
-
|
By the way, I'd like to implement support for both client and server. It's true that most of the usecases will involve interacting with the embedded device but:
And with all of this in place, I think that it should be fairly easy to create an integration with Home Assistant. Basically people could integrate TcMenu devices into their smart homes. Bulk of the logic is already in the protocol we implement as part of the library. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed. The Java api already does this. Take a look at the embedded Java example that works with Raspberry PI. It handles both client and server. And can even serve up a react.is app. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed on home integration as well, that would be useful for many people. |
Beta Was this translation helpful? Give feedback.
-
|
I know Java API does that. The only thing I would make differently is the code organization. Sometimes it's hard to tell whether the code is for server-side or client-side. I think that guys from the PyModbus did a great job in that. So that's my reference code I'd like to mimic. They support Serial and Sockets. The only difference is that their go-to protocol is Modbus. I'm familiar with them because before I found your library, we were using Modbus for remote communication. I decided to opt TagVal because it's better integrated with your solution and I don't want to have two protocols in my code. By the way, it should be relatively easy to add WebSockets support as well. Do you think it makes sense? I know you implemented WebSockets just for the JavaScript API. |
Beta Was this translation helpful? Give feedback.
-
|
@davetcc you have mentioned previously that you'd like to be more involved into TcMenu Python library development. While we're ironing out the TcMenu 4.0 release, I will shift focus on this library once again. I re-published it to Github (from my private GitLab). Perhaps we could move it under your handle/organization now and continue the development here. While I'm the only developer, I prefer to develop right into master branch but if you get involved we might switch to using feature branches. Transition should be fairly easy. We'll just need to update GitHub Action badges and README. The current code is here: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, as soon as I've got 4.0 out of the way, I'll start taking a look at this. I did an interim commit to 4.0 the other day (both UI and Lib) and I think the I18N workflow is now much better, I've even started to put more tests around it now as I think it is getting more stable. |
Beta Was this translation helpful? Give feedback.
-
As it says on the tin, a Python API that will connect to and remotely control the menu library, just like it's Java counterpart
Beta Was this translation helpful? Give feedback.
All reactions