Sporadic Ethernet Dropout Issue

[sum18223]

I've been using grblHAL on the Teensy 4.1 with the ethernet module to control a few CNC machines, and I've been experiencing some random dropouts where the light on the ethernet module goes out and the machines stop dead in their tracks. The only way to connect back to grblHAL is to power cycle the teensy (unplugging and plugging the USB back in, then the ethernet light comes back on). If the spindle is on while it drops out, it keeps spinning while disconnected.

I've probed the 5V power pin on the teensy with a scope and the power appears pretty stable.

I've tried to get to the bottom of it through considering whether it's an EMI or ESD issue. Made sure my motors/drivers are well grounded to the chassis and chassis to earth ground, and got shielded ethernet cabling with a grounded ethernet switch yet it still seems to happen sporadically.

I'm trying to isolate if there's anyway that the light would dropout on the ethernet module if there was anything code related (either grblHAL code or my streaming/controlling code from my PC), or whether the light dropping out can only be hardware related, as I'm not very knowledgeable on the Ethernet PHY's on the teensy and how that would interact with the signal light on the magjack connector.

One thing I do is I use the spindle PWN signal to control a separate axillary system, but I need to use it while homing an axis so I toggle the default enable inversion pin and do a hard reset (CTRL+X) on the grbl to refresh that (and enable the system(, and then home that axis (so that the system and homing run at the same time, as grblHAL disables the spindle during homing). When homing is complete I flip that bit and do a hard reset again. Could doing that cause any weird issues like this?

Coming here for any guidance or other thoughts on what may be causing it and if anything software related could be causing it.

Thank you!

[terjeio]

Sounds like the controller is crashing, due to EMI? What do you mean by "sporadically" - every few days or more often? And does it always happen when the machine is running gcode? Never when sitting idle?

unplugging and plugging the USB back in

Is the power supplied by a PC or a wall wart? If a PC, have you tried to run the controller via the USB connection?

grounded ethernet switch

Too many ground connections can sometimes be a problem as well, creating ground loops.

I'm not very knowledgeable on the Ethernet PHY's on the teensy and how that would interact with the signal light on the magjack connector.

I do not know how the indicator LEDs are controlled, but likely by the MCU ethernet peripheral? However, I do not belive the LED status is significant.

I do not believe it is software related since there are many users that connects the sender via ethernet, myself included. But one can never be sure. BTW which version of the firmware are you running? Built with the Web Builder?

[sum18223]

I have a handful of machines, sometimes they will run for a day or few days without any issues, but sometimes it will drop out back to back every few minutes and multiple times throughout the day. When one drops out, there seems to be coupling, where others may drop out at the same time. I've isolated combined ones to different power sources and still seem like coupling, so I suspect that's just dropout through ethernet coupling as opposed to power but I cant be sure. Measuring scope lines, doesn't look like I'm getting any power dips on the 5V lines to the teensy for power, so i doubt its the teensy getting dirty power and rebooting from a dip.

For power, I have my own breakouts where teensy power is connected to a 12V wall-wart stepped down and regulated to 5V, as well as some where I power direct from USB to a hub. Since I use ethernet connection, none are really connected direct to the PC through USB.

My original hunch was power/EMI or ground loops. I've taken a lot of precautions to try to avoid that and still have issues. It may still be it but wanted to see if there was anyway I could isolate out grbl itself. I doubt there's a crash dump or any way I can query previous state on reboot is there?

I have my own aggressive buffering sending protocol with byte counting implemented in python and don't use IOSender during operation since I have some automation involved around the machines. One thing to note is I increased the planning buffer size. Some machines have 256, others 512. With smaller sizes I had performance degradation on run speed. I've seen it causing issues in the past but seemingly teensyduino 1.54 and up should have solved that. I never get weird operation, it just drops out instantly and I cannot communicate/connect to the socket anymore.

Here is the console log when connecting with IO sender. I presume VER:1.1f.20220925: is the build. I program it with Arduino and Teensyduino 1.55 (am aware to avoid pre-1.54 for some ethernet timing issues).

GrblHAL 1.1f ['$' or '$HELP' for help]
[VER:1.1f.20220925:]
[OPT:VNMHSL,256,1024,3,0]
[AXS:3:XYZ]
[NEWOPT:ENUMS,RT+,HOME,ES,REBOOT,TC,SED,RTC,ETH]
[FIRMWARE:grblHAL]
[NVS STORAGE:*FLASH]
[DRIVER:iMXRT1062]
[DRIVER VERSION:220922]
[DRIVER OPTIONS:USB.2]
[BOARD:T41U5XBB]
[AUX IO:4,3,0,0]
[IP:169.254.89.103]
[NETCON:Telnet]
[G54:-210.597,-350086.343,-199.996]
[G55:0.000,0.000,0.000]
[G56:0.000,0.000,0.000]
[G57:0.000,0.000,0.000]
[G58:0.000,0.000,0.000]
[G59:0.000,0.000,0.000]
[G59.1:0.000,0.000,0.000]
[G59.2:0.000,0.000,0.000]
[G59.3:0.000,0.000,0.000]
[G28:0.000,0.000,0.000]
[G30:0.000,0.000,0.000]
[G92:0.000,0.000,0.000]
[HOME:0.000,0.000,0.000:0]
[TLO:0.000,0.000,0.000]
[PRB:0.000,0.000,0.000:0]

Thanks,
I appreciate your help!

[terjeio]

When one drops out, there seems to be coupling, where others may drop out at the same time.

This suggests to me that there is something externally that affects your setup. Hard to believe that code running on several boards should fail at more or less the same time.

I doubt there's a crash dump or any way I can query previous state on reboot is there?

Correct, there isn't. And I am not sure the controller has crashed completely - perhaps just the network stack is down?
One way to find out would be to toggle a pin and check if toggling stops on a hang. Adding a small plugin would allow that.

Again, does the hang happen when all controllers are running idle?

[sum18223]

I agree it's unlikely they'd crash at the same time, I wasn't sure if perhaps one going down could cause a network drop on others.

They don't crash when all controllers are running idle, or at least haven't noticed that. Sometimes the feed PWM keeps running once connection is lost but I imagine that that may still be the case if software crashed. Otherwise things tend to stop dead in their tracks.

My algorithm interfacing with grbl is something like this. I loop this over multiple pieces using automation and controlling a feed with the spindle PWM. I use the PWM as the step pulse for stepper motors for the feed. As previously mentioned, I swap the enable pin to turn on/off feed since I need it running during the homing routine, but am forced to do a hard reset (0x18) each time and wait for grbl boot to flip the "spindle off" settings. I also disconnect from the grbl and reconnect between each piece that I operate on.

Do you see any issues with using grbl this way since it seems like a bit of a hack around the existing functionality. I'm not sure if the constant hard resets and waiting for the boot message could pose issues since I'm not sure what is involved in hard reset at the low level and if it would affect networking.

1. Connect to Grbl Socket
2. Send($X\n)
3. Poll Y limit switch to see if cleared (material cleared) ('?/n' at 0.2s polling interval)
4. Enable Feed
   send('$33=' + str(math.floor(PWM_freq)) + '\n')
   send('$34=50\n') # Set on PWM
   send('$16=1\n') # Invert Spindle Enable Pin
   send('M5\n') # Turn Spindle "OFF" to track changes
   hard_reset{
   send(0x18)
   Wait_for_boot routine (0.15s polling interval waiting for "GrbL Help" wakeup message
   flush socket
   }
5. Home Axes
6. Disable Feed
   send('$34=0\n') # Set off PWM to 0
   send('$16=0\n') # Flip Enable Pin
   send('M5\n') # Turn Spindle "OFF" to track changes
   hard_reset
7. Stream G Code commands
8. Wait for Idle ('?/n' at 0.2s polling interval, waiting Idle response).
9. Disconnect grbl socket

Anywho, between the PWM still running sometimes, and my psuedo-code routine above, if nothing seems like known potential for a software related issue, I'll continue to look elsewhere on the hardware front. Just wanted to see if anything stood out on the SW side.

[terjeio]

They don't crash when all controllers are running idle, or at least haven't noticed that.

That would suggest interference arising from running the controllers is the source?
Noisy spindles (you do not have any running?), static from dust extraction, ...?

I wasn't sure if perhaps one going down could cause a network drop on others.

I believe that is unlikely, the switch should keep the different controllers separated. An idea to try a different switch?

I use the PWM as the step pulse for stepper motors for the feed.

That is a bit odd way of doing it, in a soon to be released version I will add a stepper type spindle that might be better suited.

I'm not sure if the constant hard resets and waiting for the boot message could pose issues since I'm not sure what is involved in hard reset at the low level and if it would affect networking.

Sending 0x18 is a soft reset, not a hard reset, so should not be a problem. A hard reset will drop the connection.