Allow to slave devices execute ifup-post. #109

k0ste · 2017-07-20T05:23:22Z

Hello.
I find out that slave devices in bond LACP LAG can't be tuned via ifup-local. As I understand the exit after SLAVE key is for safety (ignore another options). So for save this behaviour I just added exec for ifup-post.

Work of this patch tested on various baremetal servers. For example:

#!/bin/bash
case "$1" in
ten0)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten1)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten2)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten3)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
vlan2)
sleep 1s
/usr/bin/irq-tune -j /etc/irq-tune/irq_static.json
/usr/bin/irq-tune -a -j /etc/irq-tune/irq_auto.json
;;
esac
exit 0

journal log:

-- Reboot --
Jun 18 16:17:40 ceph-osd0 systemd[1]: Starting LSB: Bring up/down networking...
Jun 18 16:17:40 ceph-osd0 network[623]: Bringing up loopback interface:  [  OK  ]
Jun 18 16:17:41 ceph-osd0 network[623]: Bringing up interface bond0:  Set ring params for ten0:
Jun 18 16:17:41 ceph-osd0 network[623]: Set features for ten0:
Jun 18 16:17:42 ceph-osd0 network[623]: Interrupt moderation for ten0:
Jun 18 16:17:42 ceph-osd0 network[623]: Set ring params for ten1:
Jun 18 16:17:43 ceph-osd0 network[623]: Set features for ten1:
Jun 18 16:17:43 ceph-osd0 network[623]: Interrupt moderation for ten1:
Jun 18 16:17:44 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:44 ceph-osd0 network[623]: Bringing up interface bond1:  Set ring params for ten2:
Jun 18 16:17:45 ceph-osd0 network[623]: Set features for ten2:
Jun 18 16:17:45 ceph-osd0 network[623]: Interrupt moderation for ten2:
Jun 18 16:17:46 ceph-osd0 network[623]: Set ring params for ten3:
Jun 18 16:17:46 ceph-osd0 network[623]: Set features for ten3:
Jun 18 16:17:47 ceph-osd0 network[623]: Interrupt moderation for ten3:
Jun 18 16:17:47 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:53 ceph-osd0 network[623]: Bringing up interface vlan2:  Set IRQ [megasas]: 27 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/27/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 28 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/28/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 29 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/29/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 30 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/30/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 31 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/31/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 33 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/33/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: ****** ERROR ****** The IRQ 34, for megasas does not exists
Jun 18 16:17:53 ceph-osd0 network[623]: ****** ERROR ****** The IRQ 35, for megasas does not exists
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-0]: 41 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/41/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-1]: 42 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/42/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-2]: 43 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/43/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-3]: 44 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/44/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-4]: 45 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/45/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-5]: 46 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/46/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-6]: 47 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/47/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-7]: 48 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/48/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-0]: 60 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/60/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-1]: 61 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/61/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-2]: 62 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/62/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-3]: 63 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/63/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-4]: 64 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/64/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-5]: 65 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/65/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-6]: 66 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/66/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-7]: 67 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/67/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-0]: 78 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/78/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-1]: 79 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/79/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-2]: 80 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/80/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-3]: 81 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/81/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-4]: 82 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/82/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-5]: 83 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/83/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-6]: 84 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/84/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-7]: 85 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/85/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-0]: 96 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/96/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-1]: 97 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/97/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-2]: 98 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/98/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-3]: 99 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/99/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-4]: 100 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/100/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-5]: 101 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/101/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-6]: 102 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/102/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-7]: 103 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/103/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:57 ceph-osd0 network[623]: Bringing up interface vlan3:  [  OK  ]
Jun 18 16:18:01 ceph-osd0 network[623]: Bringing up interface vlan4:  [  OK  ]
Jun 18 16:18:01 ceph-osd0 systemd[1]: Started LSB: Bring up/down networking.

deekej · 2017-07-24T16:00:15Z

Hello @k0ste,
we were looking into you problem description, but we do not exactly understand what's your issue...

I find out that slave devices in bond LACP LAG can't be tuned via ifup-local.

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you.

k0ste · 2017-07-25T00:56:36Z

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you.

I provided example ifup-local script and stdout of this script work ("proof of concept").

lnykryn · 2017-07-25T08:11:09Z

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you.
I provided example ifup-local script and stdout of this script work ("proof of concept").

Hmm, can't you set most of the bits from your example via ETHTOOL_OPTS?

The reason why we ask is that we would like to find a different way how to implement the behavior you want. Calling ifup-post here is not an option, it does much more than just calling ifup-local and this could definitely break someone else's setup. I would also like to avoid directly calling ifup-local here, because again I can imagine a use-case it could break.

k0ste · 2017-07-25T08:48:22Z

Hmm, can't you set most of the bits from your example via ETHTOOL_OPTS?

Nope, because sometimes (various of device) we need timeout between ethtool calls, when driver reloaded device.

If we try do this jobs on MASTER interface (i.e. bond0) with switches that unsupported LACP rate fast (like Cisco Catalyst 29**) - possible break of LAG, due timeouts without any slave.

With i40e is pretty fast:

[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: changing MTU from 1500 to 9000
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: already using mac address 3c:fd:fe:9f:ea:0c
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: Changing Tx descriptor count from 512 to 4096.
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: Changing Rx descriptor count from 512 to 4096
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: changing MTU from 1500 to 9000
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: set new mac address 3c:fd:fe:9f:ea:0c
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: Changing Tx descriptor count from 512 to 4096.
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: Changing Rx descriptor count from 512 to 4096

But with tg3 we need more time:

[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: EEE is disabled          
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: EEE is disabled          
[Sat Jul  8 18:35:17 2017] tg3 0000:03:00.0 eth0: speed changed to 0 for port eth0                                                                    
[Sat Jul  8 18:35:19 2017] tg3 0000:03:00.1 eth1: speed changed to 0 for port eth1                                                                    
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: EEE is disabled          
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: EEE is disabled

lnykryn · 2017-07-25T10:06:47Z

We already have ETHTOOL_DELAY which waits before calling ethtool, maybe we could also add it to the for loop. Something like:

diff --git a/sysconfig/network-scripts/network-functions b/sysconfig/network-scripts/network-functions
index d72901e..fbbb326 100644
--- a/sysconfig/network-scripts/network-functions
+++ b/sysconfig/network-scripts/network-functions
@@ -209,6 +209,7 @@ ethtool_set()
             /sbin/ethtool -s ${REALDEVICE} $opts
         fi
         IFS=';';
+        sleep ${ETHTOOL_DELAY_SEC}
     done
     IFS=$oldifs;
 }

lnykryn · 2017-07-25T10:26:30Z

But looking to your script, you do only one ethtool call per device, so the current ETHTOOL_DELAY should be working for you.

k0ste · 2017-07-25T12:36:47Z

But looking to your script, you do only one ethtool call per device, so the current ETHTOOL_DELAY should be working for you.

Why you ignore echo, iproute & traffic control commands?
If you do not configure your physical network interfaces or run hooks like apply htb and sfq queues, this does not mean that someone does not use such hacks, I'm just tired of apply this patch on my CentOS nodes.

lnykryn

Calling ifup-post there could break existing setups. We should come with a solution that would offer to set the necessary settings without using ifup-local script.

k0ste · 2017-11-15T09:44:17Z

I created issue #153, because don't see any movements about this PR.

deekej · 2017-11-20T12:42:42Z

Well, I currently do not have any solution which would satisfy your needs and wouldn't break any existing user setups. Do you? :)

k0ste · 2017-11-20T13:33:58Z

Well, I currently do not have any solution which would satisfy your needs and wouldn't break any existing user setups.

Please, explain for me: how it break existing user setups? May be I don't understand something and will search another solution.

deekej requested review from lnykryn and deekej July 24, 2017 16:00

deekej self-assigned this Jul 24, 2017

deekej added the question label Jul 24, 2017

lnykryn requested changes Jul 26, 2017

View reviewed changes

k0ste mentioned this pull request Nov 15, 2017

bonding: network-scipts not execute ifup-local on slaves physdev. #153

Open

Allow to slave devices execute ifup-post.

32abf8e

k0ste force-pushed the NewFeature branch from b8d1b0c to 32abf8e Compare June 8, 2018 08:18

jamacku force-pushed the master branch from 665b9e1 to f90b19b Compare September 30, 2021 14:36

mergify bot added Fedora network-scripts labels Jan 20, 2022

mergify bot removed the Fedora label Aug 8, 2022

mergify bot added the Fedora label Aug 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow to slave devices execute ifup-post. #109

Allow to slave devices execute ifup-post. #109

k0ste commented Jul 20, 2017

deekej commented Jul 24, 2017 •

edited

Loading

k0ste commented Jul 25, 2017 •

edited

Loading

lnykryn commented Jul 25, 2017 •

edited by deekej

Loading

k0ste commented Jul 25, 2017

lnykryn commented Jul 25, 2017

lnykryn commented Jul 25, 2017

k0ste commented Jul 25, 2017

lnykryn left a comment

k0ste commented Nov 15, 2017

deekej commented Nov 20, 2017

k0ste commented Nov 20, 2017

Allow to slave devices execute ifup-post. #109

Are you sure you want to change the base?

Allow to slave devices execute ifup-post. #109

Conversation

k0ste commented Jul 20, 2017

deekej commented Jul 24, 2017 • edited Loading

k0ste commented Jul 25, 2017 • edited Loading

lnykryn commented Jul 25, 2017 • edited by deekej Loading

k0ste commented Jul 25, 2017

lnykryn commented Jul 25, 2017

lnykryn commented Jul 25, 2017

k0ste commented Jul 25, 2017

lnykryn left a comment

Choose a reason for hiding this comment

k0ste commented Nov 15, 2017

deekej commented Nov 20, 2017

k0ste commented Nov 20, 2017

deekej commented Jul 24, 2017 •

edited

Loading

k0ste commented Jul 25, 2017 •

edited

Loading

lnykryn commented Jul 25, 2017 •

edited by deekej

Loading