Is AF_XDP socket thread-safe

[ZENOTME]

Can I run multiple threads and call sendto() for a same AF_XDP socket concurrently?

[magnus-karlsson]

Yes, but there is no point in doing this. You will likely just slow down your application somewhat.

[ZENOTME]

Thanks for your reply and suggestions and sorry for reply late! I am considering separating the rx && tx task into two threads. I am creating two sockets(one for rx and one for tx) and assigning them to two threads.

// Init rx first
 status = xsk_socket__create_shared(
      &p->xsk, params->iface, params->iface_queue, params->pool->umem, &p->rxq,
      NULL, &p->umem_fq, &p->umem_cq, &params->xsk_cfg);

// Init rx
status = xsk_socket__create_shared(
      &p->xsk, params->iface, params->iface_queue, params->pool->umem, NULL,
      &p->txq, NULL, &p->umem_cq, &params->xsk_cfg);

But I get the error when I init 2 eth(which means that init 4 socket). And the errno is 22. It's happened at bind in xsk_socket__create_shared. I search errno 22 and seems it indicates that the socket is already bound to an address or the address is invalid.

RX Port 0 created successfully.
TX Port 0 created successfully.
RX Port 1 created successfully.
Fail to create xsk socket
TX Port 1 initialization failed.

1. Is the way I use xdp socket wrong?
2. I think another way to solve this is that init one socket and assign the rx_queue&&fill_queue and tx_queue&&complete_queue into different thread. Can this way work?

[magnus-karlsson]

It all depends on what exact combination of netdev and queue id you are trying to bind to in the two processes. There are two modes: one if they are both the same, and another one if they differ. Take a look at the documentation in Documentation/networking/af_xdp.rst in the kernel sources. For examples you can look at these two from the bpf-examples repo:

https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-example (when they are the same)
https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-forwarding (when they are different)

[ZENOTME]

Thanks for your help! I think what I try to do is that use two mode both. E.g. I have two veth: veth1, veth2, both of them have one queue 0. And I try to bind 4 sockets for them. I do this successfully by a little trick because seems the interface xsk_socket__create_shared and xsk_socket__create didn't support this scene well. Let me explain what problem I meet and the way I solve it:

1. I create a umem
2. I use xsk_socket__create_shared to bind the first socket(veth 1,queue 0) to this umem. It will use umem->fd as the fd of the first socket.
3. I use xsk_socket__create to bind the second socket(veth 1, queue 0) to this umem. It will call xsk_create__create_shared again. And it will bind set the sxdp_shared_umem_fd to umem->fd.. In this time, it's correct because as in doc, if we want to bind a second socket in the same dev and same queue, we need to set the sxdp_shared_umem_fd to the fd of first socket, and the umem->fd is the fd of first socket.
4. I use xsk_socket__create_shared to bind the third socket(veth 2,queue 0) to this umem. It also set sxdp_shared_umem_fd to the umem->fd(the fd of first socket). According to doc, it's correct.
5. I use xsk_socket__create to bind the fourth socket(veth 2, queue 0). Error happens here. Because it try to set sxdp_shared_umem_fd to the umem->fd(the fd of first socket). But in this time, we need to set the sxdp_shared_umem_fd as the fd of third socket. What we are doing is to let the fourth socket share umem with the third socket(same dev, same queue).

And seems the way I solve this is to modify the umem->fd before I call xsk_socket__create for the fourth socket. This way looks weird. Can we have an interface with a parameter to indicate the fd which sxdp_shared_umem_fd set?

[magnus-karlsson]

Thanks for your help! I think what I try to do is that use two mode both. E.g. I have two veth: veth1, veth2, both of them have one queue 0. And I try to bind 4 sockets for them. I do this successfully by a little trick because seems the interface xsk_socket__create_shared and xsk_socket__create didn't support this scene well. Let me explain what problem I meet and the way I solve it:

1. I create a umem
2. I use xsk_socket__create_shared to bind the first socket(veth 1,queue 0) to this umem. It will use umem->fd as the fd of the first socket.

No need to use the shared version here as there is nothing to share with.

3. I use xsk_socket__create to bind the second socket(veth 1, queue 0) to this umem. It will call xsk_create__create_shared again. And it will bind set the sxdp_shared_umem_fd to umem->fd.. In this time, it's correct because as in doc, if we want to bind a second socket in the same dev and same queue, we need to set the sxdp_shared_umem_fd to the fd of first socket, and the umem->fd is the fd of first socket.

It is correct that you do not need to use the shared version of the call here as that call is for the case when you share a umem between different netdevs and/or queue ids.

4. I use xsk_socket__create_shared to bind the third socket(veth 2,queue 0) to this umem. It also set sxdp_shared_umem_fd to the umem->fd(the fd of first socket). According to doc, it's correct.

Mixing the two modes is not supported by bind, but there is a trick you can apply. Create a new umem here, but have it overlap the first umem area perfectly. Then repeat step 2 and 3 but with veth2 and the new umem.

5. I use xsk_socket__create to bind the fourth socket(veth 2, queue 0). Error happens here. Because it try to set sxdp_shared_umem_fd to the umem->fd(the fd of first socket). But in this time, we need to set the sxdp_shared_umem_fd as the fd of third socket. What we are doing is to let the fourth socket share umem with the third socket(same dev, same queue).

And seems the way I solve this is to modify the umem->fd before I call xsk_socket__create for the fourth socket. This way looks weird. Can we have an interface with a parameter to indicate the fd which sxdp_shared_umem_fd set?

[ZENOTME]

Mixing the two modes is not supported by bind, but there is a trick you can apply. Create a new umem here, but have it overlap the first umem area perfectly. Then repeat step 2 and 3 but with veth2 and the new umem.

Thanks!

[ZENOTME]

Mixing the two modes is not supported by bind, but there is a trick you can apply. Create a new umem here, but have it overlap the first umem area perfectly. Then repeat step 2 and 3 but with veth2 and the new umem.

I'm curious about the difference between this way and the xsk_socket__create_shared? Seems I also can use this trick to share a memory between two socket for different device.

[magnus-karlsson]

The same restrictions apply. xsk_socket__create_shared() is just xsk_socket__create() with the ability to specify fill and completion rings.