Add new iio_block based API and IIOD async. protocol #846
Conversation
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
9 times, most recently
from
a21e3a8
to
6230f8c
Compare
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
2 times, most recently
from
a057579
to
ea83ce0
Compare
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
4 times, most recently
from
1b83a3f
to
012c99b
Compare
|
@catkira It depends what modifications you need. If your timestamps appear as a separate streamable channel, then it should work with latest master without changes. |
|
@catkira probably not a good idea to base your branch off this one, because I'm not done working on it and I will force-push some changes from time to time. Maybe a better idea would be to base your branch off the "dev" branch instead, which is where all new features are merged, and will never be force-pushed to. Just keep in mind that the API in the "dev" branch won't be stable until it's eventually merged into master. |
|
@catkira The "dev" branch is basically "libiio2" (which will actually be libiio 1.x, since libiio1 was 0.x). It will be merged into master right after we release libiio v0.24, which will most likely be the last one of the 0.x series. Then the development will continue in the master branch until we are happy enough with the code to make a libiio 1.x release. I don't think we will change anything in libiio 1.x regarding timestamps. They will simply appear as an extra channel, that you can enable (or not). |
|
@catkira Right now what's implemented and works, is timestamp reporting at the sample level, if your HDL and kernel driver support that. Then the timestamp values appear as a separate channel. What @tfcollins talks about in the link you pasted is supporting timestamps at the buffer level, where there is one timestamp value for a whole block of samples, which represents (for instance) the time at which the very first sample was read. This kind of meta-data is not supported yet, and it is unclear if we want to add ways to support it. If we have meta-data embedded into the samples stream, then we can still transport it all the way to the application, which then must know that the first few bytes of each buffer is the timestamp, maybe with an extra kernel attribute that specifies that there is a timestamp for this buffer. This shifts the responsability of handling the timestamp to the application, which isn't ideal. If we don't want to taint the samples stream with meta-data, then maybe the kernel could also somehow update a "timestamp" buffer attribute each time a block of samples is dequeued. This could be handled by libiio as a special attribute, and you'd get something like a We tend to see this problem as part of a bigger one: how should meta-data (in general) generated by the HDL or hardware be handled by the kernel, libiio, and the application? We don't really have an answer for that at the moment. |
|
@catkira so I didn't profile it yet to see exactly what is the new bottleneck, but I'm pretty sure it's the CPU. The data is copied by the CPU from the IIO buffer to the network socket yes, and it's the only time it's copied. At 81 MiB/s the CPU usage was very high. |
|
@catkira well I'm not testing on a Pluto ;) But on a ZCU706 with a FMCOMMS-3 board. |
Ah that makes sense. My AntSDR should arrive soon, then I can test over its GBit ethnernet interface. Can You share Your benchmark script? Or do You just stream to file for a while and then divide size/duration? |
|
@catkira I just use the --benchmark option of iio_rwdev. |
|
The The |
|
I think we should discuss the timestamp feature somewhere else, feel free to open an issue about it. |
yes, that makes sense. Its a bit too much spam here. I delete my posts here and open another issue when I have a working prototype. |
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
from
d9ece56
to
b3d2e9f
Compare
Add support for the new asynchronous binary (datagram-based) interface in iiod-client, using the iiod-responder library. This new interface should work much better, as commands can be sent sequentially without having to wait for the answers between each command. Signed-off-by: Paul Cercueil <[email protected]>
Buffer-specific attributes should be addressed as such and not as device attributes with a "buffer/" prefix. Signed-off-by: Paul Cercueil <[email protected]>
Add the required callbacks to support the new blocks-based API. Signed-off-by: Paul Cercueil <[email protected]>
Add support for the new DMABUF based interface that is in the process of being upstreamed to the Linux kernel. Signed-off-by: Paul Cercueil <[email protected]>
The old MMAP interface is not well suited to the new block-based API. It requires some gymnastics for the blocks allocation and deletion, and for keeping track of the dequeued blocks; but it is still possible to support it. Signed-off-by: Paul Cercueil <[email protected]>
Add support for the new iio_block based API. This requires a few new callbacks, which are simple wrappers around functions provided by the iiod-client code. Signed-off-by: Paul Cercueil <[email protected]>
Add support for the new iio_block based API. This requires a few new callbacks, which are simple wrappers around functions provided by the iiod-client code. Signed-off-by: Paul Cercueil <[email protected]>
Add support for the new iio_block based API. This requires a few new callbacks, which are simple wrappers around functions provided by the iiod-client code. Signed-off-by: Paul Cercueil <[email protected]>
Update all the tools to use the new iio_block and iio_stream APIs. Signed-off-by: Paul Cercueil <[email protected]>
The examples have a lot of duplicated code... The streaming part was moved to a new iiostream-common.c file to factorize a bit the code. Signed-off-by: Paul Cercueil <[email protected]>
Rework the code of IIOD around the use of get_mask(). Instead of allocating the binary mask within that function, allocate it outside and do proper bound checking. The get_mask() function is then just a ascii-to-binary mask translator. Signed-off-by: Paul Cercueil <[email protected]>
Update IIOD to use the new iio_buffer / iio_block based APIs, the new iio_channels_mask object, etc. Signed-off-by: Paul Cercueil <[email protected]>
Add support for the BINARY command, which enables the new asynchronous binary protocol in IIOD. This new protocol is based on datagrams, instead of commands sent in ASCII. The low-level functionality is implemented by the iiod-responder library. Signed-off-by: Paul Cercueil <[email protected]>
Instead of using the device data storage to store a pointer to the DevEntry, which is set when the device is opened and cleared when the device is closed, set the device data pointer to a iio_device_pdata structure, which will live as long as the context exists. This will later allow to add more device-specific fields. Signed-off-by: Paul Cercueil <[email protected]>
Instead of calling iio_device_set_kernel_buffers_count(), which will be removed soon, we store the new value for the number of kernel buffers into the device data structure. The next time a buffer from that device will be created, the new value will be used. Signed-off-by: Paul Cercueil <[email protected]>
Remove all the callbacks that are related to the old API, which will soon be removed completely. Signed-off-by: Paul Cercueil <[email protected]>
Remove all the callbacks that are related to the old API, which will soon be removed completely. Signed-off-by: Paul Cercueil <[email protected]>
Remove all the callbacks that are related to the old API, which will soon be removed completely. Signed-off-by: Paul Cercueil <[email protected]>
Remove all the callbacks that are related to the old API, which will soon be removed completely. Signed-off-by: Paul Cercueil <[email protected]>
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
from
b3d2e9f
to
ed0f70c
Compare
|
@catkira try again? It should be working now. |
Remove old API functions, that have been superseded by the new iio_block API. Signed-off-by: Paul Cercueil <[email protected]>
A new ZIP has been uploaded, that contains the libzstd library. Signed-off-by: Paul Cercueil <[email protected]>
Libzstd is not shipped by default in the Linux images. Make the CI script install it before building the Linux dependencies. Point Libiio's CMake script to the downloaded libzstd on Windows since it can't be auto-detected. Signed-off-by: Paul Cercueil <[email protected]>
Update the Python binding to use the latest libiio 1.x API. It's been modified heavily, and is therefore not compatible with the libiio 0.x Python bindings. Signed-off-by: Paul Cercueil <[email protected]>
Update the C# binding to use the latest libiio 1.x API. It's been modified heavily, and is therefore not compatible with the libiio 0.x C# bindings. Signed-off-by: Paul Cercueil <[email protected]>
pcercuei
force-pushed
the
pcercuei/new-blocks-api
branch
from
ed0f70c
to
bbaf24a
Compare
This PR contains my latest work on the new Libiio 1.x API.
The API changed quite a bit, to be more versatile, to map better to the latest IIO kernel APIs, and to enable more performance. Along with the API change, the communication protocol between libiio and IIOD changed; it will now optionally use an asynchronous protocol (if supported by both parties) to boost the throughput.
For instance, running iio_readdev (iio_rwdev on the "dev" branch) with blocks of 16 KiB each, over a direct network cable between my PC and a ZCU102 equipped with a FMCOMMS-3:
new libiio + new iiod: 81 MiB/s sustained
Any other configuration: 44 MiB/s sustained
The diff is huge so take your time to review.