VPP memif #2005

lebauce · 2019-09-17T14:50:59Z

No description provided.

lebauce · 2019-09-26T11:19:03Z

run skydive-cdd-overview-tests
run skydive-scale-tests

lebauce · 2019-09-26T11:41:01Z

run skydive-functional-tests-backend-orientdb

lebauce · 2019-09-26T13:13:25Z

run skydive-cdd-overview-tests
run skydive-functional-tests-backend-orientdb

lebauce · 2019-10-04T15:51:41Z

run skydive-go-fmt

lebauce · 2019-10-04T15:55:34Z

run skydive-go-fmt

lebauce · 2019-10-04T17:25:05Z

run skydive-cdd-overview-tests

lebauce · 2019-10-04T22:00:57Z

run skydive-functional-hw-tests

…ker container Signed-off-by: Filip Gschwandtner <[email protected]>

lebauce · 2019-10-07T14:07:42Z

run skydive-scale-tests

lebauce · 2019-10-07T14:27:49Z

run skydive-scale-tests

lebauce · 2019-10-14T11:29:25Z

@fgschwan We recently merged the support for seeds in Skydive : it's a single Skydive probe executed as a standalone process in the Docker namespace.
This PR modifies the VPP existing probe to be able to run inside the namespace, using the socket interface. I integrated the memifs retrieval that you implemented and the links between them. I also cherry-picked the tests from your PR.
It would be very nice if you could give it a try.

fgschwan · 2019-10-16T14:08:52Z

@fgschwan We recently merged the support for seeds in Skydive : it's a single Skydive probe executed as a standalone process in the Docker namespace.
This PR modifies the VPP existing probe to be able to run inside the namespace, using the socket interface. I integrated the memifs retrieval that you implemented and the links between them. I also cherry-picked the tests from your PR.
It would be very nice if you could give it a try.

@lebauce
Hi, i'm currently quite busy with other things, but my colleague Milan Lenco could take a look at this PR and try to use it with running VPPs inside docker containers.

milanlenco · 2019-10-17T17:45:33Z

scripts/ci/jobs/jobs.yml

@@ -374,7 +374,7 @@
 builders:
 - skydive-cleanup
 - skydive-test:
- test: BACKEND=elasticsearch ARGS="-ovs.oflow.native" scripts/ci/run-functional-tests.sh
+ test: BACKEND=elasticsearch WITH_DOCKER_VPP=true ARGS="-ovs.oflow.native" scripts/ci/run-functional-tests.sh


WITH_DOCKER_VPP=true can be removed, not used in this PR

milanlenco · 2019-10-17T18:18:30Z

topology/probes/vpp/vpp.go

@@ -2,6 +2,7 @@

 //go:generate go run git.fd.io/govpp.git/cmd/binapi-generator --input-file=/usr/share/vpp/api/interface.api.json --output-dir=./bin_api
 //go:generate go run git.fd.io/govpp.git/cmd/binapi-generator --input-file=/usr/share/vpp/api/vpe.api.json --output-dir=./bin_api
+//go:generate go run git.fd.io/govpp.git/cmd/binapi-generator --input-file=/usr/share/vpp/api/memif.api.json --output-dir=./bin_api



The major issue with the design of VPP binary APIs, is that practically any change breaks backward compatibility. The binary API files generated in this PR are I suppose for 19.04 VPP. I tried to run skydive against container with VPP 19.08 and got some expected errors, such as:

skydive-agent_1 | 2019-10-17T10:09:29.960Z WARN vpp/vpp.go:649 (*Probe).Fire vpp-93026bb9-358b-4bff-414b-6c852edd130b: time="2019-10-17T10:09:29Z" level=warning msg="Unknown message received, ID: 829" skydive-agent_1 | 2019-10-17T10:09:27.753Z ERROR probes/wrapper.go:52 (*serviceManager).Start.func1 vpp-a31cbfb5-a85f-4503-5ad3-9b5225f85436: no reply received within the timeout period 1s

This is because any change in binary API message changes its CRC which will cause it to be refused and dropped on either VPP or GoVPP side. Similary, a newly added or removed message can shift sequence IDs of existing messages, once again breaking compatibility between client and VPP.
Although you have added the .vppbinapi makefile target that allows to build skydive for a custom VPP version, a single skydive binary will always be limited to only one version of VPP and might not interpret network containing multiple different containerized VPPs properly. Unfortunately, the interface.api that you use to dump interfaces changes pretty much with every release. For example, SwInterfaceDump structure has been edited (interface name is string, not array of bytes) in 19.08 and probes/vpp/vpp.go needs to be edited to reflect the change for it to compile. Actually, between 19.04 and 19.08 even the API language changed a bit (string type was introduced) and govpp had to be adjusted. Therefore to generate APIs for 19.08 you would also need to update the govpp dependency first. To make this even worse, in 19.08 the path to installed *.api.json files has changed - core APIs and plugin APIs are further separated to core and plugins sub-directories.

What we do in ligato/vpp-agent to mitigate this issue, is that we hide interaction with VPP behind an interface. Then we provide implementation of that interface for the latest VPP version (top of the master branch, frequently updated) and for the last 2 releases (for example: https://github.com/ligato/vpp-agent/tree/master/plugins/vpp/ifplugin/vppcalls). Each of these implementations has its own generated binary APIs corresponding to its version. For example in this file we define APIs for managing VPP interfaces. Then every implementation of this API registers itself into this map. When connection to a VPP instance is made, CheckCompatibility method is used to find and select that implementation which is compatible with the given VPP. In your case you could define API with methods like FetchMemifs(), FetchInterfaces(), SubscribeInterfaceEvents and define your own structures for interface metadata with fields that you actually use.

If support of multiple VPP versions is not a priority right now, then maybe at least document which version is supported out-of-the-box right now and perhaps allow to switch the .vppbinapi makefile target off, so that skydive can be compiled with binary APIs already generated and included in the repo, without having to have VPP installed in the host.

milanlenco · 2019-10-17T18:25:47Z

topology/probes/vpp/vpp.go

-func (p *Probe) interfacesEvents() {
- defer p.wg.Done()
+type memifSocket struct {
+ *memif.MemifDetails


This is an example where interface configuration is defined be directly re-using structures from the VPP binary API. Since that changes quite often, it might not compile if skydive is build against a different VPP version.

milanlenco · 2019-10-17T18:31:13Z

topology/probes/vpp/vpp.go

+ return nil, err
+ }
+
+ sharedMemifSockets[msg2.SocketID].MemifDetails = msg2


Actually there can be multiple (listener) memifs (on single VPP) that use the same socket. Master-Slave memif connection is created not only when the same socket file is used, but also when IDs (field ID) are the same.
In other words there can be multiple listeners with different IDs using the same socket and based on the ID of the connecting memif the match is made.

milanlenco · 2019-10-17T18:32:35Z

topology/probes/vpp/vpp.go

+func (p *Probe) GetABLinks(node *graph.Node) (edges []*graph.Edge) {
+ socketFilename, _ := node.GetFieldString("VPP.SocketFilename")
+ socketFilename, _ = p.getHostSocketFilename(socketFilename, node)
+ nodes, _ := p.slaveMemifs.Get(socketFilename)


Use also IDs to determine the links between memifs.

milanlenco · 2019-10-17T18:35:50Z

topology/probes/vpp/vpp.go

- conn, err := govpp.Connect(p.shm)
+// Do starts the VPP probe and get all interfaces
+func (p *Probe) Do(ctx context.Context, wg *sync.WaitGroup) error {
+ conn, err := govpp.Connect(p.addr)


I'm getting lot's of these error messages for the host and containers that do not run any VPP:

skydive-agent_1 | 2019-10-17T12:23:27.844Z ERROR probes/wrapper.go:52 (*serviceManager).Start.func1 dev-VirtualBox: VPP connection error: VPP API socket file /run/vpp/api.sock does not exist skydive-agent_1 | 2019-10-17T12:23:28.016Z ERROR probes/wrapper.go:52 (*serviceManager).Start.func1 vpp-9d93ad24-9cc6-4d85-584e-c333eb1777ae: VPP connection error: VPP API socket file /run/vpp/api.sock does not exist

Perhaps Do() method could start by checking if the socket files exists or if a process named vpp exists and continue only if it is the case to avoid polluting the log.

milanlenco · 2019-10-17T18:40:25Z

topology/probes/vpp/vpp.go

+ msg2 := &memif.MemifDetails{}
+ reqCtx = ch.SendMultiRequest(req2)
+ for {
+ if lastReply, err := reqCtx.ReceiveReply(msg2); lastReply {


there is a bug: msg2 := &memif.MemifDetails{} should be inside the for loop just before ReceiveReply(). Otherwise it is the same struct instance that is being rewritten with each iteration and all the memifs will end up in sharedMemifSockets with the same configuration of the last memif. (i.e. this issue occurs if VPP has multiple memifs)

milanlenco · 2019-10-17T18:47:17Z

topology/probes/vpp/vpp.go

+ }
+ }
+ }
+


Btw, the original Filip's PR also contained a support for AF-PACKET interfaces. If it is of an interest for you, this is how you can tell if the given interface is AF_PACKET and this is how to obtain the name of the Linux interface to which it is attached.

milanlenco · 2019-10-17T19:03:30Z

I was able to get it running and overall it looks good, only added few comments.

Also one general comment: If it is not already done/planned, I would also welcome if it was documented how to build and use vpp probe. For example mention that WITH_VPP=true has to be defined, that vpp probe is not loaded by default and needs to be listed in agent.topology.probes config and also document which version of VPP is supported out-of-the-box (for which version the binary API files have been generated and included in the repo).

lebauce · 2019-10-18T07:46:12Z

@milanlenco Thanks a lot for taking the time to test and for the review. This is really useful, I will try to address the issues

lebauce added the wip label Sep 17, 2019

lebauce force-pushed the vpp-memif branch 19 times, most recently from 9aa2091 to f98b78c Compare September 20, 2019 13:17

lebauce force-pushed the vpp-memif branch from f98b78c to a743de8 Compare September 26, 2019 08:19

lebauce removed the wip label Sep 26, 2019

lebauce force-pushed the vpp-memif branch 5 times, most recently from 423357b to 4de13bf Compare September 26, 2019 10:48

lebauce force-pushed the vpp-memif branch from 39cea0e to a3f4f31 Compare October 4, 2019 17:09

lebauce requested review from nplanel and safchain October 7, 2019 12:19

lebauce and others added 15 commits October 7, 2019 15:42

vpp: bump govpp library for UNIX socket support

3bb93df

common: support all namespace types

d361b77

vpp: properly disconnect from VPP

250ded0

vpp: use an indexer to keep track of created interfaces

846fed7

docker: report containers bind mounts

de5ca88

vpp: make use of retry probe wrapper

e040e3b

nsm: generate easyjson for metadata

0acc97a

graffiti: separate index management from graph events handling

5fddaef

topology: add GetOwner method

c196327

vpp: run seeds in containers and handle memifs

95adbec

test: added functional tests for topology probe of VPP running in doc…

1c25389

…ker container Signed-off-by: Filip Gschwandtner <[email protected]>

tests: build skydive executable to run VPP tests

009060a

graffiti: fix races when a subscriber with a filter connects

cb3aa78

docker: use pointer for metadata

948f49a

graffiti: ensure indexer do not return nil nodes

5685f7d

lebauce force-pushed the vpp-memif branch from a3f4f31 to 5685f7d Compare October 7, 2019 13:43

milanlenco reviewed Oct 17, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VPP memif #2005

VPP memif #2005

lebauce commented Sep 17, 2019 •

edited

lebauce commented Sep 26, 2019

lebauce commented Sep 26, 2019

lebauce commented Sep 26, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 7, 2019

lebauce commented Oct 7, 2019

lebauce commented Oct 14, 2019

fgschwan commented Oct 16, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco Oct 17, 2019

milanlenco commented Oct 17, 2019

lebauce commented Oct 18, 2019

VPP memif #2005

Are you sure you want to change the base?

VPP memif #2005

Conversation

lebauce commented Sep 17, 2019 • edited

lebauce commented Sep 26, 2019

lebauce commented Sep 26, 2019

lebauce commented Sep 26, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 4, 2019

lebauce commented Oct 7, 2019

lebauce commented Oct 7, 2019

lebauce commented Oct 14, 2019

fgschwan commented Oct 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milanlenco commented Oct 17, 2019

lebauce commented Oct 18, 2019

lebauce commented Sep 17, 2019 •

edited