Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Q: back-end scenario and file contents (not just paths) #1447

Open
4ntoine opened this issue Sep 13, 2021 · 7 comments · May be fixed by #1536
Open

Q: back-end scenario and file contents (not just paths) #1447

4ntoine opened this issue Sep 13, 2021 · 7 comments · May be fixed by #1536
Assignees
Labels
topic: gRPC Related to the gRPC interface type: support OT: Request for help using the project

Comments

@4ntoine
Copy link
Contributor

4ntoine commented Sep 13, 2021

Question

I've asked on SO but did not get any answers unfortunately.
Sorry for cross posting here.

Is there any opportunity to get new files content, compiled binaries content done vis cli vis gRPC calls?

Current behavior

All the methods return arduino cli host local fs paths. This limits the use cases and require direct access
to cli host file system or alternative services to provide access (remote fs, ssh, etc).

Expected behavior

Expected to have an opportunity to pass the content of the files, not just paths. This could be done by just returning oneof depending on the (new) request arguments (paths/content).

Environment

  • CLI version (output of arduino-cli version): 0.19.0
  • OS and platform:

Additional context

How does it work with "arduino create"?

@4ntoine 4ntoine changed the title Q: back-end scenario Q: back-end scenario and file contents (not just paths) Sep 13, 2021
@silvanocerza silvanocerza added the type: support OT: Request for help using the project label Sep 13, 2021
@silvanocerza
Copy link
Contributor

The Sketch build path is generated automatically if one is not set explicitly by the user, on terminal it's done using the --build-path flag of the compile command, on gRPC using CompileRequest.build_path.

The build path is generated by this function:

func GenBuildPath(sketchPath *paths.Path) *paths.Path {
path := ""
if sketchPath != nil {
path = sketchPath.String()
}
md5SumBytes := md5.Sum([]byte(path))
md5Sum := strings.ToUpper(hex.EncodeToString(md5SumBytes[:]))
return paths.TempDir().Join("arduino-sketch-" + md5Sum)
}

The upload command, and the relative gRPC function, first searches for the binaries in the build path that would be generated using the specified Sketch path, that's only done if the --import-dir or --import-file are not specified. On gRPC they're UploadRequest.import_file and UploadRequest.import_dir respectively.

Does this help?

How does it work with "arduino create"?

Well, that's quite complicated, also it doesn't use the Arduino CLI fully as of now.

@4ntoine
Copy link
Contributor Author

4ntoine commented Sep 13, 2021

@silvanocerza thanks for the feedback. I was not clear enough. All this assumes that the client and the service are on the same host. But i'm talking about the use case when the client and the service are on different hosts. So it means not paths but actual content (in the responses) should be available over gRPC (or additional service available that returns the files content). is it possible at the moment/future? I can show what i mean in .proto files (probably even in go impl). This can make cli interesting to lots of millions lite clients (eg. mobile) users

@silvanocerza
Copy link
Contributor

Interesting. 🤔

I seem to understand from the SO question that you already managed to compile a Sketch on your host so I'll assume that's working correctly.

What is not clear to me is what you're trying to accomplish after that, do you want to get the compiled binaries from the host to upload them through the client?

In any case as of now you must use the paths on the host that's running the Arduino CLI, so if you create a file on the client you must transfer it to that host to compile it and you then can upload it to a board only from the host, you can't upload from the client since it's not running the Arduino CLI.

@4ntoine
Copy link
Contributor Author

4ntoine commented Sep 13, 2021

Yup, the client can be able to upload itself (eg. OTA) or the back-end can be used for performance reason only. Can you be interested in getting a MR that supports what i propose? So it would act as is for existing use cases but will support remote back-end use cases that i'm asking about?

4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Sep 13, 2021
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Sep 13, 2021
Add TODO where to return the content.
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Sep 13, 2021
@silvanocerza
Copy link
Contributor

Am not sure this is feasible, I think there are certain platforms that produce more than one binary. Also the size concerns me, protocol buffers are not meant to send large data sets as the official docs says.

@4ntoine
Copy link
Contributor Author

4ntoine commented Sep 15, 2021

That could be solved. For example - all build dir could be archived and returned as byte array. That could solve both multiple binary files count and size issues. In general i'd say there are multiple ways of solving it and having a separate service (eg. "filesystem" service) that would just return files bodies for the path request - that would help to separate the things and not mix everything. If it limit the paths for requests (the ones that are used for compilation and new sketches) - it would also solve possible privacy/security issue.

4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 3, 2021
Add .proto file (messages and a service), generate go files.
(Draft to demonstrate the proposal).
@4ntoine
Copy link
Contributor Author

4ntoine commented Nov 3, 2021

@silvanocerza ^ that's what i meant, would love to hear what you think about it. if it looks reasonable, i can implement and publish a MR

4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Move to a proper location (fix compilation).
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Move to a proper location (fix compilation).
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Move to a proper location (fix compilation).
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Move to a proper location (fix compilation).
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Simplify API (remove error)
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Add `FilesService` impl.
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Add `FilesService` impl.
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Add `FilesService` impl.
4ntoine added a commit to 4ntoine/arduino-cli that referenced this issue Nov 4, 2021
Add TODO (Type support).
@per1234 per1234 assigned cmaglie and unassigned silvanocerza Mar 31, 2022
@per1234 per1234 added the topic: gRPC Related to the gRPC interface label Mar 31, 2022
@per1234 per1234 linked a pull request Mar 31, 2022 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: gRPC Related to the gRPC interface type: support OT: Request for help using the project
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants