-
-
Notifications
You must be signed in to change notification settings - Fork 782
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add duckdb to packages #4684
base: main
Are you sure you want to change the base?
Add duckdb to packages #4684
Conversation
f8f17ce
to
ee76f92
Compare
It looks like the mypy build failed, not sure how to proceed. |
cc @ryanking13 and @hoodmane, looks like y'all reviewed the previous attempt of duckdb. |
15583b6
to
4cfeebe
Compare
Not sure how to get past the mypy build failure. |
Well the mypy failure looks pretty unrelated to you. Try disabling it for now: --- a/packages/mypy/meta.yaml
+++ b/packages/mypy/meta.yaml
@@ -1,6 +1,7 @@
package:
name: mypy
version: 1.8.0
+ _disabled: true
top-level:
- mypyc
- mypy |
It looks like I might be hitting a timeout, the build just died after 1 hour. |
Not entirely sure what the expectation is around build times here, but in the corresponding duckdb PR it takes 10-15 minutes to build one |
c820271
to
2fc6fc0
Compare
I'm happy to keep disabling packages until the duckdb package builds successfully, but that doesn't seem like a great solution. |
Ah interesting. I guess we really need to figure out how to improve our package system at this point. I think @ryanking13 has thought about this more than I have. We could break the job up into two sets somehow... |
Hopefully I'm not opening a can of worms here, but GitHub actions has a 6 hour timeout. Would moving these jobs to GHA be on the table? |
It looks like there's a decent amount of circleci code to port in any case. |
I think a lot of aspects of the CircleCI system is a lot better than github actions, including "rerun with ssh access," better handling of test results, the nice workflow chart showing the task DAG, etc. But a longer timeout would be useful. As I said, the CI and package build setup is something that @ryanking13 has been more directly working on of late so let's see what he has to say about it. |
While this can be a temporary solution, we don't want to see CI running time continue to increase. A more fundamental solution is to move away from the way Pyodide currently builds all packages.
I'm working on separating the pyodide main module and package builds from this repository (https://github.com/ryanking13/pyodide-recipes-mirror). This requires some modifications to our build system (and distribution logic), so there's some work to be done. While I'm working on that, as a temporary solution, we might consider pre-building some packages that take a very long time to build, but have no dependencies on other packages. mypy is one example (I think duckdb can be another candidate too), which takes 10+ minutes to build, but has no dependencies on other packages, making it easier to build and test separately. |
Ah, ok. Great to hear. Is there anything I can do to help move this along? |
It will probably take some time for me to work on our build system, so in the meantime I'd recommend building duckdb out-of-tree and see if it works well. Here are some useful links you might find helpful
|
@ryanking13 Yep, last week a PR of mine was merged into DuckDB duckdb/duckdb#11531. The wheels are building in CI there for both Python 3.10 and Python 3.11 I think it's really just about the CI issue here. |
2fc6fc0
to
9aa701c
Compare
Description
Adds
duckdb
to the list of packages shipping with pyodide.The included patches can be removed after the next duckdb release.
Checklists