-
-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: add an end to end smoke test using vscode-extension-tester #270
base: main
Are you sure you want to change the base?
test: add an end to end smoke test using vscode-extension-tester #270
Conversation
Would be nice to also have it here: https://github.com/vitest-dev/vitest-ecosystem-ci (should just reference the name of the script in this repo) To catch regressions in Vitest itself. |
@@ -126,10 +126,14 @@ | |||
"compile": "tsup ./src/extension.ts --external vscode --minify", | |||
"watch": "tsup ./src/extension.ts --external vscode --watch --sourcemap", | |||
"test": "vitest run --config vite.global-config.ts", | |||
"test:e2e": "rimraf .test-extensions && rimraf out && tsc -p ./ && extest setup-and-run './out/tests/e2e/*.test.js' --extensions_dir .test-extensions", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are building with tsup, not tsc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am having some problems configuring mocha to process typescript files directly. Can we leave it as-is for now, and then I will explore a better solution in the next PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't you add this to options?
Line 7 in b5ca1a8
preload: 'tsx/cjs', |
I haven't tried this but I found this one from webdriverio https://github.com/webdriverio-community/wdio-vscode-service, which looks similar/alternative to I'm not familiar with neither selenium nor webdriverio, so I cannot tell the difference myself, but maybe it's worth comparing? |
I like their API! I wonder if Playwright has a similar thing 🤔 It's impossible to google because they have their own extension 😄 |
Both of them are very similar. The only difference is the underlying tech ("This project was highly inspired by the vscode-extension-tester project which is based on Selenium. This package takes the idea and adapts it to WebdriverIO."). It is possible to use Playwright, but then we would have to do the setup ourselves (downloading VSCode, launching it, getting access to the window, etc.), which is not really worth it in my opinion, as we will have just a smoke test. We can use vscode-extension-tester as a starting point, and down the line check out the progress of other options (switching them is trivial). |
7ea7a79
to
85c6dc9
Compare
…d-testing-infrastructure
Why use vs-code-extension-tester at all and not just rely on the official vscode-test which is already end-to-end and spins up a real vscode instance? |
vscode-test is not a true end to end testing solution. vscode-extension-tester, and wdio-vscode-service operate on a level of exposing the access to VSCode's page objects, which is suitable for a smoke test. A lot of popular extensions use vscode-test primarily while using other tools for a smoke test, including: https://marketplace.visualstudio.com/items?itemName=ritwickdey.LiveServer ...and even Microsoft's official Playwright extension: https://github.com/microsoft/playwright-vscode/blob/main/tests-integration/tests/basic.test.ts |
Thanks for the links. Playwright one is very interesting.
As you've already mentioned, essentially it's just "downloading VSCode, launching it, getting access to the window", but this is implemented as test fixture, so in principle, I think we could even do that as Vitest fixture too. I think what vscode-extension-tester and wdio-vscode-service additionally do is to provide many builtin "proxy object" to easily access known UI components on Vscode view. It's getting interesting for me, so I'll probably dig into how these ideas are implemented later. Personally, going with |
I went through playwright, wdio-vscode-service, and vscode-extension-tester and got a rough idea what they do, so let me share my little survey.
Though I went through all this, actually my quick take away from this is that, even with "page objects" utilities, it looks quite difficult to write tests in this way compared to |
Hi, the aim was not to replace vscode-test, but to write a single end to end smoke test that truly tests the complete flow, for example:
Maintenance of such test would be close to zero, while providing value through realistic flow. |
Ah, don't worry. I totally understand the intention of this PR and it makes sense to have only bare minimal test for this. |
Sorry for yet another update. I was very wrong about playwright electron and it actually supports recorder/inspector already, so test authoring works quite nice. I updated my demo to replace I know it's not really relevant to this specific PR, but I hope it still has some value to survey the testing ecosystem as a Vitest team. |
vscode-extension-tester will download the latest Visual Studio Code, package the extension, run it in isolated Visual Studio Code instance, and run the tests.