System Tests

[justintaft]

DO NOT MERGE. EXPERIMENTAL.

Whenever light table runs remote debugging is available. This is a security concern and we may want to change this in the future...

LightTable/deploy/core/main.js

Line 107 in baa2037

app.commandLine.appendSwitch('remote-debugging-port', '8315');

To access remote debugging tools:

• run light table as normal
• navigate to http://localhost:8315

To debug with web driver:

• run chromedriver -v
• run the  contents of system-tests.clj (binary location will need to be changed)
• Launch light table

Sometimes light table does not initialize all the way. If this happens, close light table & retry.

I would like the system tests to use a cljs or js library, so we don't have to boot up the jvm for testing. Any Thoughts?

[kenny-evitt]

Thanks! This will be great to get working.

[justintaft]

Not sure how to use taxi's send-keys method with switch-to-active return value...

[justintaft]

Hm, made a issue ticket on clj-webdriver for discussing semantics of taxi/switch-to-active. IMO this probably won't be used too often. We can create a simple function called "current-element" which can be used as the query value for taxi if need be.

https://github.com/semperos/clj-webdriver/issues/153

[kenny-evitt]

@justintaft FYI, let's use this PR for ongoing discussion.

[justintaft]

There is a command "Dev: Open Developer Tools". This shows the familiar developer tools in chrome. Easy way to inspect the DOM =).

We should have a system test for creating, saving, and loading a file. However, a native file dialog window pops up during these processes and you can't use web driver to click save. In other projects I've wrote a program to click native buttons, but it was always a hacky solution.... anyone have a better suggestion? This file dialog window also appears when adding a file/directory to the workspace tree.

For now I can keep things simple. I'll create a temporary file, launch the editor passing in the path to this file, verify the contents are correct, modify the contents, save, then verify the changes have been stored to disk. Since the file exists, we won't have the dialog window pop up for now...

[kenny-evitt]

@justintaft Would it overly defeat the purposes of these tests to fake or mock the native OS UX? We don't need to test it directly – it should be enough that the fake or mock returns specific values to the calling code.

[justintaft]

@kenny-evitt

This should be ok.

If not done so already, their should exist a cljs function used to show the dialog window. This will contain the electron call to show the native file window selection screen. We can then assert this code is correct in a test, and then replace it with our mocking code.

[justintaft]

@kenny-evitt

Needs a bit of refactoring, but the dialog window should be mockable now =).

[kenny-evitt]

@justintaft Nice!

[kenny-evitt]

@justintaft We have a QA checklist of tests to do manually prior to release – these would be great to automate.

[justintaft]

@kenny-evitt Thanks =)

Hopefully once I get everything bootstrapped I can get some help from the community.

Ok, so I wrote the code for the mock dialog window. Question is, how do we inject this mock?

Few options:
-Write a "test" plugin which is used to setup the environment.
-Compile the script and modify the user behavior to autoload the js.
-Open up a repl, type the code in, and evaluate (this is really nasty).

Is there a way to sandbox light table so it won't find my personal user behavior file & installed plugins?

[justintaft]

Ah alright, looks like the "user" directory is overridable.

LightTable/src/lt/objs/files.cljs

Line 360 in 1d9dd0c

(defn lt-user-dir

As for injecting the mock, I'll go the plugin route for now, unless otherwise suggested =).

[kenny-evitt]

@justintaft A plugin seems perfectly fine.

[justintaft]

@kenny-evitt

Added some assertions for saving & loading a file.

I couldn't use $('.CoreMirror-code')[0].CodeMirror to get an instance. Webdriver reported $ wasn't defined. Weird thing is in dev tools, $ is available... I just asserted against the text in the editor for now.

I added some nasty timeouts in the code, they should be replaced with some function that continually polls until some condition is met, and errors out after a certain time if it doesn't.

I have an example in the code for actions (moving up to an editor tag and clicking the "x" button). This will be similar for dragging tab tests.

Also, going to try and get chromedriver&webdriver working on travis-ci.
Found this commit in a different repo and looked like they had success EFForg/https-everywhere@ecdff2b

[justintaft]

@kenny-evitt

Save any local work before using scripts below

script/run-tests.sh should run the automated tests.

It downloads & starts chromedriver, and then runs lein test. The run test script needs to handle different operating systems. I updated travis yml to auto run them, but have not tested everything.

If you need to download the mac version of chrome driver, then uncomment the mac line in run-tests.sh.

Lastly, light table doesn't initialize all the way for whatever reason when chrome driver is running. Running "lein test" a few times usually gets the tests running. You might have to kill the old light table process if it hangs.

[kenny-evitt]

@justintaft This is really great work. 👏

I'm working on manually QA-ing the 0.8.0 alpha release now but I'd like to try to cover those tests with this stuff.

[justintaft]

@kenny-evitt

It looks likes the tests are running on travis =). I expected 1 to fail and 2 to succeed. The failing test was was to verify the tests actually run.

https://travis-ci.org/LightTable/LightTable/builds/83831752#L443

Also, https://github.com/LightTable/LightTable/pull/1996/files#diff-60356cebc67cdde1a0e1379669a603e5R40 . I'm running bash and passing in a script to start LT. This is so I can redirect STDERR and STDOUT in the bash script, without adding some try/catch blocks inside the cojure test suite to capture the output shell/sh if an exception is thrown by web driver. This...is kind of nasty.

[justintaft]

Found a cygwin appveyor.yml project example
https://github.com/DynamicDevices/mono/blob/master/appveyor.yml
https://github.com/Intervox/clj-progress/blob/master/appveyor.yml

[justintaft]

@kenny-evitt

Windows CI setup is coming along. I still need to download chrome for windows & unzip in the right place. The console output is looking good though.

https://ci.appveyor.com/project/kenny-evitt/lighttable/build/1.0.36

[kenny-evitt]

@justintaft looking good

[justintaft]

@kenny-evitt tests are running on windows AND linux now =)

Again, the failure is expected
https://travis-ci.org/LightTable/LightTable/builds/86075536#L443
https://ci.appveyor.com/project/kenny-evitt/lighttable/build/1.0.81#L451

[kenny-evitt]

@justintaft Thanks for working on this. What's next?

[Mouvedia]

Most* of the bullets of the QA should be converted to checkboxes so that we can tick the ones that you have already covered.

Did you you finish 1. and 3.? If so good job!
It seems we will have to give up 2. for now.
4. might be something that @sbauer322 be interested in.

* not everything can be tested […]

[justintaft]

#1 is taken care of. I implemented a polling function which re-runs until it passes (ie, waiting for an element to become visible). Default is wait 1 second at most, and re-try every 100 ms.

#3, I cleaned up the code a bit but it honestly can be cleaned up more. I just don't have the time to in the foreseeable future, sorry.

I tackled #4 a bit, gave a high level overview in docs/testing.md and added docstrings to functions.

[justintaft]

OSX is now being tested.

Updated .travis.yml file to manually install leiningen, worked out of the box.
Hopefully this PR is useful and will save the LT team a bunch of time =).

[Mouvedia]

You could add [email protected] as well.
cf https://github.com/orgs/LightTable/people

[Mouvedia]

@lighttable/maintainers can one of you review this PR?

[sbauer322]

Due to the large number of changes, and my lack of familiarity with the topic, I will be deferring to the other maintainers for this review.

[Mouvedia]

lighttable_homedir_for_testing?

[Mouvedia]

automated

[Mouvedia]

separated

[justintaft]

Is there a need for system-level tests on macOS, Linux, and windows? I can update the PR if needed.

[prertik]

@justintaft sure! if you have the time. This looks good, by my changes(in local) need to be somewhat modified after looking at this.