feat(Programmatic API): Add programmatic API #14062

nicojs · 2023-04-09T20:28:53Z

Summary

This pull request adds a documented programmatic API to run Jest. It can be used to run using the new experimental API.

import {createJest} from 'jest';

const jest = await createJest();
jest.globalConfig = {
  collectCoverage: false,
  ...jest.globalConfig,
};
const {results} = await jest.run();
console.log(`run success, ${result.numPassedTests} passed tests.`);

The new Jest instance has the following properties:

globalConfig: The global configuration to use for this run. You can use it to override arbitrary options (albeit by overriding the entire object reference).
projectConfigs: The project configurations to run. You can use it to override arbitrary project options (albeit by overriding the entire array reference).
run: Function that kicks of a test run.

Motivation

This change adds a documented programmatic API to run Jest, which makes it easier to build tools that use Jest or for users to run Jest from their own scripts.

Test plan

I have run it using yarn test. I've also added a unit test.

TODO

Document the programmatic API in /docs.
Add an e2e test that uses the programmatic API.
Test the API from my code.

@SimenB can you confirm that I'm on the right track here?

Closes #14034
Closes #5048

Add a documented programmatic API to run jest. It allows this use case: ```ts import { run, readConfigs } from 'jest'; const { globalConfig, configs } = await readConfigs(process.argv, [process.cwd()]); // change globalConfig or configs as you see fit const results = await run(globalConfig, configs); console.log(results); ```

nicojs · 2023-04-09T20:45:35Z

The build is failing because packages/jest-core/build/cli/index.d.ts now has a reference to pnpapi. I have no clue why it would have that:

 /**
  * Copyright (c) Meta Platforms, Inc. and affiliates.
  *
  * This source code is licensed under the MIT license found in the
  * LICENSE file in the root directory of this source tree.
  */
 /// <reference types="node" />
 /// <reference types="node" />
 /// <reference types="node" />
 /// <reference types="node" />
 /// <reference types="node" />
+/// <reference types="pnpapi" />
 import type { AggregatedResult } from '@jest/test-result';

packages/jest-core/src/cli/index.ts

mrazauskas · 2023-04-10T07:51:54Z

For my use case I would need something like this:

import {run} from 'jest';

run(['path/to/some.test.ts'], {coverage:true});

Reading configs is useful, of course. But how to make it possible to override some option or selecting files almost like from CLI. runCLI() is not a solution, because I would like to be able to pass InitialOptions and it does not allow selecting files.

What if programmatic API would be shaped like this:

import {Jest} from 'jest';

const jest = new Jest();

jest.readConfig(); // same as `readConfigs()`

jest.setConfig({coverage: true});  // or as a shorthand new Jest({coverage: true})

jest.selectFiles(['path/to/some.test.ts']);
jest.selectProjects(['backend']);

const result = await jest.run();
// or await jest.watch();

nicojs · 2023-04-10T17:01:50Z

Thanks for your review @mrazauskas! I've implemented your suggestions and created a separate runCore API. I've moved the entire run stuff to a new file, as it makes more sense now than a cli/index.ts file.

I've also decided to make runCore return a new object:

/**
 * The result of running runCore.
 */
export type RunCoreResult = {
  result: AggregatedResult;
};

That way, we can later introduce more return results without breaking the API.

Reading configs is useful, of course. But how to make it possible to override some option or selecting files almost like from CLI. runCLI() is not a solution, because I would like to be able to pass InitialOptions and it does not allow selecting files.

This is already possible. In fact, this is how Stryker currently works:

import {runCLI} from 'jest';
import {readInitialOptions} from 'jest-config';
const initialOptions = readInitialOptions();
initialOptions.testRegex = ['path/to/some.test.ts'];
const result = await runCLI(
      {
        $0: 'unused',
        _: [],
        config: JSON.stringify(initialOptions)
      },
      [process.cwd()]
    );

What if programmatic API would be shaped like this:

import {Jest} from 'jest';

const jest = new Jest();

jest.readConfig(); // same as `readConfigs()`

jest.setConfig({coverage: true});  // or as a shorthand new Jest({coverage: true})

jest.selectFiles(['path/to/some.test.ts']);
jest.selectProjects(['backend']);

const result = await jest.run();
// or await jest.watch();

This could be done, but this would be a deviation from the functional API that has been the norm so far. Note that your example can also be done with the new runCore functionality:

import {runCore, readConfigs} from 'jest';
const {globalConfig, configs} = await readConfigs(process.argv, [process.cwd()]);
globalConfig.collectCoverage = true;
globalConfig.testPathPattern = ['path/to/some.test.ts'];
const results = await runCore(globalConfig, configs);
const projectConfigs = configs.filter(({displayName}) => displayName = 'fooProject');
const {result} = await runCore(globalConfig, projectConfigs);

I've allowed for more return properties in the future, so we might add a jest instance or something in the result of runCore, which could allow for faster subsequent runs (this is actually what we would want in the end for Stryker, as we need to run thousands of times with slightly different global variables for switching between active mutants).

mrazauskas · 2023-04-10T17:28:27Z

Sure, many things can be done with current API. That is not a question. Usability is missing. What I am looking for an API which is as easy as jest some.test.ts or jest --config={"coverage":true} --selectProjects second.

import {Jest} from 'jest';

const jest = new Jest();

const result = await jest.run(['some.test.ts']);

import {Jest} from 'jest';

const jest = new Jest({coverage: true});
jest.selectProjects(['second']);

const result = await jest.run();

import {Jest, readConfig} from 'jest';

const config = readConfig('.');
config.coverage = undefined;

const jest = new Jest(config);
const result = await jest.run();

All I ask is feedback, happy to implement it myself (; Only that this has to wait for major release.

nicojs · 2023-04-11T20:29:28Z

All I ask is feedback, happy to implement it myself (; Only that this has to wait for major release.

I'm not sure. I personally don't mind either API. If you want to go the class syntax route, does this PR still make sense? Also: maybe the class should be called Jester? 😜

I need some help with the failing build. Does anyone know why pnpapi is suddenly referenced?

Error: /home/runner/work/jest/jest/packages/jest-core/build/runCore.d.ts has the following non-node type references:

/// <reference types="pnpapi" />

    at file:///home/runner/work/jest/jest/scripts/buildTs.mjs:225:17

mrazauskas · 2023-04-12T06:23:46Z

The pnpapi issue looks familiar. See #12783

Class was just a rough idea. Actually a function like createJest() instead of the class could be better idea, because it can be async. That way it could take anything exported from Jest config file (just another way to use it):

import jestConfig from '../jest.config.js';

const jest = await createJest(jestConfig);

const result = await jest.run();

nicojs · 2023-04-21T19:39:40Z

I've been able to fix the <reference types="pnpapi" /> error (great linting error, btw!). I've added some e2e tests and unit tests. I'm still busy with the docs, but you should take a look.

@mrazauskas I'm unopinionated about your API proposal. Maybe @SimenB can also pitch in here since he previously expressed an opinion about the programmatic API. I wouldn't want to do more work just to be shot down later...

SimenB · 2023-04-22T06:49:55Z

/easycla

linux-foundation-easycla · 2023-04-22T06:50:21Z

The committers listed above are authorized under a signed CLA.

✅ login: nicojs / name: Nico Jansen (908c73d, e2c65f8, 6a75acf, 7d70600, 8155081, 79dbfd5, d0fd9a9, b9e9278, d8f6e26, 68c924f, 68ac3ef, a375b0d, 0f8f6f2, 1935947, 77723b3, a749243, 48a5a94, ac2b879, e5c254f, 59cfa1a, d570bc8, 80d3c23, 098cc21, 65fe53f, 9f1a113, 8da9922, 6a26f7f, fe45d4f, ef783ed, 46a7ce9, 9f056b8, 92d6d44)
✅ login: SimenB / name: Simen Bekkhus (0b6f93d, 6877134, 26e87e1, f3e41ef, 2677e1d, f6d0c5a)
✅ login: KhaledElmorsy / name: Khaled Elmorsy (c3b95f8)
✅ login: renovate[bot] (8ec975a)
✅ login: alexreardon / name: Alex Reardon (ccc7e4e)
✅ login: nolddor / name: Jack Nolddor (f04118d)
✅ login: Mutesa-Cedric / name: MUTESA Cedric (f8e0259)

SimenB · 2023-04-22T06:53:53Z

I like @mrazauskas's suggestion. It should be with a function and not a class so it can be async like mentioned. I'd expect the initial function to either take a path to a config file, inline config, or nothing (which would be Jest try to find the config manually).

SimenB · 2023-10-04T06:46:47Z

@segrey @DmitryMakhnev Is this something that would be usable from IntelliJ (or Fleet)? Anything obvious missing (or wrong)?

@connectdotz same question - does this help the vscode extension?

Would love to hear your feedback 🙂

segrey · 2023-10-04T11:20:42Z

@SimenB IntelliJ/Fleet both run Jest tests using Jest CLI. It appears that we're not currently interested in it.

SimenB · 2023-10-04T11:31:41Z

Fair enough 🙂

If you wanna use the API instead of CLI and reporter at some point in the future, I'd love to figure out what's missing then 👍

connectdotz · 2023-10-04T20:35:23Z

@SimenB I took a quick read on the api document; it seems these APIs are mainly to manage config, invoke the CLI/core logic, then get back the aggregated test results from the run. It could probably save us a few steps like managing child processes (for CLI), parsing the json result file, etc. But given these are already implemented and working, changing the CLI-based eco-system is not trivial, and I didn't see extra features or additional use cases enabled by the API for our extension... so not sure we will be using the API, at least for now...

I do have some general questions though:

The CLI output is pretty useful. Does the API still provide a way to output these in real-time (not at the end of the run)?
are the reporters still supported? what will a reporter looks like with the API approach? Reporter mainly provides a more real-time, incremental information, which the API looks like only supports batch mode?
I assume the API also supports coverage reporting?

… failing snapshot matchers (jestjs#14313)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

nicojs · 2023-10-14T12:11:33Z

@SimenB I think I'm done now 😀

nicojs · 2023-10-16T12:34:37Z

I've updated the PR text to reflect the recent changes. I don't know why the build is failing.

SimenB

sorry, forgot about this 😅 made some quick comments, will look through more throughly later

SimenB · 2023-10-30T13:11:16Z

docs/ProgrammaticApi.md

+jest.globalConfig = {
+ collectCoverage: false,
+ watch: false,
+ ...jest.globalConfig,
+};


I don't think we should expose something that can be mutated in this way. I'd rather have a mergeConfig or some such that is invoked with the current config and expects to get a new config back it assigns internally

Do you mean something like this?

import {createJest, mergeConfig} from 'jest'; const jest = await createJest(); // Override global options jest.globalConfig = mergeConfig(jest.globalConfig, { collectCoverage: false, });

Or more like this?

import {createJest} from 'jest'; const jest = await createJest(); // Override global options jest.mergeConfig(jest.globalConfig, { collectCoverage: false, });

import {createJest} from 'jest'; const jest = await createJest(); // Override global options jest.mergeGlobalConfig({ collectCoverage: false, });

or

jest.globalConfig.merge({ collectCoverage: false, });

The latter would mean globalConfig was some sort of class or something instead of just the object

How would you override a project-specific config?

I think allowing you to mutate config in-place makes more sense, actually. So removing the read-only feature. I don't think freezing the config serves a real purpose. 🤷‍♀️

I generally prefer encapsulation - makes it easier to debug when all mutations come though known methods and you can set a breakpoint in it.

How would you override a project-specific config?

jest.mergeProjectConfig(index, { testEnvironment: 'node', });

This actually brings an important point to mind - any config modifications must go through its part in normalize within jest-config, to e.g. resolve testEnvironment to an absolute path.

I guess we can punt on that for now though. But if we do kick it down the line for later, I want an

jest.unstable_modifyConfig() or some such so we can add proper config normalization later without a breaking change.

SimenB · 2023-10-30T13:12:12Z

e2e/run-programmatically-multiple-projects/run-jest.js

@@ -5,18 +5,20 @@
 * LICENSE file in the root directory of this source tree.
 */

-const {runCLI} = require('@jest/core');
+const {createJest} = require('jest');


@nicojs missed?

SimenB · 2023-10-30T13:12:36Z

e2e/run-programmatically/package.json

+ "testEnvironment": "node",
+ "maxWorkers": 1


why is this needed?

It's not. Will remove

SimenB · 2023-10-30T13:13:10Z

packages/jest-core/src/jest.ts

+ return {contexts, hasteMapInstances};
+};
+
+export const _run = async (


why do we export with _ prefix?

Good catch. Fixed by removing the _

nicojs · 2023-10-31T20:23:45Z

Thanks @SimenB, welcome back 🎃

I think fixed all your remarks except for one where I have a question.

nicojs · 2023-11-07T06:36:36Z

@SimenB, what would you think about just making the config objects mutable? I personally don't see a reason to make it more complicated than it needs to be. What are we actually afraid of? It was always 'shallow immutable' anyway. 🤷‍♂️

SimenB · 2023-12-25T11:36:29Z

Main issue isn't mutability, but the fact you can provide invalid configuration values. E.g. jsdom is not valid configuration at this point - it has to be an absolute path to a module. Our normalize function needs to do its work on any input from the user, thus it needs to go via a setter of some kind.

mbaumanndev · 2024-02-01T19:43:04Z

@SimenB @nicojs I work on a codebase using Stryker for my tests where this PR would be a huge help to simplify the codebase. Is there anything I can do to help on this PR ? I've not contributed to many projects but I'm willing to invest some time to help here as it would help me a lot in my daily work

SimenB · 2024-03-16T08:24:46Z

Hey @mbaumanndev! I think we're only missing normalization of configuration when setting it. If you could contribute that, I think we're ready to land this 🙂

facebook-github-bot added the cla signed label Apr 9, 2023

nicojs marked this pull request as draft April 9, 2023 20:37

nicojs mentioned this pull request Apr 9, 2023

[Feature]: Programmatic api to resolve the entire config and run jest #14034

Open

mrazauskas reviewed Apr 10, 2023

View reviewed changes

packages/jest-core/src/cli/index.ts Outdated Show resolved Hide resolved

packages/jest-core/src/cli/index.ts Outdated Show resolved Hide resolved

packages/jest-core/src/cli/index.ts Outdated Show resolved Hide resolved

mrazauskas reviewed Apr 10, 2023

View reviewed changes

packages/jest-core/src/cli/index.ts Outdated Show resolved Hide resolved

mrazauskas reviewed Apr 10, 2023

View reviewed changes

packages/jest-core/src/cli/index.ts Outdated Show resolved Hide resolved

Implement review comments

e2c65f8

Fix lint: add copyright header

6a75acf

Import performance from perf_hooks

7d70600

nicojs added 7 commits April 12, 2023 09:32

Fix type error

8155081

Remove pnpapi from types

79dbfd5

Add e2e test for runCore

d0fd9a9

Update e2e test for programmatic run

b9e9278

Fix lint e2e

d8f6e26

fix prettier

68c924f

Add unit tests

68ac3ef

nicojs marked this pull request as ready for review April 21, 2023 19:36

Add copyright header

a375b0d

Add docs for programmatic api

0f8f6f2

KhaledElmorsy and others added 8 commits October 13, 2023 20:26

fix(expect, jest-snapshot): Pass test.failing tests when containing…

c3b95f8

… failing snapshot matchers (jestjs#14313)

chore: update ESLint dependencies (jestjs#14595)

2677e1d

chore(deps): update yarn to v3.6.4 (jestjs#14600)

8ec975a

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

feat: add jest.advanceTimersToFrame() (jestjs#14598)

ccc7e4e

fix: coverage badge is broken at readme (jestjs#14611)

f04118d

docs: provide README for jest-haste-map package (jestjs#14613)

f8e0259

docs: update with "experimental" message

098cc21

docs: remove mention of old runCore

65fe53f

nicojs added 3 commits October 14, 2023 15:07

Replace old api docs with the new api using jest instance.

9f1a113

Merge branch 'main' into feat/programmatic-api

8da9922

prettier

6a26f7f

nicojs mentioned this pull request Oct 14, 2023

Exposing the built-in jest config angular/angular-cli#25434

Open

Merge branch 'main' into feat/programmatic-api

f6d0c5a

SimenB reviewed Oct 30, 2023

View reviewed changes

nicojs added 5 commits October 31, 2023 20:50

Merge branch 'main' into feat/programmatic-api

fe45d4f

refactor: remove unessesary config

ef783ed

fix: use jest's native esm api in e2e test

46a7ce9

refactor: remove _ prefix

9f056b8

docs: remove reference to old readInitialOptions.

92d6d44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Programmatic API): Add programmatic API #14062

feat(Programmatic API): Add programmatic API #14062

nicojs commented Apr 9, 2023 •

edited

nicojs commented Apr 9, 2023

mrazauskas commented Apr 10, 2023

nicojs commented Apr 10, 2023

mrazauskas commented Apr 10, 2023 •

edited

nicojs commented Apr 11, 2023

mrazauskas commented Apr 12, 2023 •

edited

nicojs commented Apr 21, 2023

SimenB commented Apr 22, 2023

linux-foundation-easycla bot commented Apr 22, 2023 •

edited

SimenB commented Apr 22, 2023

SimenB commented Oct 4, 2023 •

edited

segrey commented Oct 4, 2023

SimenB commented Oct 4, 2023

connectdotz commented Oct 4, 2023

nicojs commented Oct 14, 2023

nicojs commented Oct 16, 2023

SimenB left a comment

SimenB Oct 30, 2023

nicojs Oct 31, 2023

SimenB Nov 1, 2023

nicojs Nov 1, 2023

SimenB Nov 6, 2023

SimenB Oct 30, 2023

SimenB Oct 30, 2023

nicojs Oct 31, 2023

SimenB Oct 30, 2023

nicojs Oct 31, 2023

nicojs commented Oct 31, 2023

nicojs commented Nov 7, 2023

SimenB commented Dec 25, 2023

mbaumanndev commented Feb 1, 2024

SimenB commented Mar 16, 2024

feat(Programmatic API): Add programmatic API #14062

Are you sure you want to change the base?

feat(Programmatic API): Add programmatic API #14062

Conversation

nicojs commented Apr 9, 2023 • edited

Summary

Motivation

Test plan

TODO

nicojs commented Apr 9, 2023

mrazauskas commented Apr 10, 2023

nicojs commented Apr 10, 2023

mrazauskas commented Apr 10, 2023 • edited

nicojs commented Apr 11, 2023

mrazauskas commented Apr 12, 2023 • edited

nicojs commented Apr 21, 2023

SimenB commented Apr 22, 2023

linux-foundation-easycla bot commented Apr 22, 2023 • edited

SimenB commented Apr 22, 2023

SimenB commented Oct 4, 2023 • edited

segrey commented Oct 4, 2023

SimenB commented Oct 4, 2023

connectdotz commented Oct 4, 2023

nicojs commented Oct 14, 2023

nicojs commented Oct 16, 2023

SimenB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicojs commented Oct 31, 2023

nicojs commented Nov 7, 2023

SimenB commented Dec 25, 2023

mbaumanndev commented Feb 1, 2024

SimenB commented Mar 16, 2024

nicojs commented Apr 9, 2023 •

edited

mrazauskas commented Apr 10, 2023 •

edited

mrazauskas commented Apr 12, 2023 •

edited

linux-foundation-easycla bot commented Apr 22, 2023 •

edited

SimenB commented Oct 4, 2023 •

edited