Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: cache compiled glob #10037

Closed
wants to merge 1 commit into from

Conversation

sapphi-red
Copy link
Member

Description

This PR makes Vite cache compiled globs. This improved performance for ~4% on my project.

micromatch.isMatch always calls picomatch(patterns, options)(str) which leads compiling glob every time.
https://github.com/micromatch/micromatch/blob/002d0d184c95e76775528fa1dbe0c446518879b2/index.js#L123
I've changed this to cache picomatch(patterns, options) to avoid compiling it.

Additional context

Is it safe to rewrite micromatch into picomatch?

picomatch says micromatch supports more more advanced matching with braces. But actually it currently does not for some functions (micromatch/micromatch#169 (comment)) and as you can see from the implementation I quoted above, micromatch.isMatch is just a shorthand.

Why micromatch.scan is rewrote into picomatch.scan?

micromatch.scan is also just an alias of picomatch.scan. But in future the implementation might be fixed. Because I changed micromatch.isMatch to use picomatch directly, I think we should use picomatch.scan directly too to avoid misalignments.
https://github.com/micromatch/micromatch/blob/002d0d184c95e76775528fa1dbe0c446518879b2/index.js#L403

Bundle size

By rewriting micromatch functions to picomatch functions, I was able to remove micromatch dep.
Because fast-glob depends on micromatch, I thought it does not affect bundle size.
But actually it slightly increased bundle size.

package size / unpacked size: 758.9kB / 3.3MB => 759.6kB / 3.3 MB

That said, I think it's acceptable as it's insignificant.


What is the purpose of this pull request?

  • Bug fix
  • New Feature
  • Documentation update
  • Other

Before submitting the PR, please make sure you do the following

  • Read the Contributing Guidelines.
  • Read the Pull Request Guidelines and follow the Commit Convention.
  • Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
  • Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
  • Ideally, include relevant tests that fail without this PR but pass with it.

@sapphi-red sapphi-red added the p2-nice-to-have Not breaking anything but nice to have (priority) label Sep 8, 2022
Comment on lines +73 to +76
const allGlobsMatch = picomatch(
result.matches.flatMap((i) => i.globsResolved)
)
server._importGlobMap.set(id, allGlobsMatch)
Copy link
Member Author

@sapphi-red sapphi-red Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before

type glob = string

const allGlobs: glob[][] = result.matches.map((i) => i.globsResolved)
if (allGlobs.some((glob) => micromatch.isMatch(file, glob)))
  modules.push(...(server.moduleGraph.getModulesByFile(id) || []))

after

type glob = string

const allGlobs: glob[] = result.matches.flatMap((i) => i.globsResolved)
const allGlobsMatch = picomatch(allGlobs)
if (allGlobsMatch(file))
  modules.push(...(server.moduleGraph.getModulesByFile(id) || []))

allGlobs.some(globs => micromatch.isMatch(file, globs))
= allGlobs.some(globs => globs.some(glob => micromatch.isMatch(file, glob)))
= allGlobs.flat().some(glob => micromatch.isMatch(file, glob))
= micromatch.isMatch(allGlobs.flat(), file)
= picomatch(allGlobs.flat())(file)
https://github.com/micromatch/picomatch/blob/13efdf0c7d0e07ee30bf78d0079184aa4a0ec7d4/lib/picomatch.js#L32-L43

@sapphi-red sapphi-red marked this pull request as draft September 8, 2022 09:25
@patak-dev
Copy link
Member

Great PR @sapphi-red. Leaving some references to previous related discussions. createFilter that we now expose through Vite and is used everywhere, uses picomatch. When Fran reported an issue we had with minimatch, we decided to use micromatch to align with fast-glob in this PR #5610 but we already wanted to use picomatch and we even documented that only picomatch syntax is supported. I think we can merge this in 3.2, just to be on the safe side.

@patak-dev patak-dev added this to the 3.2 milestone Sep 8, 2022
@sapphi-red
Copy link
Member Author

Thanks for the pointer ❤️

I marked this PR as draft because I found a bug in import.meta.glob. As I said above ("Is it safe to rewrite micromatch into picomatch?"), micromatch.isMatch does not work with brace-expansion. But fast-glob uses micromatch() which brace-expansion works. So when brace-expansion is used (e.g. import.meta.glob('./{package,package-lock}.json'), it faces the bug (HMR does not happen when editing package.json, similar to #5511).

we even documented that only picomatch syntax is supported.

It seems we weren't doing that.

https://vitejs.dev/guide/features.html#glob-import-caveats
https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-entries

We could pass braceExpansion: false option to make fast-glob align with picomatch.
Or expand brace-expression before passing to picomatch which fast-glob is doing similar.
https://github.com/mrmlnc/fast-glob/blob/79260ad1d322162e71c63dd7eabfcb35b3ad0b9b/src/utils/pattern.ts#L146-L157
https://github.com/micromatch/micromatch/blob/002d0d184c95e76775528fa1dbe0c446518879b2/index.js#L429-L461
Or we could do that in Vite 3 and add a warn that "brace-expression" is deprecated, and drop it in Vite 4.

About server.fs.deny, we don't have any documentation about glob except for jsdoc saying "Glob patterns are supported.". So I think this one is safe to do. (also because the behavior does not change.)

I'll split this PR into two parts, one for server.fs.deny and one for import.meta.glob. The perf improvement is from the server.fs.deny change and this one is noncontroversial and can be merged in 3.1.

@patak-dev
Copy link
Member

we even documented that only picomatch syntax is supported.

It seems we weren't doing that.

vitejs.dev/guide/features.html#glob-import-caveats vitejs.dev/config/dep-optimization-options.html#optimizedeps-entries

We could pass braceExpansion: false option to make fast-glob align with picomatch. Or expand brace-expression before passing to picomatch which fast-glob is doing similar. mrmlnc/fast-glob@79260ad/src/utils/pattern.ts#L146-L157 micromatch/micromatch@002d0d1/index.js#L429-L461 Or we could do that in Vite 3 and add a warn that "brace-expression" is deprecated, and drop it in Vite 4.

Oh, looks like we only did it for an some docs in the types: https://github.com/vitejs/vite/pull/5610/files#diff-9d674f574784ad7dbb518bca67092c190d677c30ccc30fb851b98ecd942b3cc7R10

@sapphi-red
Copy link
Member Author

It seems brace expansion works with micromatch.isMatch 🤔
https://stackblitz.com/edit/node-cpkqke?file=index.js
I am puzzled that brace expansion works with picomatch (=micromatch.isMatch).

Because I guess rewriting getAffectedGlobModules won't affect perf much and I'm not sure if there's a bug about misalignment between micromatch and micromatch.isMatch, I'll close this PR.

@sapphi-red sapphi-red closed this Sep 23, 2022
@sapphi-red sapphi-red deleted the perf/cache-compiled-glob branch September 23, 2022 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p2-nice-to-have Not breaking anything but nice to have (priority)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants