Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can parameters limitPatternLength and limitLiteralCount be increased? #429

Open
rongekuta opened this issue Mar 31, 2024 · 1 comment
Open

Comments

@rongekuta
Copy link

rongekuta commented Mar 31, 2024

i prepare to use hyperscan to match patterns like:

  1. .*abc{dict1}abc
  2. .*123{dict2}123

dict1 and dict2 are big string set (stored in file), like:
dict1:
jack
rose
mike
...

dict2:
tom
jerry
lily
...

i transform my patterns by expanding strings of dict as:

  1. .*abc(jack|rose|mike|...)abc
  2. .*123(tom|jerry|lily)123

because dict1 and dict2 is so large (>20MB)
so transformed pattern is too large, so hyperscan raise error:
"Pattern length exceeds limit"

beloved source code is :
https://github.com/intel/hyperscan/blob/master/src/hs.cpp#L538
https://github.com/intel/hyperscan/blob/master/src/grey.cpp#L148

so can parameters limitPatternLength and limitLiteralCount be increased to match a large pattern with a dict?

@rongekuta
Copy link
Author

i modify limitPatternLength and limitLiteralCount to 1MB, and use large pattern to compile,
and found hyperscan compile is too slow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant