Add test for parsing brackets in range queries

[benchaplin]

Description

I've added some unit tests to protect/display the QueryParser's ability to parse brackets in range query terms when in quotes.

This issue raised a question about the QueryParser's ability to handle brackets in a range query's terms, even when escaped, for example:

queryParser.parse( "[ a\\[i\\] TO b\\[i\\] ]" );

/* 
org.apache.lucene.queryparser.classic.ParseException: Cannot parse '[a\[i\] TO b\[i\]]': Encountered " "]" "] "" at line 1, column 6.
Was expecting:
    "TO" ...
 */

You can get around this exception by wrapping each range query term in quotes:

queryParser.parse( "[\"a[i]\" TO \"b[i]\"]" );

This is because of this JavaCC grammar rule. <RANGE_GOOP> cannot handle a closing bracket, even an escaped one, while <RANGE_QUOTED> can.

I've added some unit tests to protect this functionality.

---

Any thoughts on whether queryParser.parse( "[ a\\[i\\] TO b\\[i\\] ]" ); should work? I did play around with updating the grammar, it should be possible but I need to think more about how it might break backwards compatibility.

[dweiss]

Any thoughts on whether queryParser.parse( "[ a\[i\] TO b\[i\] ]" ); should work? I did play around with updating the grammar, it should be possible but I need to think more about how it might break backwards compatibility.

Looking at the javacc file it seems range terms are parsed differently compared to normal tokens. This is strange but I bet there was a reason for it - now long forgotten. I think, logically, it should accept the same term notation that is used elsewhere. It will break backward compatibility but maybe consistency is worth this cost (for the major release)?

[benchaplin]

I was thinking a fix like this could work:

| <RANGE_GOOP:   (~[ " ", "]", "}" ] | "\\ " | "\\]" | "\\}" )+ >

simply allowing the range term to continue parsing through an escaped space or closing bracket.

"Breaking" changes:

• This would permit some queries that previously threw exceptions (like the one described in the original issue)
• I'm trying to think of a query that was permitted and now would break (all existing unit tests pass). My first thought was a range query ending with an escaped bracket, however those are disallowed by the Term can not end with escape character rule. Any thoughts?

[dweiss]

The joys of escaping... Never easy (hello, Windows command-line users...). You may want to add a randomized test that constructs those terms using a mix of characters and "allowed" escape sequences - a corner case will pop up quickly, I think.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!

[benchaplin]

Thanks for the advice @dweiss, I will play around with some test cases for this fix. In the meantime, I think we can merge this and I will open another PR with the grammar changes if I can get something working.