=~ is interactive sensitive (sometimes)

[phidebian]

I bumped into this and since I am 're-learning' ksh from scratch, I feel a bit disorientated, so pardon if there is a trivial answer to this.

PW$ /d1/ksh-latest/arch/gnu.i386-64/bin/ksh  --version
  version         sh (AT&T Research) 93u+m/1.1.0-alpha+f215b10a/MOD 2022-10-28
PW$ /d1/ksh-latest/arch/gnu.i386-64/bin/ksh -ci '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a]
logout
PW$ /d1/ksh-latest/arch/gnu.i386-64/bin/ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a] b [c]

I learned that =~ is equivalent to == ~(E) but I see nothing in my installed man(1) (ubuntu 22.04, 93u+m/1.0.0-beta.2 2021-12-17) that add a interactive condition to the =~ form.

It looks like for interactive =~ is equivalent to == ~(-E) while for non interactive seems that =~ is equivalent to == ~(E) (as documented)

So the =~ greed seems to be different for interactive vs non-interactive, yet if using something else for [ or ] for instance : then ~(E) is always non greedy.

And indeed the ~(E)ere with ? to control the greed is working, so I guess ~(E) should always be greedy and the greed be controlled with ?

PW$ /d1/ksh-latest/arch/gnu.i386-64/bin/ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+?\] ]] && echo ${.sh.match}' 
[a]

Well I don't know what is normal or not for ksh93 but the variable behavior for interactive vs non-interactive is disturbing to (not orthogonal) to me.

Any helps appreciated.

[McDutchie]

Cannot reproduce... in normal interactive use, with '-c', and with '-i -c', it all prints the same on my end.

$ arch/darwin.arm64-64/bin/ksh --version                                                                                             
  version         sh (AT&T Research) 93u+m/1.1.0-alpha+4ec93745 2022-10-31
$ ENV=/./dev/null arch/darwin.arm64-64/bin/ksh                                                                                       
$ [[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}                                                                             
[a] b [c]
$ exit
$ arch/darwin.arm64-64/bin/ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}'                                           
[a] b [c]
$ arch/darwin.arm64-64/bin/ksh -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}'
[a] b [c]

So I have no idea what's going on at your end. But the /MOD at the end of your version string suggests you have uncommitted modifications. Try undoing those (git checkout -f), updating the branch (git pull), recompiling (bin/package make) and then try again.

Note that the gnu.i386-64 directory name was the result of a temporary bug in the package script (introduced in b8a8595, fixed in 8ff5410), so you should delete that and use linux.i386-64.

[McDutchie]

I was just able to reproduce this on Debian aarch64 with gcc 10.2.1-6 – though only when entering the reproducer directly at a command line, not when using either -i -c or -c. Even then, the behaviour seems intermittent as just recompiling the binary seems to make it go away again, so at this point it's like chasing a ghost. Investigating.

Relevant commit: b1b8d6f

[McDutchie]

@JohnoKing, can you reproduce this?

[JohnoKing]

I have reproduced the bug in ksh2020 and the latest commit of ksh93u+m on Arch Linux x86_64 (built with gcc 12.2.0). So far I've only been able to reproduce this issue with binaries compiled using LTO.

[McDutchie]

Bug filed: #577

[phidebian]

Sorry I was out, ok seems a bug then, just to be precise I redo all my test with the very latest git dev branch, here are some results.

1. OS linux x86-64

uname -a
Linux phiw 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

PW$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

2. Compilers

PW$ gcc --version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

PW$ clang --version
Ubuntu clang version 14.0.0-1ubuntu1

3. Official ubuntu ksh93 version (package)

PW$ ksh --version
  version         sh (AT&T Research) 93u+m/1.0.0-beta.2 2021-12-17

3.1 Script (non-interactive)

PW$ ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a] b [c]     <--- GOOD, but may be thought as a bug since BASH (and may be other shells) are not greedy

PW$ bash -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo $BASH_REMATCH' # === BASH ====
[a]             <--- IMHO WRONG, but majority may say GOOD

3.2 Interactive

PW$ ksh -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a]             <---- WRONG should be greedy (as it is in script non-interactive above) though as bash think it is GOOD
logout

3.3 ERE greed control
Per https://en.wikibooks.org/wiki/Regular_Expressions/POSIX-Extended_Regular_Expressions
Searching for ^F greed we can read

Modern regular expression tools allow a quantifier to be specified as non-greedy, by putting a question mark after the quantifier: (\[\[.*?\]\]). 

So applied to ksh93

PW$ ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a] b [c]  
PW$ #---------------------------------------v
PW$ ksh -c '[[ "[a] b [c] d" =~ ^\[[^]]+?\] ]] && echo ${.sh.match}' 
[a]

So by implementing = or (E) as greedy by default, we leave a chance to control the greed with ? , yet may be people would call the bash behavior standard (may be posix say so) then we are hose as there is no way to get the greedy mode to work .

There is nothing in the ksh93 doc (I have ksh(1)) saying the ERE is greedy, the only ref of greedy is about ~(g) so not ~(E) yet ~(E) claim it is egrep(1) compatible then AFAIK egrep is greedy (as sed, etc...)

3.4 More trouble
I am not sure what == ~(-E) is supposed to do (doc not clear on that) yet I got the impression it is another way to control the greed, yet result a disorientating too.

PW$ ksh -c '[[ "[a] b [c] d" == ~(E)^\[[^]]+\] ]] && echo ${.sh.match}'     # Script mode greedy mode ~(E) ==>OK
[a] b [c]
 
PW$ ksh -c '[[ "[a] b [c] d" == ~(-E)^\[[^]]+\] ]] && echo ${.sh.match}'    # Script mode NON-greedy mode ~(-E) ==>OK
[a] b [c]

PW$ ksh -i -c '[[ "[a] b [c] d" == ~(E)^\[[^]]+\] ]] && echo ${.sh.match}'   # Interactive greede mode ~(E) ==>BAD
[a]
logout

PW$ ksh -i -c '[[ "[a] b [c] d" == ~(-E)^\[[^]]+\] ]] && echo ${.sh.match}' # Interactive NON-greedy mode ~(-E) ==>BAD
[a]
logout

This later test again I am not sure ~(-E) mean non-greedy, yet interative vs script display strange behavior.

4. My latest git pull gcc -O0 -g3.
   Same behavior as packaged ksh93

PW$ KSH="/d1/ksh-latest/arch/linux.i386-64/bin/ksh"
PW$ $KSH --version
  version         sh (AT&T Research) 93u+m/1.1.0-alpha+4ec93745 2022-10-31

PW$ $KSH -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a]
logout
PW$ $KSH -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a]
logout

5. My latest git pull ubuntu 22.04 x86-64 gcc -O3.
   Same behavior as packaged ksh93

6. My latest git pull ubuntu 22.04 x86-64 clang -O3.
   Same behavior as packaged ksh93

7. My latest git pull ubuntu 22.04 s390x gcc -O0 -g3

P390$ KSH=/home/linux1/ksh-latest/arch/linux.s390-64-64/bin/ksh
P390$ $KSH -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a] b [c]
P390$ $KSH -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+\] ]] && echo ${.sh.match}' 
[a] b [c]
# Greed controled by ? is OK too
P390$ $KSH -c '[[ "[a] b [c] d" =~ ^\[[^]]+?\] ]] && echo ${.sh.match}' 
[a]
P390$ $KSH -i -c '[[ "[a] b [c] d" =~ ^\[[^]]+?\] ]] && echo ${.sh.match}' 
[a]

All is ok (greedy) and non-greedy (via ?) both interactive and script (like @McDutchie, don't reprod)

8. My latest git pull ubuntu 22.04 s390x gcc -O3
   Same as 7.

9. My latest git pull ubuntu 22.04 s390x clang -O3
   Same as 7.

Unfortunatly I can't test other OS'es, win, ios, bsd, solaris, aix, hpux, etc....

At this point I am inclined to think there is something specific to ubuntu 22.04 x86-64 as ubuntu 22.04 s390x seems good, may be in the libc....

If I recover some cycle this week I may look why we got this....

[phidebian]

After reviewing the regex(7) docco it sounds like greed is described with

       In the event that an RE could match more than one substring of a  given
       string, the RE matches the one starting earliest in the string.  If the
       RE could match more than one  substring  starting  at  that  point,  it
       matches  the  longest.   Subexpressions also match the longest possible
       substrings, subject to the constraint that the whole match be  as  long
       as possible, with subexpressions starting earlier in the RE taking pri-
       ority over ones starting later.  Note that higher-level  subexpressions
       thus take priority over their lower-level component subexpressions.

It is way above my english understanding, specially the notion of priority. Anyway it sounds like

'^([[^]]+])' should match only [a] when input is [a] b [c] d then bash and ksh93 are GOOD, only when output only [a]

Then ksh scripting is WRONG when outputing something else [a] b [c]

[McDutchie]

Should now be fixed in edf23e3. Thanks, Phi!

[phidebian]

NP :)