bash is a single-pass language

[zimbatm]

My ideas are not totally clear yet, so let me use this post as a way to unroll them.

Bash is a single-pass language where the parser and interpreter are inter-mingled. This allows doing crazy things like embed binary data into a script:

#!/usr/bin/env bash

# do some things here
exit
----MARKER----
<BINARY DATA>

One of the things I noticed in this project's design, is that the parser and interpreter are separate. This makes sense because the project is also a code formatting tool, it's useful to have a separate parser. The formatted should probably not deal with crazy scripts like above.

I have a sense that this makes the interp package more difficult to implement. One indication is that there are more instances of the parser being spawned off instead of re-using the same one. It's a thought I had while working on #650.

So it's just a thought. Maybe more clarity will appear with more conversation.

[theclapp]

Can you clarify what suggestion you are making (if any)? Or are you just curious whether we/@mvdan agree(s) that the parser and the interpreter being separate makes the latter harder to implement?

[zimbatm]

One idea is to change the syntax.Parser API to make the iteration external.

Instead of:

p := syntax.NewParser()
err := p.Stmts(r, func(*Stmt) bool {
 // ...
})

You could do:

p := syntax.NewParser()
iter := p.Stmts(r)
for stmt, err := iter.Next(); err != nil {
  // ...
}

This would make things more flexible at the iterator can be passed around. It's just a hunch, for now, I don't know that it would remove the need of having multiple instances of the parser.

[mvdan]

We could always add this as StmtsIter if we think it would help, but right now it's not obvious to me that it would help with bash compatibility.

The removal of the callback could help with custom control flows, though. The current API only lends itself well to simple iterations which aren't paused or interleaved with other work.

[mvdan]

I should give a bit of historical context. The first use case was the formatter, so I only needed a static parser and printer. The interpreter was an extension of the project that started about two years later.

You're right in saying that the project could be more sh/bash compliant if the parser was tightly coupled to an interpreter. You cite one case where that can be an advantage for real scripts - never parsing some lines. There are others, such as being able to parse array expressions differently depending on the interpreter state; see https://github.com/mvdan/sh#caveats.

I still believe that having an entirely static parser is a good thing, and something we should keep around. But for those users who mainly use the interpreter and want close compatibility with sh/bash, I'm certainly open to ideas to improve compatibility.

If all we want for now is to just be able to parse one line at a time when interpreting a script, I think the current parser should already support that. If we want more, such as interpreter-aware parsing of array expressions, that would likely require new API.