Parsing strings skipping the quotes

[viperML]

Hello 👋

I just started with the library, and basically in lang dev, so bear with me 🙂

I am trying to just transform a string to a list of tokens with the following parser and types:

#[derive(Debug, PartialEq, PartialOrd)]
enum Token {
    Number(i32),
    Bind,
    String(String),
}

type Span = SimpleSpan;
type Spanned<T> = (T, Span);

fn lexer<'src>() -> impl Parser<'src, &'src str, Vec<Spanned<Token>>> {
    let number = text::int(10).map(|s: &str| Token::Number(s.parse().unwrap()));

    let string = just('"')
        .ignore_then(any().filter(|c| *c != '"').repeated())
        .then_ignore(just('"'))
        .map_slice(|s| Token::String(String::from(s)));

    let bind = just('=').map(|_| Token::Bind);

    bind.or(number)
        .or(string)
        .map_with_span(|token, span| (token, span))
        .padded()
        .repeated()
        .collect()
}

My problem is that the string parser includes the quotation marks, so parsing "hello" yields a Token::String("\"hello\"") . What am I missing here?

The full code (not much more): https://github.com/viperML/len/blob/9a9737624101575093bb958e93b846b946e3a9d1/src/lib.rs

[zaynetro]

Let's analyze your parser:

let string = just('"')
  .ignore_then(any().filter(|c| *c != '"').repeated())
  .then_ignore(just('"'))
  .map_slice(|s| Token::String(String::from(s)));

It will parse a quote then everything that is not a quote and finally a quote again. So for input "hello" the parser will consume "hello". You use map_slice which will respond with a complete input that parser consumed.

So a couple of suggestions:

1. Use map instead of map_slice
2. Use a different parser

For (2) I personally just used:

let str_ = none_of('"')
  .ignored()
  .repeated()
  .slice()
  .padded_by(just('"'));

You can see that slice is before padded_by hence quotes are not included.

P.S I am learning chumsky myself at the moment so I tried my best to help you but I might not be 100% correct.

[viperML]

You use map_slice which will respond with what a complete input that parser consumed.

By quickly reading .map_slice description, I didn't think this was the behaviour, as I tried to put it together from other examples/repos.

I came up with this, which seems to work:

    let string = just('"')
        .ignore_then(any().filter(|c| *c != '"').repeated().collect())
        .then_ignore(just('"'))
        .map(|s: String| Token::String(s));

Also thanks for (2). It is nice to see different solution to the same problem.

[Zij-IT]

@zaynetro is correct. map_slice captures everything that the parser has parsed thus far, and provides it to you as a slice. A more optimal version of the solution that you have found @viperML is to do this:

    let string = just('"')
        .ignore_then(none_of('"').repeated().slice())
        .then_ignore(just('"'))
        .map(|s| Token::String(s.to_string()));

The reason not to use collect is that it makes a String by pushing the char's into an empty String one at a time. By using a combination of repeated().slice() and str::to_string we ensure that only a single allocation is made.

PS: @zaynetro The first ignored of your solution I believe is unnecessary, as a .repeated() not followed by a .collect() call automatically returns () :)

[zesterer]

Personally, this would be my desired solution.

let string = any().filter(|c| *c != '"').repeated()
    .map_slice(|s| Token::String(String::from(s)))
    .delimited_by(just('"'), just('"'));

The behaviour is no different to other solutions in this thread though :)

Something worth noting is that you can actually avoid allocating entirely by having your token store a string slice that refers back to the original source, like so:

enum Token<'a> {
    Number(i32),
    Bind,
    String(&'a str),
}

...

let string = any().filter(|c| *c != '"').repeated()
    .map_slice(Token::String)
    .delimited_by(just('"'), just('"'));

Granted, you may not want to do this if you find it inconvenient to have a lifetime parameter attached to your token type.