Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace recursion by loop to parse huge files #27

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

bbencina
Copy link

@bbencina bbencina commented Mar 19, 2024

When trying to parse huge bibtex files (e.g. one of the cryptobib files) recursion in the entries parser at src/parser.rs:368:51 exhausts the stack and causes an overflow. Try using e.g. backtrace_on_stack_overflow to verify this.

Example code:

extern crate nom_bibtex;

use std::fs;
use nom_bibtex::Bibtex;

fn main() {
    unsafe {
        backtrace_on_stack_overflow::enable()
    };
    let filename: &str = "crypto.bib";
    let buf = fs::read_to_string(filename).expect("Couldn't open file");
    let bib = Bibtex::parse(buf.as_str()).unwrap(); // <-- HERE
    println!("{:?}", bib); // or something
}

Using a loop instead resolves this issue (see diff). This also removes the need for the O(n) insert operation because the entries are parsed in the original order instead of reversed, so push can be used.

When parsing huge files (e.g. cryptobib) recursion exhausts the stack.
Using a loop instead resolves this.
@A6GibKm
Copy link
Contributor

A6GibKm commented Mar 19, 2024

Hello,

Would it be possible to add a test case here?

@bbencina
Copy link
Author

Absolutely, I can add a fixed copy of crypto.bib to samples/. Should I just test the parsing goes through?

String concatenation now works with bracketed strings as well. Switched
back to original crypto.bib file (as of now).
@bbencina
Copy link
Author

I also added the option to have bracketed strings in concatenation, so the crypto.bib file can be parsed without any fixes to it. The bibtex binary also accepts this syntax as valid.

Switched test file to original crypto.bib that can be found here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants