Name		Name	Last commit message	Last commit date
parent directory ..
doc		doc
grammars		grammars
test		test
text		text
.gitignore		.gitignore
README.md		README.md
S1.gr		S1.gr
S2.gr		S2.gr
Vocab.gr		Vocab.gr
allowed_words.txt		allowed_words.txt
cgw.ipynb		cgw.ipynb
check-sample.py		check-sample.py
draw_tree.py		draw_tree.py
generate_grammar_from_tree.sh		generate_grammar_from_tree.sh
moreSentences.py		moreSentences.py
parse.ipynb		parse.ipynb
parse_tree.py		parse_tree.py
pcfg_parse_gen.py		pcfg_parse_gen.py
print_tree.py		print_tree.py
requirements.txt		requirements.txt
runtest.sh		runtest.sh
s2.py		s2.py
s2_unigram.py		s2_unigram.py
unseen.tags		unseen.tags
vocab_generator.py		vocab_generator.py

README.md

HW1

Edit input tree location & output grammar locations inside generate_grammar_from_tree.sh Then run the script

./generate_grammar_from_tree.sh

Non terminals are supposed to be all upper case letters, while terminals are supposed to be not all upper case letters. but some terminals have one single upper case letter. For example 'B' is a terminal. This makes it hard to tell vocabularies(terminals) from non terminals.
Parenthesis exists in the given text. For example in line 410 of devset.txt: ( Fetchez la vache . ) wha ?. This results in unmatched parenthesis in devset.tree and empty non terminal and terminal in the parsed grammar. We should consider substitute the parenthesis with other symbol when generating grammar trees. Seems like it is solved by our text parser that generates grammar trees.
How to test S2.gr? Based on some experiments, when the train set is large enough, and all words are added to Vocab.gr, all sentences can be handled by S1.gr