Consider using the spaCy tokenizer

[koaning]

When looking at the config.yml I read:

  - name: "SpacyEntityExtractor"
    # Note: It is not possible to use the SpacyTokenizer + SpacyFeaturizer in 
    #       combination with the WhitespaceTokenizer, and as a result the
    #       PERSON extraction by Spacy is not very robust.
    #       Because of this, the nlu training data is annotated as well, and the
    #       DIETClassifier will also extract PERSON entities.

This reads as a fair warning, but it also begs the question: why aren't we using the spaCy tokenizer here? I'll gladly make a PR but I'm curious if there's something that I'm currently not considering.

[ArjaanBuijk]

@koaning ,
Great question!
When using the spaCy tokenizer, the performance of the time/duration entity extraction by duckling goes down.
That was the reason not to switch over to the spaCy tokenizer, but it definitely deserves a re-evaluation.

[koaning]

Ah yeah. This was brought up during a research meeting a while ago.

I ended up making an experimental alternative to Duckling, should you be interested. It's part of rasa-nlu-examples. If you're exploring alternatives to Duckling, I'm all ears to any feedback.