[WIP] TokenPartitionTree refactoring / "Choice" nodes support #1328
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Objectives
Design decisions
TokenPartitionTree uses a variant for storing nodes instead of regular polymorphism
Main reason: I wanted to do something that would work without requiring bigger changes in code that uses TokenPartitionTree. Regular polymorphism would change node objects stored in Children list into pointers, which would require a lot of changes at once in the whole codebase.
With the use of a variant there was only one thing that required a change in the whole code base:
Children()
now returns pointer instead of reference, and can return nullptr. This is mostly change of.
into->
.nullptr
is handled mostly through the use ofis_leaf()
instead ofChildren()->empty()
.Minor reason: this will allow linear storage of child nodes in memory. I didn't do any benchmarks, so I can only guess whether this helps much.
Custom Variant implementation
I've implemented Variant class from scratch instead of using
std::variant
.Main reason: I wanted a possibility to "cast" pointer to object stored in variant into a pointer to the Variant holding it. This requires a guarantee that the variant's data member which stores the value is the first member, or at least access to this member in order to use
offsetof
. Neither is provided bystd::variant
.This possibility is used in TokenPartition node implementations (which are stored in a Variant) to "cast"
this
into pointer to Variant, which is then assigned as parent pointer in child nodes.