[ENH] full-text search index reader and writer #2178
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
| async fn commit_and_flush(self) -> Result<(), Box<dyn ChromaError>> { | ||
| // TODO should we be `await?`ing these? Or can we just return the futures? | ||
| self.posting_lists_blockfile_writer | ||
| .commit::<u32, &Int32Array>()? |
There was a problem hiding this comment.
Note: we no longer store the positional posting list directly as a value, but rather as (token, doc_id) key and positions value. I think this is strictly better?
There was a problem hiding this comment.
So
prefix: token
key: doc_id
positions: [pos]?
There was a problem hiding this comment.
I think thats fine, its functionally the same at storage. Can we remove the pos value type from storage then and only use it as a builder?
There was a problem hiding this comment.
That's the fun part -- it's already not in Storage! If you're talking about in-memory storage at least
| // for the character have been seen/used in the matching algorithm. By | ||
| // leaving them ordered per the query, we can stick to the more straightforward | ||
| // but less efficient matching algorithm. | ||
| // token_frequencies.sort_by(|a, b| a.1.cmp(&b.1)); |
There was a problem hiding this comment.
Should we do the work to sort by token frequency now? It'll take some time to get it right and test. This bug was lurking in the old implementation (yet another argument for property testing).
| } | ||
| } | ||
| Ok(()) | ||
| } | ||
|
|
||
| fn search(&mut self, query: &str) -> Result<Vec<i32>, Box<dyn ChromaError>> { | ||
| async fn commit_and_flush(self) -> Result<(), Box<dyn ChromaError>> { |
There was a problem hiding this comment.
I don't think we want this method at the index level, if you look at the higher level pattern we want commit and flush as two steps with a type safe transition
| } | ||
| if res.len() > 1 { | ||
| panic!("Multiple tokens found in frequencies blockfile"); |
| for position_for_doc_id in positions_for_doc_id.values() { | ||
| if position_for_doc_id - token_offset == *position { | ||
| for pos in positions.iter() { | ||
| if pos.unwrap() == position + token_offset { |
There was a problem hiding this comment.
If it's not it's an invariant violation. Changed to a Result
|
|
||
| #[derive(Error, Debug)] | ||
| pub enum FullTextIndexError { | ||
| #[error("Multiple tokens found in frequencies blockfile")] |
There was a problem hiding this comment.
I don't understand this error? (just missing something)
There was a problem hiding this comment.
The frequencies blockfile should have exactly zero or one entries for a given token. If it has multiple then something very bad has happened (constraint violation/data corruption)
| } | ||
| } | ||
|
|
||
| pub(crate) struct FullTextIndexFlusher { |
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustDocumentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?