* Shard transfer progress

* Add test

* Round to two decimals

* Use ringbuffer crate

* Text-based human readable report

* Introduce JsonPathString

* Fix fomatting

* Add first stubs for WAL delta shard transfer method

* Repurpose queue proxy, use it for transferring WAL diff as well

* Integrate WAL delta transfer is transfer selection logic

* Add WalDelta shard transfer type which is not exposed in public API

* Basic implementation of falling back to stream records transfer

* Share await_consensus_sync function

* Ask remote shard for recovery point

* During WAL delta transfer, resolve shard diff locally for recovery point

* Rebase on latest dev, support empty WAL diff

* Rebase on latest dev, support empty WAL diff

* Use partial snapshot state for WAL delta transfer

* Set cutoff point on remote shard after shard WAL delta transfer

* Set cutoff point on remote shard after stream records transfer

* During WAL delta transfer, set s tate from partial snapshot to partial

* Describe WAL delta transfer in a comment

* Allow updating cutoff point in stream records transfer to fail

* Do not set cutoff point on remote shard on WAL delta transfer

* Make await consensus sync logic easier to read and reason about

* Fix fallback to other shard transfer method on WAL delta transfer fail

* Various minor improvements

* Add TODO for just ignoring API unimplemented errors

* Only allow stream records cutoff point error if remote is older version

* Allow switching to partial to fail when falling back

* Only change shard state to partial if not in partial state already

* Change default shard transfer method back to stream records

* Add important TODO back

* Add WAL delta shard transfer method in gRPC

* Prefer configured shard transfer method as default

* Extract shard transfer fallback logic into separate function

…o` (qdrant#3677)

Introduced in qdrant#3510

* Add WAL delta transfer integration test for empty delta

* Report size of WAL delta, assert this in integration tests

* Add WAL delta transfer integration test for shard recovery

* Add WAL delta transfer integration test for falling back

* Minor improvements

* Fix typo

* Fix data race in WAL delta integration test, kill upsertions on count

* In WAL delta tests, compare all points, vectors and payload exactly

* add test for fallback in into straming if diff is big

---------

Co-authored-by: generall <[email protected]>

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.196 to 1.0.197.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.196...v1.0.197)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.113 to 1.0.114.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.113...v1.0.114)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [ahash](https://github.com/tkaitchuck/ahash) from 0.8.8 to 0.8.9.
- [Release notes](https://github.com/tkaitchuck/ahash/releases)
- [Commits](tkaitchuck/aHash@v0.8.8...v0.8.9)

---
updated-dependencies:
- dependency-name: ahash
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [geo](https://github.com/georust/geo) from 0.27.0 to 0.28.0.
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md)
- [Commits](georust/geo@geo-0.27.0...geo-0.28.0)

---
updated-dependencies:
- dependency-name: geo
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Give file system some time clean up temporary snapshot files

* Be more careful with test_snapshot_operations_non_wait test

* Remove sleep

* compressed posting list definition

log compressed size

compress sorted

fix unit tests

flatten chunks

use visitor pattern to avoid multiple decompressions

check postings list boundaries

debug increasing order check

simplify ranges check

unit tests for visitor

remove debug size measurements

fix build

* more comments

* review remarks

* are you happy clippy

* don't compress remainder

* review remarks

* rename methods

…nt#3717)

* Cover mmap storages in compatibility tests

* regen archive from scratch

* update collection used for detecting up

* [snapshots]: Define trait for snapshots managemet

* Define a trait with methods for managing snapshots after their
  creation. The goal of this trait is to allow implementation of
  snapshot management logic for different storage backends without major
  changes in the code.

Signed-off-by: Alexandru Cihodaru <[email protected]>

* [snapshots]: Move snapshot deletion logic

* Move the snapshot deletion logic into the implementation of
  SnapshotStorage for LocalFileSystemConfig.
* Replace snapshot deletion logic with calls to the SnapshotStorage
  implementation.

Signed-off-by: Alexandru Cihodaru <[email protected]>

* [snapshots]: Move snapshot list logic

* Move the snapshot listing logic into the implementation of
  SnapshotStorage for LocalFileSystemConfig.

Signed-off-by: Alexandru Cihodaru <[email protected]>

* [snapshots]: Move snapshot creation logic

* Move the snapshot creation logic into the implementation of
  SnapshotStorage for LocalFileSystemConfig.

Signed-off-by: Alexandru Cihodaru <[email protected]>

* review fixes

* fmt

* refactor full storage snapshot

* create directory on get_stored_file + make sure temp archive file is removed

---------

Signed-off-by: Alexandru Cihodaru <[email protected]>
Co-authored-by: generall <[email protected]>

* Add consensus operation to restart shard transfer with different config

* Require shard transfer restart to have a changed configuration

* implement api

---------

Co-authored-by: generall <[email protected]>

* Switch to JsonPathV2

* Use new safe_to_set/safe_to_remove functions, fix PayloadIndex::assign

* minor review fixes

* Rename and inverse

---------

Co-authored-by: generall <[email protected]>

* wip: include datatype in vector params API

* generate api schemas

* propagate datatype to segment creation

* fix review

* fmt

* Add simple `ProxySegment::flush` unit-test

* fixup! Add simple `ProxySegment::flush` unit-test

Clarify comments explaining test cases

* Fix Arc unwrap function not respecting timeout

* Update lib/collection/src/collection_manager/holders/segment_holder.rs

Co-authored-by: Roman Titov <[email protected]>

* Cleanup `try_unwrap_with_timeout`

---------

Co-authored-by: Roman Titov <[email protected]>

* byte storage quantization fix and test

* apply quantization_preprocess in quantization scorers

* exact true

* exact true

* calculate sames count

* wrong distance getter

* fix Manhattan distance getter

* less acc check

* fix build

* update acc

* Median based TopK for sparse vectors scoring

* add test with identical scores

* Bump Rustls 0.22.2 to 0.22.4

* Bump Rustls 0.21.10 to 0.21.11

Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.6...v3.7)

---
updated-dependencies:
- dependency-name: idna
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…rant#4084)

* When reading with a filter, go over non-appendable segments first

* Use existing iterator, shortcutting on empty is not necessary

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.115 to 1.0.116.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](serde-rs/json@v1.0.115...v1.0.116)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.12.3 to 0.12.4.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](seanmonstar/reqwest@v0.12.3...v0.12.4)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.58 to 1.0.59.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](dtolnay/thiserror@1.0.58...1.0.59)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.30.10 to 0.30.11.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](GuillaumeGomez/sysinfo@v0.30.10...v0.30.11)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [serial_test](https://github.com/palfrey/serial_test) from 3.0.0 to 3.1.0.
- [Release notes](https://github.com/palfrey/serial_test/releases)
- [Commits](palfrey/serial_test@v3.0.0...v3.1.0)

---
updated-dependencies:
- dependency-name: serial_test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [io-uring](https://github.com/tokio-rs/io-uring) from 0.6.3 to 0.6.4.
- [Commits](https://github.com/tokio-rs/io-uring/commits)

---
updated-dependencies:
- dependency-name: io-uring
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [rmp-serde](https://github.com/3Hren/msgpack-rust) from 1.1.2 to 1.2.0.
- [Release notes](https://github.com/3Hren/msgpack-rust/releases)
- [Commits](https://github.com/3Hren/msgpack-rust/commits/rmp-serde/v1.2.0)

---
updated-dependencies:
- dependency-name: rmp-serde
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [thread-priority](https://github.com/iddm/thread-priority) from 0.16.0 to 1.0.0.
- [Commits](https://github.com/iddm/thread-priority/commits)

---
updated-dependencies:
- dependency-name: thread-priority
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.197 to 1.0.198.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.197...v1.0.198)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

helps in providing table metadata or properties