Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Global db-cache invalidation on mutation #3155

Open
porunov opened this issue Jul 25, 2022 · 0 comments · May be fixed by #3581
Open

Global db-cache invalidation on mutation #3155

porunov opened this issue Jul 25, 2022 · 0 comments · May be fixed by #3581
Labels
kind/enhancement
Milestone
Release v1.1.0

Comments

@porunov
Copy link
Member

porunov commented Jul 25, 2022 •
edited

Partially discussed in Invalidation topic here: https://lists.lfaidata.foundation/g/janusgraph-dev/topic/discussion_janusgraph/93326202

Currently we invalidate db-cache only in the next 3 scenarios:

  1. Enough time passed (cache.db-cache-time).
  2. Evicted due to cache size limitation (cache.db-cache-time).
  3. Evicted on current JanusGraph instance only due to being mutated on the current JanusGraph instance.

I would propose adding a feature which adds a possibility to evict JanusGraph cache on all JanusGraph instances during mutation on any of the instances (something like cache.db-cache-global-eviction = true).
This would be a good feature for situations when there are no many writes but many reads. Of course, if there are too many writes this would pressure other JanusGraph nodes which isn't good but then this feature shouldn't be used in such use-cases.

I didn't think on how to implement this feature but I guess maybe reusing JanusGraph messaging mechanism could be a good start.
porunov added a commit to porunov/janusgraph that referenced this issue Aug 26, 2022
@porunov
Add possibility to manual db-cache invalidation
d688d82 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov mentioned this issue Aug 26, 2022
Merged
9 tasks
porunov added a commit to porunov/janusgraph that referenced this issue Aug 26, 2022
@porunov
Add possibility to manual db-cache invalidation
d462ea0 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Aug 26, 2022
@porunov
Add possibility to manual db-cache invalidation
432075f 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Aug 31, 2022
@porunov
Add possibility to manual db-cache invalidation
011160e 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Aug 31, 2022
@porunov
Add possibility to manual db-cache invalidation
8d4cdbf 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Sep 1, 2022
@porunov
Add possibility to manual db-cache invalidation
87d1778 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Sep 2, 2022
@porunov
Add possibility to manual db-cache invalidation
9e77576 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Sep 22, 2022
@porunov
Add possibility to manual db-cache invalidation
64c6239 
Related to JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit that referenced this issue Oct 22, 2022
@porunov
Add possibility to manual db-cache invalidation
7ac198f 
Related to #3155

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov mentioned this issue Oct 24, 2022
Closed
porunov added a commit to porunov/janusgraph that referenced this issue Oct 24, 2022
@porunov
Send vertex label to user log processor for all mutations
dff3e2c 
Fixes JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov mentioned this issue Oct 24, 2022
Closed
9 tasks
porunov added a commit to porunov/janusgraph that referenced this issue Nov 7, 2022
@porunov
Add possibility to invalidate cached indices with indexOnly constraint
a671b8d 
Related JanusGraph#3155 JanusGraph#3263 JanusGraph#3264

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov mentioned this issue Nov 7, 2022
Merged
9 tasks
porunov added a commit that referenced this issue Nov 7, 2022
@porunov
Add possibility to invalidate cached indices with indexOnly constraint
d039caa 
Related #3155 #3263 #3264

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov added this to the Release v1.0.0 milestone Nov 7, 2022
porunov added a commit to porunov/janusgraph that referenced this issue Feb 13, 2023
@porunov
Add log processor for global db-cache invalidation
5c1d7a0 
TODO:
- Fix `buildCacheInvalidationChangeProcessor` to be able to invalidate updated vertices which are already removed by the time processor is started
- Add parameter to JanusGraph which will allow automatically starting this log processor
- Cover added logic with tests
- Document global db-cache invalidation logic (like: lag time between transaction is committed and cache entry is evicted, log configuration for db-cache invalidation log processor, transactions usage for it to work)

Fixes JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
porunov added a commit to porunov/janusgraph that referenced this issue Feb 13, 2023
@porunov
Add log processor for global db-cache invalidation
d7747eb 
TODO:
- Fix `buildCacheInvalidationChangeProcessor` to be able to invalidate updated vertices which are already removed by the time processor is started
- Add parameter to JanusGraph which will allow automatically starting this log processor
- Cover added logic with tests
- Document global db-cache invalidation logic (like: lag time between transaction is committed and cache entry is evicted, log configuration for db-cache invalidation log processor, transactions usage for it to work)

Fixes JanusGraph#3155

Signed-off-by: Oleksandr Porunov <[email protected]>
@porunov porunov linked a pull request Feb 13, 2023 that will close this issue
Draft
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement
Projects
None yet
Milestone
Release v1.1.0
Development

Successfully merging a pull request may close this issue.

Add log processor for global db-cache invalidation porunov/janusgraph
1 participant
@porunov