Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v0.37.x:state/kvindexer: port 0.34 query fix #77

Merged
merged 28 commits into from
Feb 8, 2023
Merged

v0.37.x:state/kvindexer: port 0.34 query fix #77

merged 28 commits into from
Feb 8, 2023

Conversation

jmalicevic
Copy link
Contributor

Original PR tendermint/tendermint#9876

This is a forward port of tendermint/tendermint#9759 to fix the querying behaviour where attributes should be returned only if they match within the same event. It will include fixes for the above mentioned PR: #76 and tendermint/tendermint#9877

jmalicevic and others added 4 commits December 14, 2022 13:27
@jmalicevic @ancazamfir
@sergio-mena @romac @thanethomson
state/kvindexer: associate event attributes with events (#9759)
b6d99be 
* Updated event sequencing and added query keyword

* code cosmetics

* Documentation update

* Added per event indexing and querying to txindexer

* rpc test fix

* Added support for older versions where event sequencing is not supported

* Added support for old versions to tx indexer

* Added RPC match flag, fixed bugs in tx indexer, added tests

* Removed reference to match.events from the docs

* Openapi update

* Added height deduplication
Co-authored-by: Thane Thomson <[email protected]>

Co-authored-by: Anca Zamfir <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
Co-authored-by: Romain Ruetschi <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>
@jmalicevic
Backport kvindexer fix
6fe2b13
@jmalicevic
Fixed match.events order in light client rpc
b9f5a5b
@mergify @cmwaters @thanethomson
cli: add --hard flag to rollback command to remove block as well (bac…
bd9ed68 
…kport #9261) (#9465)

* cli: add --hard flag to rollback command to remove block as well  (#9261)

Co-authored-by: Levi Aul <[email protected]>
(cherry picked from commit e84d43e)

* Fix lint

Signed-off-by: Thane Thomson <[email protected]>

Signed-off-by: Thane Thomson <[email protected]>
Co-authored-by: Callum Waters <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>
@jmalicevic jmalicevic added indexer wip Work in progress labels Jan 5, 2023
@jmalicevic jmalicevic self-assigned this Jan 5, 2023
@jmalicevic jmalicevic marked this pull request as ready for review January 5, 2023 16:47
@jmalicevic jmalicevic requested a review from a team as a code owner January 5, 2023 16:47
@thanethomson
Copy link
Contributor

So you'll merge the changes from #76 into this PR? If so please convert this to a draft PR until then.

Also, please change the target branch from v0.37.x to release/v0.37.1, as we are currently collecting all changes going into v0.37.1 on that release branch.

@jmalicevic
Copy link
Contributor Author

No, the changes from the above mentioned PRs are already in the code. Those are basically bugfixes to the original (merged into 0.34.x) PR for the kvindexer. I discovered them when working on the 0.37 port but I mentioned them in the comment so people know all the changes from 0.34 that are ported here.

lasarojc
lasarojc reviewed Jan 6, 2023
View reviewed changes
Copy link
Contributor

@lasarojc lasarojc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a first pass. I will do another early next week.

abci/example/kvstore/kvstore.go
Attributes: []types.EventAttribute{
{
Key: "foo",
Value: "100",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use something derived from Height as values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll clean this up a bit, you are right we could add the % condition in the event value. In principle, the only reason why I separated these two use cases is to make sure there are events that do not occur at every height.

abci/example/kvstore/kvstore.go Outdated Show resolved Hide resolved
abci/example/kvstore/persistent_kvstore.go Show resolved Hide resolved
docs/app-dev/indexing-transactions.md Outdated Show resolved Hide resolved
docs/app-dev/indexing-transactions.md Outdated Show resolved Hide resolved
rpc/client/rpc_test.go Show resolved Hide resolved
state/indexer/block/kv/kv.go Outdated Show resolved Hide resolved
state/indexer/block/kv/kv.go Show resolved Hide resolved
state/indexer/query_range.go Outdated Show resolved Hide resolved
state/txindex/kv/kv.go Show resolved Hide resolved
@jmalicevic jmalicevic changed the base branch from v0.37.x to release/v0.37.1 January 9, 2023 17:26
@jmalicevic jmalicevic changed the title state/kvindexer: porting from 0.34 query fix state/kvindexer: port 0.34 query fix to 0.37 Jan 9, 2023
@jmalicevic
Update CHANGELOG_PENDING.md
4cfc9cb 
Created entry for 0.37.1
lasarojc
lasarojc reviewed Jan 11, 2023
View reviewed changes
state/indexer/block/kv/kv.go

// Check whether in this iteration we have not found an overlapping height (tmpHeight == nil)
// or whether the events in which the attributed occurred do not match (first part of the condition)
if tmpHeight == nil || !bytes.Equal(tmpHeight, v) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On line 348, should it return a ctx.Err() along with the partial result?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, concerning ctx.Err(), why do you think should this be treated as an error? Now if we do get a context done before processing all the result, and do return the partial result (which we are in this case - we have not looped through all the temporary files), that might not be what we want.

state/indexer/query_range.go Show resolved Hide resolved
state/txindex/kv/kv.go

// If we have a query range over height and want to still look for
// specific event values we do not want to simply return all
// transactios in this height range. We remember the height range info
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

			// transactions in this height range. We remember the height range info

state/txindex/kv/kv.go
continue
}

txi.setTmpHashes(tmpHashes, it)
// Potentially exit early.
select {
case <-ctx.Done():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this cause ctx.Err() to be returned?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the logic here is that, if there is an error with the currently retrieved entry from the kvstore, we will ignore this entry and move on to the next one.

lasarojc
lasarojc previously approved these changes Jan 12, 2023
View reviewed changes
thanethomson and others added 2 commits January 12, 2023 07:13
@thanethomson
Update CODEOWNERS (#99)
ca08390 
Signed-off-by: Thane Thomson <[email protected]>

Signed-off-by: Thane Thomson <[email protected]>
@jmalicevic
security/p2p: prevent peers who errored being added to the peer_set (…
16ffabc 
…backport #9500) (#87)

* security/p2p: prevent peers who errored being added to the peer_set (#9500)

* Mark failed removal of peer to address security bug
@jmalicevic jmalicevic mentioned this pull request Jan 26, 2023
Closed
7 tasks
@jmalicevic jmalicevic changed the base branch from release/v0.37.1 to v0.37.x January 27, 2023 08:09
@jmalicevic jmalicevic dismissed lasarojc’s stale review January 27, 2023 08:09

The base branch was changed.

@jmalicevic
Copy link
Contributor Author

Rebased to 0.37.x. We need to merge the PRs from #213 to be able to review this PR (otherwise all those changes are included in this PR).

@jmalicevic jmalicevic changed the title state/kvindexer: port 0.34 query fix to 0.37 v0.37.x:state/kvindexer: port 0.34 query fix Jan 31, 2023
@jmalicevic jmalicevic merged commit 21c00c5 into v0.37.x Feb 8, 2023
@jmalicevic jmalicevic deleted the jasmina/kvindexer-eventq-0.37 branch February 8, 2023 22:26
JayT106 pushed a commit to JayT106/cometbft that referenced this pull request Feb 10, 2023
@cmwaters @milosevic
@tac0turtle @melekes @ancazamfir @dongsam @sunnya97 @ebuchman @liamsi @erikgrinaker @tessr @xla @konnov @brapse @dependabot @josef-widder @andrey-kuprianov @hxrts @robert-zaremba @nano-o @shahankhatch @ValarDragon @istoilkovska @alexanderbez @tychoish @fedekunze @jordaaash @xiangjianmeng @luohaha @crypto-facs @williambanfield @goral09 @thanethomson
spec: migrate v0.7.1. into v0.34 (#9262)
1f0cf77 
* Initial commit

* Add three timeouts and align pseudocode better with existing algorithm

* Align protocol with Tendermint code and add find valid value mechanism

* Prepare to Nuke Develop (cometbft#47)

* state -> step

* vote -> v

* New version of the algorithm and the proof

* New version of the algorithm and the proofs

* Added algorithm description

* Add algorithm description

* Add introduction

* Add conclusion

* Add conclusion file

* fix warnings (caption was defined twice)

- only the latter is used anyways (centers captions)
- this makes it possible to autom. building the paper

* Update grammar

* s/state_p/step_p

* Address Ismail's comments

* intro: language fixes

* definitions: language fixes

* consensus: various fixes

* proof: some fixes

* try to improve reviewability

* \eq -> =

* textwrap to 79

* various minor fixes

* proof: fix itemization

* proof: more minor fixes

* proof: timeouts are functions

* proof: fixes to lemma6

* Intro changes and improve title page

* Add Marko and Ming to acks

* add readme

* Format algorithm correctly

Clarify condition semantic and timeouts

Improve descriptions

* patform -> platform

* Ensure that rules are mutually exclusive

- various clarifications and small improvements

* Release v0.6

* small nits for smoother readability

* This PR is to create signed commits to be able to merge (cometbft#50)

Signed-off-by: Marko Baricevic <[email protected]>

* Add consesnus and blockchain specs, (cometbft#52)

- Open questions
	- Do  we want to split lite client work from consesnsus
	- From the blockchain spec, is encoding nessecary in the spec

Signed-off-by: Marko Baricevic <[email protected]>

* Add ABCI SPEC (cometbft#51)

- move the abci spec from tendermint to spec repo

Signed-off-by: Marko Baricevic <[email protected]>

* spec/consensus/signing: add more details about nil and amnesia (cometbft#54)

- Add more details about nil votes and about amnesia attacks

Signed-off-by: Marko Baricevic <[email protected]>

* Add Section for P2P (cometbft#53)

* Add Section for P2P

- moved over the section on p2p

Signed-off-by: Marko Baricevic <[email protected]>

* add some more files

Signed-off-by: Marko Baricevic <[email protected]>

* Fix model section

* Add non-recursive specification of Bisection algorithm

- Fix timing issues by introducing Delta parameter

* spec: update spec with tendermint updates (cometbft#62)

* spec: update spec with tendermint updates

- this in preperation of deleting the spec folder in docs in tendermint/tendermint

Signed-off-by: Marko Baricevic <[email protected]>

* spec: added in reactors & p2p

Signed-off-by: Marko Baricevic <[email protected]>

* spec: update readme in spec to comply with docs site

Signed-off-by: Marko Baricevic <[email protected]>

* docs: addded more changes from tednermint/tendermint

Signed-off-by: Marko Baricevic <[email protected]>

* reflect breaking changes made to Commit (cometbft#63)

* reflect breaking changes made to Commit

PR: tendermint/tendermint#4146
Issue: tendermint/tendermint#1648

* types: rename Commit#Precommits to Signatures

* update BlockIDFlagAbsent comment

* remove iota

* Clean up error conditions and simplify pseudocode

* Apply suggestions from code review

Co-Authored-By: Anca Zamfir <[email protected]>

* Add spec doc about unconditional_peer, persistent_peers_max_dial of ADR-050 (cometbft#68)

* Add spec doc about unconditional_peer_ids, persistent_peers_max_dial_period of ADR-050

* Add indefinitely dialing condition

* Add sr25519 amino documentation (cometbft#67)

* sr25519 amino

* Update spec/blockchain/encoding.md

Co-Authored-By: Marko <[email protected]>

* some suggestions for pseuodocode changes

* Improved error handling

* Add explanation on difference between trusted models

* Address reviewer's comments

* Addressing reviewer's comments

* Separating algorithm from proofs

* Intermediate commit (aligning spec with the code)

* Removing Store from API and providing end-to-end timing guarantees

* Address reviewer comment's. Intermediate commit

* light client dir and readmes

* titles

* add redirects

* add diagram

* detection TODO

* fix image

* update readme

* Aligh the correctness arguments with the pseudocode changes

* lite->light

* Fix link in readme

./light -> ./light-client

* p2p: Merlin based malleability fixes (cometbft#72)

* Update the secret connection spec with the use of merlin to eliminte handshake malleability

* Update spec/p2p/peer.md

Co-Authored-By: Anton Kaliaev <[email protected]>

* Update spec/p2p/peer.md

Co-Authored-By: Anton Kaliaev <[email protected]>

* Update spec/p2p/peer.md

Co-Authored-By: Anton Kaliaev <[email protected]>

Co-authored-by: Anton Kaliaev <[email protected]>

* docs: update specs to remove cmn (cometbft#77)

- cmn was remvoed in favor of sub pkgs. cmn.kvpair is now kv.pair

Signed-off-by: Marko Baricevic <[email protected]>

* evidence: Add time to evidence params (cometbft#69)

* evidence: Add time to evidence params

- this pr is grouped together with tendermint/tendermint#4254, once that PR is merged then this one can be as well.

Signed-off-by: Marko Baricevic <[email protected]>

* remove note

Signed-off-by: Marko Baricevic <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Anton Kaliaev <[email protected]>

Co-authored-by: Anton Kaliaev <[email protected]>

* update link to the pex reactor

* add markdown link checker

* changed tab spacing

* removed folder-path flag

* first attempt at fixing all links

* second attempt at fixing all links

* codeowners: add code owners (cometbft#82)

* codeowners: add code owners

- added some codeowners
please comment if youd like to be added as well.

Signed-off-by: Marko Baricevic <[email protected]>

* remove comment of repo maintainers

* remove .idea dir (cometbft#83)

Signed-off-by: Marko Baricevic <[email protected]>

* RFC-001: configurable block retention (cometbft#84)

* Added RFC for truncated block history coordination

* Clarified minimum block retention

* Added hard checks on block retention and snapshot interval, and made some minor tweaks

* Genesis parameters are immutable

* Use local config for snapshot interval

* Reordered parameter descriptions

* Clarified local config option for snapshot-interval

* rewrite for ABCI commit response

* Renamed RFC

* add block retention diagram

* Removed retain_blocks table

* fix image numbers

* resolved open questions

* image quality

* accept RFC-001 (cometbft#86)

* abci: add basic description of ABCI Commit.ResponseHeight (cometbft#85)

Documentation for block pruning, once it's merged: tendermint/tendermint#4588.

Minimum documentation, for now - we probably shouldn't encourage using this feature too much until we release state sync.

* abci: add MaxAgeNumBlocks/MaxAgeDuration to EvidenceParams (cometbft#87)

* abci: update MaxAgeNumBlocks & MaxAgeDuration docs (cometbft#88)

* document state sync ABCI interface and P2P protocol (cometbft#90)

The corresponding Tendermint PRs are tendermint/tendermint#4704 and tendermint/tendermint#4705.

* Revert "document state sync ABCI interface and P2P protocol (cometbft#90)" (cometbft#92)

This reverts commit 9842b4b.

* blockchain: change validator set sorting method (cometbft#91)

* abci: specify sorting of RequestInitChain.Validators

* blockchain: change validator sorting method

Refs tendermint/tendermint#2478

* reactors/pex: specify hash function (cometbft#94)

https://github.com/tendermint/tendermint/pull/4810/files

* document state sync ABCI interface and P2P protocol (cometbft#93)

* Revert "Revert "document state sync ABCI interface and P2P protocol (cometbft#90)" (cometbft#92)"

This reverts commit 90797ce.

* update with new enum case

* fix links

Co-authored-by: Erik Grinaker <[email protected]>

* Update evidence params with MaxNum (cometbft#95)

evidence params now includes maxNum which is the maximum number of evidence that can be committed on a single block

* reactors/pex: masked IP is used as group key (cometbft#96)

* spec: add ProofTrialPeriod to EvidenceParam (cometbft#99)

* spec: modify Header.LastResultsHash (cometbft#97)

Refs: tendermint/tendermint#1007
PR: tendermint/tendermint#4845

* spec: link to abci server implementations (cometbft#100)

* spec: update evidence in blockchain.md (cometbft#108)

now evidence reflects the actual evidence present in the tendermint repo

* abci: add AppVersion to ConsensusParams (cometbft#106)

* abci: tweak node sync estimate (cometbft#115)

* spec/abci: expand on Validator#Address (cometbft#118)

Refs tendermint/tendermint#3732

* blockchain: rename to core (cometbft#123)

* blockchain: remove duplicate evidence sections (cometbft#124)

* spec/consensus: canonical vs subjective commit

Refs tendermint/tendermint#2769

* Apply suggestions from code review

Co-authored-by: Igor Konnov <[email protected]>

* update spec with the removal of phantom validator evidence (cometbft#126)

* bring blockchain back

* add correct links

* spec: revert event hashing (cometbft#132)

* Evidence time is sourced from block time (cometbft#138)

* RFC-002: non-zero genesis (cometbft#119)

* abci: add ResponseInitChain.app_hash (cometbft#140)

* update hashing of empty inputs, and initial block LastResultsHash (cometbft#141)

* update evidence verification (cometbft#139)

* accept RFC-002 (cometbft#142)

* add description of arbitrary initial height (cometbft#135)

* update ResponseInitChain.app_hash description (cometbft#143)

* remove unused directories and update README (cometbft#145)

This change removes unused directories (`papers` and `research`) 
and updates the README to reflect our strategy for merging the 
informalsystems/tendermint-rs specs into this repository.

Partially addresses cometbft#121.

* ci: add markdown linter (cometbft#146)

* ci: add dependabot config (cometbft#148)

* build(deps): bump gaurav-nelson/github-action-markdown-link-check from 0.6.0 to 1.0.7 (cometbft#149)

Bumps [gaurav-nelson/github-action-markdown-link-check](https://github.com/gaurav-nelson/github-action-markdown-link-check) from 0.6.0 to 1.0.7.

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* docs: add sections to abci (cometbft#150)

* spec: update abci events (cometbft#151)

* spec: extract light-client to its own directory (cometbft#152)

Co-authored-by: Callum Waters <[email protected]>

* spec: remove evidences (cometbft#153)

* add a stale bot (cometbft#134)

* Current versions of light client specs from tendermint-rs (cometbft#158)

* current versions of light client specs from tendermint-rs

* markdown lint

* linting

* links

* links

* links

Co-authored-by: Marko Baricevic <[email protected]>

* Fastsync spec from tendermint-rs (cometbft#157)

* fastsync spec from tendermint-rs

* fixed broken link

* fixed linting

* more fixes

* markdown lint

* move fast_sync to rust-spec

Co-authored-by: Marko Baricevic <[email protected]>

* Update README.md (cometbft#160)

* spec/reactors/mempool: batch txs per peer (cometbft#155)

* spec/reactors/mempool: batch txs per peer

Refs tendermint/tendermint#625

* update

* spec: Light client attack detector (cometbft#164)

* start with new detection and evidence spec

* more definitions at top

* sketch of functions

* pre post draft

* evidence proof

* typo

* evidence theory polished

* some TODOs resolved

* more TODOs

* links

* second to last revision before PR

* links

* I will read once more and then make a PR

* removed peer handling definitions

* secondary

* ready to review

* detector ready for review

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* Update rust-spec/lightclient/detection/detection.md

* skip-trace

* PossibleCommit explained

* Update rust-spec/lightclient/detection/detection.md

Co-authored-by: Zarko Milosevic <[email protected]>

* comments by Zarko

* renamed and changed link in README

Co-authored-by: Zarko Milosevic <[email protected]>

* fixed an overlooked conflict (cometbft#167)

* describe valset sorting according to v0.34 requirements (cometbft#169)

* evidence: update data structures (cometbft#165)

* fix markdown linter (cometbft#172)

* TLA+ specs from MBT revision (cometbft#173)

* remove setOption (cometbft#181)

* spec: protobuf changes (cometbft#156)

Co-authored-by: Anton Kaliaev <[email protected]>

* first check latest with secondary (cometbft#184)

* Extending the blockchain specification (in the light client) to produce different ratios of faults (cometbft#183)

* cleaning unused definitions

* introduced the ratio of faulty processes

* Update README.md (cometbft#185)

* build(deps): bump gaurav-nelson/github-action-markdown-link-check from 1.0.7 to 1.0.8 (cometbft#188)

Bumps [gaurav-nelson/github-action-markdown-link-check](https://github.com/gaurav-nelson/github-action-markdown-link-check) from 1.0.7 to 1.0.8.
- [Release notes](https://github.com/gaurav-nelson/github-action-markdown-link-check/releases)
- [Commits](gaurav-nelson/github-action-markdown-link-check@1.0.7...e3c371c)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* spec: update light client verification to match supervisor (cometbft#171)

* VDD renaming of verification spec + links fixed

* latest()

* backwards

* added TODOs

* link in old file to new name

* better text

* revision done. needs one more round of reading

* renamed constants in 001 according to TLA+ and impl

* ready for PR

* forgot linting

* Update rust-spec/lightclient/verification/verification_002_draft.md

* Update rust-spec/lightclient/verification/verification_002_draft.md

* added lightstore function needed for supervisor

* added lightstore functions for supervisor

* ident

* Update rust-spec/lightclient/verification/verification_002_draft.md

* github: issue template for proposals (cometbft#190)

* Sequential Supervisor (cometbft#186)

* move from tendermint-rs but needs discussion

* markdown lint

* TODO links replaced

* links

* links

* links lint

* Update rust-spec/lightclient/supervisor/supervisor.md

* Update rust-spec/lightclient/supervisor/supervisor.md

* Update rust-spec/lightclient/supervisor/supervisor.md

* Update rust-spec/lightclient/supervisor/supervisor.md

* moved peer handling definitions to supervisor

* polishing

* rename

* Update rust-spec/lightclient/supervisor/supervisor_001_draft.md

* Update rust-spec/lightclient/supervisor/supervisor_001_draft.md

* changes to maintain StateVerified again

* ready for changes in verification

* start of supervisor

* module name

* fixed

* more details

* supevisor completed. Now I have to add function to verification

* ready for review

* tla comment

* removed issues

* Update rust-spec/lightclient/supervisor/supervisor_001_draft.md

* intro text fixed

* indentation

* Update rust-spec/lightclient/supervisor/supervisor_001_draft.md

* comment to entry points

Co-authored-by: Marko Baricevic <[email protected]>

* RFC: adopt zip 215 (cometbft#144)

Co-authored-by: Robert Zaremba <[email protected]>

* Core: move validation & data structures together (cometbft#176)

Co-authored-by: Callum Waters <[email protected]>

* docs: make blockchain not viewable (cometbft#211)

* evidence: update data structures to reflect added support of abci evidence (cometbft#213)

* encoding: add secp, ref zip215, tables (cometbft#212)

* Detector English Spec ready (cometbft#215)

Add detector English spec

* add Ivy proofs (cometbft#210)

* add Ivy proofs

* fix docker-compose command

* Light client detector spec in TLA+ and refactoring of light client verification TLA+ spec (cometbft#216)

Add light client detector spec in TLA+

* abci: lastcommitinfo.round extra sentence (cometbft#221)

* abci: add abci_version to requestInfo (cometbft#223)

* BFT requires _less than_ 1/3 faulty validators (cometbft#228)

Thanks fo spotting the imprecision in the text, @shahankhatch !

* Draft of evidence handling for discussion (cometbft#225)

* start with accountability deliverable

* problem statement

* draft function

* quite complete draft. ready to discuss with Igor

* Update isolate-attackers_001_draft.md

* Update isolate-attackers_001_draft.md

* Update isolate-attackers_001_draft.md

* Update isolate-attackers_001_draft.md

* Update isolate-attackers_001_draft.md

* ready for TLA+ to take over

* isolate

* isolateamnesiatodos

* Update isolate-attackers_001_draft.md

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* Update rust-spec/lightclient/attacks/isolate-attackers_001_draft.md

Co-authored-by: Igor Konnov <[email protected]>

* The TLA+ specification of the attackers detection (cometbft#231)

* the working attackers isolation spec, needs more comments

* the TLA+ spec of the attackers isolation

* build(deps): bump gaurav-nelson/github-action-markdown-link-check (cometbft#233)

Bumps [gaurav-nelson/github-action-markdown-link-check](https://github.com/gaurav-nelson/github-action-markdown-link-check) from 1.0.8 to 1.0.11.
- [Release notes](https://github.com/gaurav-nelson/github-action-markdown-link-check/releases)
- [Commits](gaurav-nelson/github-action-markdown-link-check@1.0.8...2a60e0f)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Computing attack types (cometbft#232)

Add light attack evidence handling

* Update README.md (cometbft#234)

* p2p: update frame size (cometbft#235)

Reflect the change made in tendermint/tendermint#5805

The MTU (Maximum Transmission Unit) for Ethernet is 1500 bytes.
The IP header and the TCP header take up 20 bytes each at least (unless
optional header fields are used) and thus the max for (non-Jumbo frame)
Ethernet is 1500 - 20 -20 = 1460
Source: https://stackoverflow.com/a/3074427/820520

* build(deps): bump gaurav-nelson/github-action-markdown-link-check (cometbft#239)

Bumps [gaurav-nelson/github-action-markdown-link-check](https://github.com/gaurav-nelson/github-action-markdown-link-check) from 1.0.11 to 1.0.12.
- [Release notes](https://github.com/gaurav-nelson/github-action-markdown-link-check/releases)
- [Commits](gaurav-nelson/github-action-markdown-link-check@1.0.11...0fe4911)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* layout: add section titles (cometbft#240)

* reactors: remove bcv1 (cometbft#241)

* abci: rewrite to proto interface (cometbft#237)

* Update supervisor_001_draft.md (cometbft#243)

* spec: remove reactor section (cometbft#242)

Co-authored-by: Tess Rinearson <[email protected]>

* non-critical bugfix in the TLA+ spec (found by new version of apalache) (cometbft#244)

* params: remove block timeiota (cometbft#248)

* proto: add files (cometbft#246)

Co-authored-by: Erik Grinaker <[email protected]>

* proto: modify height int64 to uint64 (cometbft#253)

* abci: note on concurrency (cometbft#258)

Co-authored-by: Marko <[email protected]>

* spec: merge rust-spec (cometbft#252)

* Fix list of RFCs (cometbft#266)

* readme: cleanup (cometbft#262)

* modify readme

* add rfc and proto

* add rust=spec back to avoid breakage

* lint readme

* genesis: Explain fields in genesis file (cometbft#270)

* describe the genesis

* Update spec/core/genesis.md

Co-authored-by: Dev Ojha <[email protected]>

* Apply suggestions from code review

Co-authored-by: Callum Waters <[email protected]>

* add wording on app_state

* Update spec/core/genesis.md

Co-authored-by: Callum Waters <[email protected]>

Co-authored-by: Dev Ojha <[email protected]>
Co-authored-by: Callum Waters <[email protected]>

* p2p: links (cometbft#268)

* fix links

* fix more links

* Proposer-based timestamp specification (cometbft#261)

* added proposer-based timestamp spec

* Update spec/consensus/proposer-based-timestamp/pbts_001_draft.md

Co-authored-by: Aleksandr Bezobchuk <[email protected]>

* Update spec/consensus/proposer-based-timestamp/pbts_001_draft.md

Co-authored-by: Aleksandr Bezobchuk <[email protected]>

* Update spec/consensus/proposer-based-timestamp/pbts-algorithm_001_draft.md

Co-authored-by: Marko <[email protected]>

* Update spec/consensus/proposer-based-timestamp/pbts-algorithm_001_draft.md

* Update spec/consensus/proposer-based-timestamp/pbts-sysmodel_001_draft.md

Co-authored-by: Callum Waters <[email protected]>

* fixes from PR

Co-authored-by: Josef Widder <[email protected]>
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
Co-authored-by: Marko <[email protected]>
Co-authored-by: Callum Waters <[email protected]>

* abci: reorder sidebar (cometbft#282)

* ABCI++ RFC (cometbft#254)

* ABCI++ RFC

This commit adds an RFC for ABCI++, which is a collection of three new phases of communication between the consensus engine and the application.

Co-authored-by: Sunny Aggarwal <[email protected]>

* Fix bugs pointed out by @liamsi

* Update rfc/004-abci++.md

Co-authored-by: Federico Kunze <[email protected]>

* Fix markdown lints

* Update rfc/004-abci++.md

Co-authored-by: Ismail Khoffi <[email protected]>

* Update rfc/004-abci++.md

Co-authored-by: Tess Rinearson <[email protected]>

* Update rfc/004-abci++.md

Co-authored-by: Tess Rinearson <[email protected]>

* Add information about the rename in the context section

* Bold RFC

* Add example for self-authenticating vote data

* More exposition of the term IPC

* Update pros / negatives

* Fix sentence fragment

* Add desc for no-ops

Co-authored-by: Sunny Aggarwal <[email protected]>
Co-authored-by: Federico Kunze <[email protected]>
Co-authored-by: Ismail Khoffi <[email protected]>
Co-authored-by: Tess Rinearson <[email protected]>

* RFC: ReverseSync - fetching historical data (cometbft#224)

* core: update a few sections  (cometbft#284)

* p2p: update state sync messages for reverse sync (cometbft#285)

* Update README.md (cometbft#286)

* rpc: define spec for RPC (cometbft#276)

* add rpc spec and support outline

* add json

* add more routes remove unneeded ones

* add rest of rpc endpoints

* add jsonrpc calls

* add more jsonrpc calls

* fix blockchain

* cleanup unused links and add links to repos

* Update spec/rpc/README.md

Co-authored-by: Callum Waters <[email protected]>

* add missing param from consensus param

* Update spec/rpc/README.md

Co-authored-by: Callum Waters <[email protected]>

* Update spec/rpc/README.md

Co-authored-by: Callum Waters <[email protected]>

* fix cast and add doc to readme

Co-authored-by: Callum Waters <[email protected]>
Co-authored-by: Marko Baricevic <[email protected]>

* A few improvements to the Ivy proof (cometbft#288)

* Avoid quantifier alternation cycle

The problematic quantifier alternation cycle arose because the
definition of accountability_violation was unfolded.

This commit also restructures the induction proof for clarity.

* add count_lines.sh

* fix typo and add forgotten complete=fo in comment

Co-authored-by: Giuliano <[email protected]>

* Fixed a broken link (cometbft#291)

* fix message type for block-sync (cometbft#298)

* lint: fix lint errors (cometbft#301)

* build(deps): bump actions/stale from 3 to 3.0.18 (cometbft#300)

Bumps [actions/stale](https://github.com/actions/stale) from 3 to 3.0.18.
- [Release notes](https://github.com/actions/stale/releases)
- [Commits](actions/stale@v3...v3.0.18)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump actions/stale from 3.0.18 to 3.0.19 (cometbft#302)

Bumps [actions/stale](https://github.com/actions/stale) from 3.0.18 to 3.0.19.
- [Release notes](https://github.com/actions/stale/releases)
- [Commits](actions/stale@v3.0.18...v3.0.19)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* rename HasVote to ReceivedVote (cometbft#289)

* add a changelog to track changes (cometbft#303)

* add a changelog to track changes

* Update CHANGELOG.md

Co-authored-by: Callum Waters <[email protected]>

Co-authored-by: Callum Waters <[email protected]>

* rpc: clarify timestamps (cometbft#304)

* clarify timestamps

* changelog entry

* Update spec/rpc/README.md

Co-authored-by: Callum Waters <[email protected]>

Co-authored-by: Callum Waters <[email protected]>

* rpc: add chunked genesis endpoint (cometbft#299)

* rpc: add chunked genesis endpoint

* fix lint

* feedback

* add info about error

* fix lint

Co-authored-by: marbar3778 <[email protected]>

* update ResponseCheckTx (cometbft#306)

* rpc: Add totalGasUSed to block_results response (cometbft#308)

* Add C++ code generation and test scenario (cometbft#310)

* add parameters to byzantine send action

* make net not trusted

it's not necessary since for proofs Ivy will assume that the environment
does not break action preconditions

* use require instead of assume

it seems that assume is not checked when other isolates call!

* add comment

* add comment

* run with random seed

* make domain model extractable to C++

* substitute require for assume

assumes in an action are not checked when the action is called! I.e.
they place no requirement on the caller; we're just assuming that the
caller is going to do the right thing. This wasn't very important here
but it leade to a minor inconsistency slipping through.

* make the net isolate not trusted

there was no need for it

* add tendermint_test.ivy

contains a simple test scenario that show that the specification is no
vacuuous

* update comment

* add comments

* throw if trying to parse nset value in the repl

* add comment

* minor refactoring

* add new pex messages (cometbft#312)

* build(deps): bump gaurav-nelson/github-action-markdown-link-check (cometbft#313)

Bumps [gaurav-nelson/github-action-markdown-link-check](https://github.com/gaurav-nelson/github-action-markdown-link-check) from 1.0.12 to 1.0.13.
- [Release notes](https://github.com/gaurav-nelson/github-action-markdown-link-check/releases)
- [Commits](gaurav-nelson/github-action-markdown-link-check@1.0.12...1.0.13)

---
updated-dependencies:
- dependency-name: gaurav-nelson/github-action-markdown-link-check
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update spec to reference currently used timestamp type (cometbft#317)

* build(deps): bump actions/stale from 3.0.19 to 4 (cometbft#319)

Bumps [actions/stale](https://github.com/actions/stale) from 3.0.19 to 4.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v3.0.19...v4)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* address discrepancies between spec and implementation (cometbft#322)

* update proto files for release (cometbft#318)

* stale bot: ignore issues (cometbft#325)

* evidence: add section explaining evidence (cometbft#324)

* statesync: new messages for gossiping consensus params (cometbft#328)

* rpc: update peer format in specification in NetInfo operation (cometbft#331)

* Update supervisor_001_draft.md (cometbft#334)

* core: text cleanup (cometbft#332)

* abci: clarify what abci stands for (cometbft#336)

* abci: clarify what abci stands for

* link to abci type protos.

* abci: clarify connection use in-process (cometbft#337)

* abci: clarify connection use in-process

* Update abci.md

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* invert abci explanations

* lint++

* lint++

* lint++

* lint++

Co-authored-by: M. J. Fromberger <[email protected]>

* proto: move proto files under the correct directory related to their package name (cometbft#344)

* abci.md fixup (cometbft#339)

* abci: points of clarification ahead of v0.1.0

* lint++

* typo

* lint++

* double word score

* grammar

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* Update spec/abci/abci.md

Co-authored-by: M. J. Fromberger <[email protected]>

* pr feedback

* wip

* update non-zero status code docs

* fix event description

* update CheckTx description

Co-authored-by: M. J. Fromberger <[email protected]>

* Update supervisor_001_draft.md (cometbft#333)

* Update supervisor_001_draft.md

If the only node in the *FullNodes* set is the primary, that was just deemed faulty, we can't find honest primary.

* Update supervisor_001_draft.md

* light: update initialization description (cometbft#320)

* apps.md fixups (cometbft#341)

* wip

* wip

* wip

* remove comments in favor of gh comments

* wip

* udpates to language, should must etc

* Apply suggestions from code review

Co-authored-by: M. J. Fromberger <[email protected]>

* remove tendermint cache description

Co-authored-by: M. J. Fromberger <[email protected]>

* proto: add tendermint go changes (cometbft#349)

* add missed proto files

* add abci changes

* rename blockchain to blocksync

* Update proto/tendermint/abci/types.proto

Co-authored-by: Callum Waters <[email protected]>

Co-authored-by: Callum Waters <[email protected]>

* fix mockery generation script (#9094)

Signed-off-by: Marko Baricevic <[email protected]>
Co-authored-by: Milosevic, Zarko <[email protected]>
Co-authored-by: Milosevic, Zarko <[email protected]>
Co-authored-by: Zarko Milosevic <[email protected]>
Co-authored-by: Marko <[email protected]>
Co-authored-by: Zarko Milosevic <[email protected]>
Co-authored-by: Anton Kaliaev <[email protected]>
Co-authored-by: Anca Zamfir <[email protected]>
Co-authored-by: dongsamb <[email protected]>
Co-authored-by: Sunny Aggarwal <[email protected]>
Co-authored-by: Anca Zamfir <[email protected]>
Co-authored-by: Ethan Buchman <[email protected]>
Co-authored-by: Zarko Milosevic <[email protected]>
Co-authored-by: Ismail Khoffi <[email protected]>
Co-authored-by: Zaki Manian <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Tess Rinearson <[email protected]>
Co-authored-by: Alexander Simmerl <[email protected]>
Co-authored-by: Igor Konnov <[email protected]>
Co-authored-by: Sean Braithwaite <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Josef Widder <[email protected]>
Co-authored-by: Andrey Kuprianov <[email protected]>
Co-authored-by: Igor Konnov <[email protected]>
Co-authored-by: Sam Hart <[email protected]>
Co-authored-by: Robert Zaremba <[email protected]>
Co-authored-by: Giuliano <[email protected]>
Co-authored-by: Shahan Khatchadourian <[email protected]>
Co-authored-by: Dev Ojha <[email protected]>
Co-authored-by: istoilkovska <[email protected]>
Co-authored-by: Aleksandr Bezobchuk <[email protected]>
Co-authored-by: Sam Kleinman <[email protected]>
Co-authored-by: Sunny Aggarwal <[email protected]>
Co-authored-by: Federico Kunze <[email protected]>
Co-authored-by: Marko Baricevic <[email protected]>
Co-authored-by: Giuliano <[email protected]>
Co-authored-by: Jordan Sexton <[email protected]>
Co-authored-by: MengXiangJian <[email protected]>
Co-authored-by: Yixin Luo <[email protected]>
Co-authored-by: crypto-facs <[email protected]>
Co-authored-by: Giuliano <[email protected]>
Co-authored-by: William Banfield <[email protected]>
Co-authored-by: Mateusz Górski <[email protected]>
Co-authored-by: M. J. Fromberger <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>
jmalicevic added a commit that referenced this pull request Feb 28, 2023
@jmalicevic @lasarojc
v0.37.x:state/kvindexer: port 0.34 query fix (#77)
fcd3d8a 
* state/kvindexer: associate event attributes with events (#9759)
Co-authored-by: Anca Zamfir <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
Co-authored-by: Romain Ruetschi <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>

* Backport kvindexer fix

Signed-off-by: Thane Thomson <[email protected]>
Co-authored-by: Callum Waters <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>

* By event search is now default behaviour. Including fixes from PRs added to 0.34

Co-authored-by: Lasaro <[email protected]>
@jmalicevic jmalicevic mentioned this pull request Feb 28, 2023
Merged
3 tasks
jmalicevic added a commit that referenced this pull request Mar 21, 2023
@jmalicevic @lasarojc
@thanethomson @sergio-mena
kvindexer: porting event sequences and slash parsing to main (#423)
5febe17 
* v0.37.x:state/kvindexer: port 0.34 query fix (#77)
Backport of #77 from 0.37 and #382 to handle slashes
* state/kvindexer: associate event attributes with events (#9759)
* By event search is now default behaviour. Including fixes from PRs added to 0.34

---------

Co-authored-by: Lasaro <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
@julienrbrt julienrbrt mentioned this pull request Sep 15, 2023
Closed
1 task
@faddat faddat mentioned this pull request Jan 5, 2024
Closed
3 tasks
cometcrafter pushed a commit to graphprotocol/cometbft that referenced this pull request Jun 1, 2024
@mergify @ValarDragon @PaddyMc
perf: Make every gossip thread use its own randomness instance, reduc…
43edb8b 
…… (backport cometbft#77) (cometbft#82)

* perf: Make every gossip thread use its own randomness instance, reducing contention (cometbft#3006)

Closes cometbft#3005

As noted in that issue, we currently are doing extra CPU overhead and
mutex contention for getting a random number. This PR removes this
overhead by making every performance sensitive thread have its own rand
instance.

In a subsequent PR, we can cleanup all the testing usages, and likely
just entirely delete our internal randomness package. I didn't do that
in this PR to keep it straightforward to verify.

---

- [x] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [ ] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

(cherry picked from commit f55b9f4)

* Add Changelgo

(cherry picked from commit ce04f04)

# Conflicts:
#	CHANGELOG.md

* Fix changelog further

(cherry picked from commit bd34ce6)

# Conflicts:
#	CHANGELOG.md

---------

Co-authored-by: Dev Ojha <[email protected]>
Co-authored-by: Dev Ojha <[email protected]>
Co-authored-by: PaddyMc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thanethomson thanethomson thanethomson approved these changes

@lasarojc lasarojc lasarojc left review comments

@ebuchman ebuchman Awaiting requested review from ebuchman

@adizere adizere Awaiting requested review from adizere

Assignees

@jmalicevic jmalicevic

Labels
indexer wip Work in progress
Projects
No open projects
CometBFT 2023
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jmalicevic @thanethomson @lasarojc @sergio-mena