Fix retries during inference #523

borzunov · 2023-09-28T17:27:58Z

#331 introduced a bug during inference retries that caused this:

[INFO] Route found: 0:18 via …1EBzGt
[WARN] [petals.client.inference_session.step:327] Caught exception when running inference via RemoteSpanInfo(peer_id=<libp2p.peer.id.ID (12D3KooWLRHAtX9ccW9i1NvpPLigwGX9MstGw3oyCZwmh21EBzGt)>, start=0, end=18, server_info=ServerInfo(state=<ServerState.ONLINE: 2>, throughput=1040.823002928876, public_name=':duck:FYY:sun_with_face:', version='2.2.0', network_rps=1185.4980980484086, forward_rps=9887.81852782432, inference_rps=343.100557603763, adapters=(), torch_dtype='bfloat16', quant_type='nf4', using_relay=False, cache_tokens_left=1179648, next_pings={...})) (retry in 2 sec): AssertionError("Broken input cache: span=RemoteSpanInfo(peer_id=<libp2p.peer.id.ID (12D3KooWLRHAtX9ccW9i1NvpPLigwGX9MstGw3oyCZwmh21EBzGt)>, start=0, end=18, server_info=ServerInfo(state=<ServerState.ONLINE: 2>, throughput=1040.823002928876, public_name=':duck:FYY:sun_with_face:', version='2.2.0', network_rps=1185.4980980484086, forward_rps=9887.81852782432, inference_rps=343.100557603763, adapters=(), torch_dtype='bfloat16', quant_type='nf4', using_relay=False, cache_tokens_left=1179648, next_pings={...})) shape=torch.Size([1, 579, 8192]) position=0 n_input_tokens=1")

borzunov · 2023-09-28T17:28:29Z

src/petals/client/inference_session.py

@@ -359,6 +358,10 @@ def _update_sequence(self, server_idx: int, block_idx: int, attempt_no: int) ->
 # If there is a failed span, this code replaces it, otherwise it just adds new ones
 if server_idx < n_prev_spans:
 updated_sessions[0].history = self._server_sessions[server_idx].history
+ updated_sessions[0].position = self._position


This is the fix

borzunov commented Sep 28, 2023

View reviewed changes

borzunov force-pushed the fix-inference-retry branch from 7ae6897 to 3547b57 Compare September 28, 2023 17:29

borzunov marked this pull request as draft September 28, 2023 17:33

borzunov force-pushed the fix-inference-retry branch 2 times, most recently from 160211a to 567e34b Compare September 28, 2023 18:40

borzunov force-pushed the fix-inference-retry branch 2 times, most recently from ef59cf6 to 3f70ab6 Compare October 23, 2023 16:50

Try 5% fails

63282af

borzunov force-pushed the fix-inference-retry branch from 9289d93 to 63282af Compare October 23, 2023 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix retries during inference #523

Fix retries during inference #523

borzunov commented Sep 28, 2023 •

edited

borzunov Sep 28, 2023

Fix retries during inference #523

Are you sure you want to change the base?

Fix retries during inference #523

Conversation

borzunov commented Sep 28, 2023 • edited

borzunov Sep 28, 2023

Choose a reason for hiding this comment

borzunov commented Sep 28, 2023 •

edited