whether past_key_values can be obtained

[sa1utyeggs]

Hi there.
I'm trying to combine chat.petals.dev with streaming-llm (https://github.com/mit-han-lab/streaming-llm).

However, when streaming-llm needs the output.past_key_values, I found that petals returns [RemotePastKeyValues].

And the annotation says 'A mock class representing the fact that past_key_values do exist but are stored on remote servers.'.

SO, I wonder if there is any way to get the real value of 'past_key_values'.
like this img :

looking for your reply :)

[borzunov]

Hi @sa1utyeggs,

The actual past_key_values are stored on servers and can't be viewed from clients at the moment. In theory, it's possible to extend the server to send them when necessary (e.g., when a user reads fields of the RemotePastKeyValues object). Alternatively, it may be possible to extend the transformer block code to perform operations you're interested in on servers directly, without sending extra data to the clients.