Parallelization / Batching Explanation

[lapp0]

Hi All,

I'm seeking clarity on the functionality of the --parallel option in /app/server, especially how it interacts with the --cont-batching parameter. My specific observation involves setting --ctx-size to 8192 and --parallel to 32. From the logs, it appears there are 32 slots, each handling a context segment of 256. My question is: Does this configuration imply that each slot processes a distinct segment of the context?

For instance, if I input 32 instances of an identical prompt with a length of 4096, would the first half of the slots remain idle due to the prompt already existing in the KV cache? This leads to confusion, as it seems the total number of submittable jobs is limited by the slot count. This is puzzling because different prompts might rely on the same slot for varied tasks. If the initial segment of the context is identical for multiple prompts, that segment might not require processing, as it's already in the KV cache.

I'm trying to understand the rationale behind dividing the context into segments when batching. Could you provide an explanation of how the --parallel and --cont-batching options function?

References:

• server : parallel decoding and multimodal #3589
• server : parallel decoding and multimodal (cont) #3677
• llama : custom attention mask + parallel decoding + no context swaps #3228
  • "To set the KV cache size, use the -c, --context parameter. For example, for 32 parallel streams that are expected to generate a maximum of 128 tokens each (i.e. -n 128), you would need to set -c 4096 (i.e. 32*128). If continuous batching is enabled, you would need some extra KV space to deal with fragmentation of the cache. In the example above, we conveniently set the context size to 8192 to guarantee that there will be no issues."
• llama : add support for batched inference #2813
  • "We want to be able to generate multiple sequences sharing the same context (a.k.a. prompt) in parallel."
• https://github.com/ggerganov/llama.cpp/blob/master/examples/server/server.cpp#L588-L603
  • batch handling in server.cpp dividing the n_ctx and calling llama_batch_init(n_ctx, 0, params.n_parallel);

[nfairbank]

Bump- trying to understand this as well...

[ggerganov]

The --ctx-size argument actually specifies the total size of the KV cache (legacy name, --kv-size would be better). This corresponds to the total amount of tokens that can be stored across all independent sequences.

For example, if we specify --ctx-size 8192 this means that we can process:

• 2 sequences, each of max length of 4096 tokens
• 4 sequences, each of max length of 2048 tokens
• 8 sequences, each of max length of 1024 tokens
• ...
• 32 sequences, each of max length of 256 tokens
• etc.

Simply put, if we want to be handling P sequences in parallel and we know that each sequence can have a maximum of T tokens (prompt + generated), then we want to set our KV cache size (i.e. --ctx-size) to T*P in order to be able to handle the worst-case scenario where all P sequences fill-up the maximum T tokens.

Since llama.cpp implements a "unified" cache strategy, the KV cache size is actually shared across all sequences. This means that it's allowed to have sequences with more than T tokens as long as the sum of all tokens does not exceed P*T.

From the logs, it appears there are 32 slots, each handling a context segment of 256. My question is: Does this configuration imply that each slot processes a distinct segment of the context?

No. Each sequence has it's own context. The tokens from each sequences "see" only the tokens from that same sequence. This is achieved with the KQ_mask:

llama.cpp/llama.cpp

Lines 6328 to 6368 in dd5ae06

	if (!alloc_inp_KQ_mask && strcmp(name, "KQ_mask") == 0) {
	ggml_allocr_alloc(lctx.alloc, cur);
	if (!ggml_allocr_is_measure(lctx.alloc)) {
	const int64_t n_kv = cur->ne[0];
	const int64_t n_tokens = cur->ne[1];
	float * data;
	if (ggml_backend_buffer_is_host(cur->buffer)) {
	data = (float *) cur->data;
	} else {
	lctx.buf_copy.resize(ggml_nbytes(cur));
	data = (float *) lctx.buf_copy.data();
	}
	for (int h = 0; h < 1; ++h) {
	for (int j = 0; j < n_tokens; ++j) {
	const llama_pos pos = batch.pos[j];
	const llama_seq_id seq_id = batch.seq_id[j][0];
	for (int i = 0; i < n_kv; ++i) {
	float f;
	if (!lctx.kv_self.cells[i].has_seq_id(seq_id) || lctx.kv_self.cells[i].pos > pos) {
	f = -INFINITY;
	} else {
	f = 0;
	}
	data[h*(n_kv*n_tokens) + j*n_kv + i] = f;
	}
	}
	}
	if (data != cur->data) {
	ggml_backend_tensor_set(cur, data, 0, ggml_nbytes(cur));
	}
	}
	alloc_inp_KQ_mask = true;
	}

Each llama_decode call accepts a llama_batch. The batch can contain an arbitrary set of tokens - each token has it's own position and sequences id(s). The position and the sequence ids of a token determine to which other tokens (both from the batch and the KV cache) it will attend to, by constructing the respective KQ_mask. When processed, the batch of tokens is stored in the KV cache. This way, as long as there is enough free slots in the KV cache, we can call llama_decode. In this framework, continuous batching is trivial.

Another great benefit is that different sequences can share a common prompt without any extra compute. All it takes is to assign multiple sequence ids to the common tokens in the KV cache. A basic example is a system prompt of S tokens that can sit at the beginning of the unified KV cache and be attended to by all sequences that we process.

Together with the simplicity and advantages of this implementation, there are a few disadvantages:

• The attention for each sequence is computed over the entire KV cache. This is suboptimal in the case of many sequences and very long contexts because we compute "cross-sequence attention" which is masked after that (i.e. we throw it away)
• The determinism of the results is now also a function of where the tokens of a sequence are placed in the KV cache

In order to resolve these, I think we should add a standard attention implementation where each sequence has it's own KV cache buffer and the attention is computed separately. This way, users would be able to choose which implementation to use based on their specific use case.

[enn-nafnlaus]

I did a test yesterday and got a mere 5% speed increase from changing from 1 slot to 4 (at the cost of losing 3/4ths of my context window). Not a good tradeoff.

In discussions in the issue section, it sounds like using multiple slots with multuiple simultaneous query is really only a meaningful difference when a large chunk of your queries start off the same, such as a long header text followed by a comparably short variable section.

A bigger difference: with -cb but only one slot, I got a 10% speed increase in a much simpler manner (just the "multiple simultaneous queries" aspect above), by modifying my generation script from:

Send a query -> wait for the response -> process the response -> send the next query...

to:

Send 10 queries simultaneously asynchronously -> when any of them finish, process the response in their own separate run thread; separately, send a new query any time the total number of outstanding queries drops below 10.

So, if you're not doing a lot of header duplication in your queries, I'd at least recommend that.

[SPThole]

I am also testing out cuda version of llama CPP server, with -c 32000 -cb -np 8, but when I see log it spin up different Id slot when one output is generated not simultaneously,am I missing something?I am running this on aws G5 12x large which 4 of a10g intotal 96gb, model which I am trying is MoE mixtral gguf 4km

[rmansoul]

J’ai fait un test hier et j’ai obtenu une augmentation de vitesse de seulement 5% en passant de 1 à 4 emplacements (au prix de perdre les 3/4 de ma fenêtre contextuelle). Ce n’est pas un bon compromis.

Dans les discussions de la section problème, il semble que l’utilisation de plusieurs emplacements avec plusieurs requêtes simultanées n’est vraiment une différence significative que lorsqu’une grande partie de vos requêtes commencent de la même manière, comme un long texte d’en-tête suivi d’une section variable relativement courte.

Une plus grande différence : avec -cb mais un seul emplacement, j’ai obtenu une augmentation de vitesse de 10% d’une manière beaucoup plus simple (juste l’aspect « requêtes simultanées multiples » ci-dessus), en modifiant mon script de génération de :

Envoyer une requête -> attendre la réponse -> traiter la réponse -> envoyer la requête suivante...

À:

Envoyez 10 requêtes simultanément de manière asynchrone > lorsque l’une d’entre elles se termine, traitez la réponse dans leur propre thread d’exécution séparé ; Séparément, envoyez une nouvelle requête chaque fois que le nombre total de requêtes en attente tombe en dessous de 10.

Donc, si vous ne faites pas beaucoup de duplication d’en-tête dans vos requêtes, je vous le recommande au moins.

please can you share with me the settings so that the model receives more simultaneous requests ?

[fairydreaming]

From the logs, it appears there are 32 slots, each handling a context segment of 256. My question is: Does this configuration imply that each slot processes a distinct segment of the context?

No. Each sequence has it's own context. The tokens from each sequences "see" only the tokens from that same sequence. This is achieved with the KQ_mask:

llama.cpp/llama.cpp

Lines 6328 to 6368 in dd5ae06

	if (!alloc_inp_KQ_mask && strcmp(name, "KQ_mask") == 0) {
	ggml_allocr_alloc(lctx.alloc, cur);
	if (!ggml_allocr_is_measure(lctx.alloc)) {
	const int64_t n_kv = cur->ne[0];
	const int64_t n_tokens = cur->ne[1];
	float * data;
	if (ggml_backend_buffer_is_host(cur->buffer)) {
	data = (float *) cur->data;
	} else {
	lctx.buf_copy.resize(ggml_nbytes(cur));
	data = (float *) lctx.buf_copy.data();
	}
	for (int h = 0; h < 1; ++h) {
	for (int j = 0; j < n_tokens; ++j) {
	const llama_pos pos = batch.pos[j];
	const llama_seq_id seq_id = batch.seq_id[j][0];
	for (int i = 0; i < n_kv; ++i) {
	float f;
	if (!lctx.kv_self.cells[i].has_seq_id(seq_id) || lctx.kv_self.cells[i].pos > pos) {
	f = -INFINITY;
	} else {
	f = 0;
	}
	data[h*(n_kv*n_tokens) + j*n_kv + i] = f;
	}
	}
	}
	if (data != cur->data) {
	ggml_backend_tensor_set(cur, data, 0, ggml_nbytes(cur));
	}
	}
	alloc_inp_KQ_mask = true;
	}

@ggerganov What is the purpose of h variable in this code? I mean from what I understand for (int h = 0; h < 1; ++h) will have only one iteration with h equal to zero, so why is it used in the data array index calculation: data[h*(n_kv*n_tokens) + j*n_kv + i]? Is this a preparation for some future changes in the code? I'm not sure whether I shall follow this pattern or not.

[ggerganov]

It simply emphasises that the mask is generated only for one of the attention heads and the kernels currently broadcast it for the rest of the heads. In theory, one could have a separate mask for each attention head (I have some vague memory reading about such approach, but in any case, we don't currently support it)