How to extract encoder representation?

[colinator]

After running audio through the model, I would like to extract the representation of the final encoder output. I'm curious as to whether this representation would contain enough information to perform transfer learning, to detect other things (maybe sentiment or something).

Anyway, not sure how to do it. What I was thinking:

1. I have the whisper_context

2. From that, get the whisper_state (this isn't yet exposed)

3. From that, get "embd_enc", which is

   // result of the encoder
   struct ggml_tensor * embd_conv = nullptr;
   struct ggml_tensor * embd_enc = nullptr;

4. From embd_enc, I can get the shape. In my case (using the "base" model), it's (512, 1500)

5. From embd_enc, I can get the actual data: "void * data;", and static_cast it to float*.

I don't think it's working. I can do this, but then if I visualize it like an image, it shows almost no change over time. I suspect that "void* data" isn't actually the data. Could it be coming instead from "struct ggml_backend_buffer * buffer;"?

---

Edit: nope, "embd_enc" is not the final encoder layer output, it's the output of mel->conv->mlp before it hits the encoder stack. And "void * data" is the weights. I believe.

Edit2: er, now I'm not so sure. At the end of whisper_build_graph_encoder we see "wstate.embd_enc = cur;" where 'cur' is the final tensor output of the encoder_layers. So maybe it the encoder stack output not the audio encoder output?

[ggerganov]

I suspect that "void* data" isn't actually the data.

Yes, you need to extract the data with:

whisper.cpp/ggml-backend.h

Line 47 in ccc85b4

GGML_API void ggml_backend_tensor_get(const struct ggml_tensor * tensor, void * data, size_t offset, size_t size);

Something like (not tested):

std::vector<float> my_buf(ggml_nelements(embd_enc));
ggml_backend_tensor_get(embd_enc, my_buf.data(), 0, ggml_nbytes(embd_enc));

So maybe it the encoder stack output not the audio encoder output?

It is the encoder stack output with normalization applied to it:

whisper.cpp/whisper.cpp

Lines 2009 to 2023 in ccc85b4

	cur = inpL;
	// norm
	{
	cur = ggml_norm(ctx0, cur, hparams.eps);
	// cur = ln_f_g*cur + ln_f_b
	cur = ggml_add(ctx0,
	ggml_mul(ctx0, cur, model.e_ln_w),
	model.e_ln_b);
	}
	ggml_build_forward_expand(gf, cur);
	wstate.embd_enc = cur;

This is what comes out as a result from the Encoder and used as input for the cross attention in the Decoder.

[colinator]

Ah, many thanks. A few things:

1. the output, after the ln_f_g*cur + ln_f_b step, is no longer normalized between 0 and 1. Rough values as I see them are something like min element=-17.4844, max element=14.6094.
2. I tried just casting void * data to float *, and that gave me the exact same result as ggml_backend_tensor_get(.
3. Sometimes when I run, the first time through the backend is not set, and I get an assertion failure. It's rare though. Weird.

Figured I'd start with visualization... Want to see the result? This is the 512x1500 embedding normalized and converted to 0-255 uint8, and rendered in grayscale. Time goes to the right. I talk in English for about 10 seconds into a microphone, so you can see about the first 1/3rd has embeddings that look like, well, something I guess. You can distinctly see the positional encoding coming through to the last encoder output.

[ggerganov]

Very interesting stuff! We can probably build tools for extracting such kind of data from the model and the processing.

I tried just casting void * data to float *, and that gave me the exact same result as ggml_backend_tensor_get()

Yes, it works at the moment, but will likely stop working in the future when we make some changes to the Metal backend.
Using ggml_backend_tensor_get() would also work with CUDA backend

Sometimes when I run, the first time through the backend is not set, and I get an assertion failure. It's rare though. Weird.

The backend member should always be set. Likely this is a side-effect from your modifications, unless you observe this on master too?

[colinator]

Hm - I'm not making any changes to ggml - I only add this to whisper.cpp:

struct ggml_tensor * whisper_embd_enc(struct whisper_context * ctx) {
    return ctx->state->embd_enc;
}

and then

ggml_tensor * tensor = whisper_embd_enc(ctx);
std::vector<float> tensor_data(ggml_nelements(tensor));
ggml_backend_tensor_get(tensor, tensor_data.data(), 0, ggml_nbytes(tensor));

I'm pretty sure that none of my modifications could have this side-effect. I synced against master a couple days ago. Maybe every 10 times running, I get whisper.cpp/ggml-backend.c:137: backend != NULL && "tensor backend not set" on the first run through the model. Without the call toggml_backend_tensor_get(), it doesn't crash though - it all works fine. So I guess the backend isn't needed? Or maybe there's some predicate about data length or something. Or maybe a race condition? I'll try to dig a bit.

[colinator]

And a thought - it'd be cool if I could just run the encoder, not the decoder. Furthermore, then encoder doesn't actually need 30 seconds of data, right? I'm a bit unclear on that bit. Can I just give it 25ms of data at a time?