Question about Classifier-free Guidance

[lucala]

When embedding my text for conditioning, the trick for classifier-free guidance is to drop the embedding sometimes (usually 10% of the time).

My question is, what does drop mean? It seems I have come across two variants: using a random tensor as a substitute or a zero tensor.

GLIDE mentions in section 2.3 "we sometimes replace text captions with an empty sequence" - this would be a third option, using the embedding from the empty string?

I haven't been able to find any explanation on this, does someone know?

[fostiropoulos]

@varunponda that reads like a ChatGPT answer lol

@lucala I think it does not matter which method is used for zeroing out as long as it is consistent between sampling and training. Although I do not have experiments on that myself, it makes sense.