T-SNE of features visualization

[HuadongTang]

Hi, Thanks for your nice work.
I have some questions about T-SNE visualization.

1. What's the meaning of each point in the T-SNE visualization map of your paper. (Each point is a pixel feature?). As you mentioned in the former issue, features(tensor size[8,256,256,512]) after the projection layer are used. I try to draw the T-SNE map and I reshape the features to 8256512=1048576. Then, I got TensorA (1048576, 256). After that, I randomly sample 5000 from the first dimension of A. But I got a bad T-SNE map. So I wanna know how you handle the features after the projection layer and how many images for T-SNE visualization.
2. The T-SNE map. Are the features of Pixel-wise Cross-Entropy Loss map from segmentation head? (I think it's the baseline method without contrastive loss, right?)
   But features of Pixel-wise Contrastive Learning Objective are from the projection layer. (method with contrastive loss? ) I am confused about where are the features of the two loss from.

[HuadongTang]

I reshape the features to 8x256x512=1048576 and got tensor (1048576, 256)

[xiewende]

Can you provides the code of T-SNE of features visualization? thank you very much!!!

[lennart-maack]

Hey everyone,

I implemented the tsne visualization in the following way:

1. Getting the features for each pixel
   a. In my case I have a feature embedding of [B, D, H*, W*] with B being the batch size, D being the number of features for each
   pixel and H*, W* the corresponding height and width of the feature embedding. In my case it is [32, 256, 33, 33]
   b. So I have [32, 256, 1089] --> 256 features for each of the 1089 pixels
   c. I apply PCA to only get 100 features for each of the pixels
   d. Afterwards apply tsne

I implemented it for an example with a binary segmentation mask as ground truth (getting the label for each pixel):
Link to code

[using369]

I reshape the features to 8x256x512=1048576 and got tensor (1048576, 256)

Hello，I got the same bad result. Have you solved it? Can we discuss it?

[Lemonweier]

@lennart-maack Hi ,thank you for your code. I want to ask you about D, how do I know the number of features for each
pixel. Thank u!

[using369]

@lennart-maack Hi ,thank you for your code.! I want to ask about domains. What do "either src,src_to_trgt,trgt"stand for?

[using369]

Maybe we can communicate, my email is [email protected]