Poor performance for simple text

[omrishsu]

I've tested the model with the following text:
"The movie: 'Spider-Man 3' (2007) was directed by Sam Raimi.\nDuration: 156\nGenres: J.K. Simmons\nActor1: James Franco\nActor2: Kirsten Dunst\nActor3: Action, Adventure, Romance\nTopics: sandman, spider man, symbiote, venom, villain\nUrl: http://www.imdb.com/title/tt0413300/?ref_=fn_tt_tt_1\nLanguage: English\nCountry: USA\nContent: PG-13\nIMDB Score: 6.2"

to get the probabilities of each answer i've changed the code a bit:

def _evaluate_qa(encoded_qa_pair, qae_model):
    output = qae_model(**encoded_qa_pair)
    return torch.nn.functional.softmax(output.logits)

def get_scores(encoded_qa_pairs, qae_model):
    scores = {}
    qae_model.eval()
    with torch.no_grad():
        for i in range(len(encoded_qa_pairs)):
            scores[i] = _evaluate_qa(encoded_qa_pairs[i], qae_model)

    return [v.cpu()[0][0].item() for k, v in sorted(scores.items(), key=lambda item: item, reverse=True)]

The questions and the answers that I get dose not make sense:

Q: "What is the name of the movie?"
A: "IMDB Score: 6.2"
P: 0.9807605743408203

Q: "What is the name of the movie?"
A: "Url: http://www.imdb.com/title/tt0413300/?ref_=fn_tt_tt_1"
P: 0.9784185290336609

Q: "What is the name of the movie?"
A: "Actor2: Kirsten Dunst"
P: 0.0020964324939996004

Q: "What is the name of the movie?"
A: "Country: USA"
P: 0.9956291913986206

Q: "What is the name of the movie?"
A: "Content: PG-13"
P: 0.9960542917251587

Q: "What is the name of the movie?"
A: "Topics: sandman, spider man, symbiote, venom, villain"
P: 0.9946396946907043

Q: "What is the name of the movie?"
A: "The movie: \'Spider-Man 3\' (2007) was directed by Sam Raimi."
P: 0.5569069385528564

Q: "What is the name of the movie?"
A: "Genres: J.K. Simmons"
P: 0.9990692734718323

Q: "What is the name of the movie?"
A: "Actor1: James Franco"
P: 0.9992097616195679

Q: "What is the name of the movie?"
A: "Duration: 156"
P: 0.9709019064903259

Am i missing here something? or this is an "acceptable" quality?

[AMontgomerie]

It generates nonsense quite often. It's unusual that all of the questions are the same though. Maybe try a longer text?