Embeddings - last_hidden_state vs hidden_state[-1]
#76
by
technicalanalyst
- opened
last_hidden_state is not the same as hidden_state[-1]. Is this due to some normalisation?
text = "I am going home today."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True)
last_hidden_state_1 = outputs.last_hidden_state # Directly from outputs
last_hidden_state_2 = outputs.hidden_states[-1] # Last layer from hidden_states tuple
are_equal = torch.allclose(last_hidden_state_1, last_hidden_state_2, atol=1e-6)
technicalanalyst
changed discussion status to
closed
technicalanalyst
changed discussion status to
open
text = "I am going home today."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True)
last_hidden_state_1 = outputs.last_hidden_state # Directly from outputs
last_hidden_state_2 = outputs.hidden_states[-1] # Last layer from hidden_states tuple
are_equal = torch.allclose(last_hidden_state_1, last_hidden_state_2, atol=1e-6)