Animate, or Inanimate, That is the Question for Large Language Models

By constructing a series of prompts over the datasets presented in §2.2.1, we test the models’ answers to animacy in situations where the animacy of an instance aligns with its more general type.

Sentence pairs in BLiMP Warstadt et al. (2020) is built as follows: one sentence respects the animacy constraints, and the other violates them. Hence, there is a straightforward way to evaluate the LLM’s ability to surpass the animacy test. We ask them to answer the following prompt:

Choose which example is acceptable between A and B. A) Galileo is concealed by the woman. B) Galileo is concealed by the horse. Answer:

A model gets a correct example if it chooses a sentence that respects the animacy constraint.

Following this approach, we evaluate the accuracies by performing a string matching- between the generated answers and the target values on both sub-tasks.

The results of this first experiment set are intriguing. The OpenAI family behave on par with respect to humans, and the Meta and Mistral families are catching up. Figure 2 shows the results of each model (vertical bars) and the results obtained by humans (horizontal dashed lines), as presented in Warstadt et al. (2020). The accuracy metric has to be intended as the percentage of examples in which human or artificial subjects preferred the acceptable sentence of the given pair.

Human Transitive Accuracy is reached by GPT-3.5 and topped by GPT-4. This seems to suggest that these models can handle the lexicon to determine typical animacy. Llama lags behind, but it is reaching the Human level.

Similarly, in the passive scenario, GPT-4 performs very close to humans. Concerning GPT-3.5, Llama2-70 and Mixtral have comparable and slightly lower performance in the transitive scenario and significantly lower performance in the passive scenario, respectively. Finally, the smaller models, i.e., with fewer parameters, underperform humans with average gaps of 20 points.

This difference between the transitive and passive case may be due more to differences in setting than to different animacy processing in the two scenarios. However, emerges that the composition of the choices is strongly class-related. Indeed, in the passive case, the target word, i.e., the most influential one, is always in the last position. In contrast, the target word is not the last token in the transitory case. Thus, heuristics related to the sensitivity of the choices in the input prompts’ structure may be present.

In the second Experiment, we structured the prompting phase similarly. Hence, by constructing a series of prompts over the BSP benchmark (§ 2.2.1), we test the model’s responses in situations where plausible and implausible sentences with animated and non-animated components were provided.

Following Vega-Mendoza et al. (2021), we analyze the model’s answers to the plausibility question on five inputs constructed from a sentence and completed with different types of answers. Hence, there is a straightforward way to evaluate our subject’s ability to surpass the plausibility test. Hence, given the following prompt:

Is the following sentence plausible? Answer by choosing (Yes) or (No). Sentence: In ancient Egypt the people were governed by the pyramid. Answer:

A model gets an answer if it answers the question with Yes/ No, respecting plausibility. The accuracy of the LLMs is computed in this way: for the plausible control word, the accuracy counts the percentage of Yes, and for all the implausible words the percentage of No.

In this second experiment, LLMs of the OpenAI family behave similarly to humans as in the previous. Figure 3 shows the accuracy results of each model and the results obtained by humans, as presented in Vega-Mendoza et al. (2021). Dealing with animated words (see Figure 3), humans and LLMs behave similarly. Indeed, humans perform on animate-unrelated similarly to how they perform on control words. Instead, they are less able to recognize animate-related as making target sentences implausible. The same happens for all the LLM subjects. GPT-4 performs better than humans, and it keeps the difference in recognizing the implausibility of sentences built with animated-unrelated and animated-related words. Moreover, when dealing with unanimated words (see the right plot in Figure 3), humans and LLMs behave similarly. Humans recognize the implausibility of unanimated-unrelated words but have a slight decrease in recognizing unanimated-related ones. The same trend happens for all the LLM subjects and, consistently in other experiments, the OpenAI family performs better than humans.

In humans, these differences in the plausibility of animate and non-animate cases are given by a combination of cognitive factors, as explained by Vega-Mendoza et al. (2021). Consequently, as in the experiments in § 2.3.1, GPTs perform comparably to humans and sometimes outperform. However, even in this task, there is a robust structural component related to fearfulness. The target words, i.e., those that provide the final decision, are always in the last position. Therefore, there may be a heuristic related to the sensitivity of the structure of choices in the input prompts.

The repetition experiment on Atypical Animacy Nieuwland and Van Berkum (2006) measure the N400 responses of a series of participants who listened to Dutch stories containing a typical animate or an inanimate entities behaving as if it were a human being. The N400 values are measured in three stages: the first ($T_{1}$), the third ($T_{3}$), and the fifth ($T_{5}$) mention of the entity (see, for example, Table 4 with confectioner and apple pie). Nieuwland and Van Berkum (2006) discovered that:

• •

  in the case of animated entities, participants have a moderate N400 response to the first mention ($T_{1}$) and a low response to subsequent mentions ($T_{2}$ and $T_{3}$);

• •

  in the case of inanimated entities, participants initially ($T_{1}$) have a high N400 response to the atypically animated entity, and, as the mentions progress ($T_{2}$ and $T_{3}$), their N400 responses are very close to the responses from the mentions of the animated entity.

Thus, while the humans are initially surprised by the atypically animated entity, they quickly adapt to the situation and no longer find it surprising. Moreover, they show that responses do not derive from lexical repetition but from context. In fact, in the contextual experiment, they provide a context. Only at the end did they estimate N400 responses of the participants obtaining low results for inanimate entities in atypical inanimate contexts.

To estimate a surprise value analogous to the N400, state-of-the-art studies examine token probability values. However, some of the models used in our study do not provide access to probability values, prompting us to define a series of prompts to query the model about its level of surprise systematically. In particular, for each of the 60 examples, we estimate the surprise of the animate and inanimate entity given by the context at each time-step, denoted as $T_{n}$, (which, in our case, refers to an input-prompt). For instance, to model the inanimate N400 response at $T_{1}$ in the example from Table 4, we construct the following prompt:

Choose a surprising value from 0 to 30 on the following story: A granny met the apple pie at the market with whom she started a pleasant conversation about recipes. Answer:[num]

Following the time-steps, we introduce additional prompts by contextualizing the preceding story, whether animate or inanimate. For example, for the inanimate scenario:

Given the following story: A granny met the apple pie at the market with whom she started a pleasant conversation about recipes. The apple pie confided to............. Choose a surprising value from 0 to 30 on the following story: The apple pie that this was the ultimate recipe and apologized for the misplaced distrust. Answer:[num]

Hence, we compute the average surprise value of examples containing animate and inanimate entities separately at each time-step.

The LLMs follow the general trends of human N400 responses (Figure 4). Indeed, as reported in § 2.3.3, human N400 responses for animate and inanimate critical words diverge at $T_{1}$ and come closer at $T_{3}$ and $T_{5}$. LLM subjects behave similarly. In fact, at $T_{1}$, models are surprised by the inanimate entity and trimmed by the animate one. At later steps ($T_{3}$ and $T_{5})$, the surprises of inanimate entities decrease until they reach levels similar to animate entities. LLMs seem to adapt, just as humans do. However, the raw results do not show that the models adapt to the same extent as humans.

We use the Wilcoxon signed-rank test for making the experiments robust as Nieuwland and Van Berkum (2006). We observe distinct surprise values at each time step. As with humans, LLMs have a statistically significant difference between the surprises of animate and inanimate entities, for example, $T_{1}$ of Figure 5. However, while there is no difference between humans at $T_{3}$, there are differences (p < 0.01) in most models; only the largest do not have any. At $T_{5}$, differences disappear only in the large models. Although the models can generally approximate human N400 responses to atypical animacy trends, only the most significant and most potent fully replicate human adaptation.

Nieuwland and Van Berkum (2006) discover that contextual appropriateness seems to neutralize animacy violations, that is, non-appropriate adjectives (such as “worried”, for example, in Table 3) are not generating much surprise if the context suggests them. Moreover, context can even make an animacy-violating predicate more preferred than an animacy-obeying canonical predicate if the context justifies this.

By using examples for the context experiment (as in Table 3), we ask for a surprising value for animate and inanimate adjectives for each of the 60 stories proposed by Nieuwland and Van Berkum (2006). In particular, we use input-prompt structures closer to the previous:

Given the following context: +Context A girl told a sandwich that an attack was imminent. The sandwich wailed that his family was in danger. The girl told the sandwich that public places were the most dangerous. The sandwich immediately started calling everyone he knew. Baseline Choose a surprising value from 0 to 30 on the following story: The sandwich was delicious and wanted to make sure none of his loved ones were in danger. Answer:[num]

To estimate absolute values, we also ask for baseline surprises, that is, those of the inanimate adjective without the context of the whole story.

Even in this experiment in Figure 5, LLM subjects behave similarly to human subjects. The animate baseline is larger for all subjects than the inanimate baseline. Even in the baseline, there is conflicting information as to the presence or absence of animacy of the selected subject. Moreover, the surprise drop with the context is more significant with inanimated than animated adjectives. This is in line with the human experiments.