Dynamic Evaluation of Large Language Models by Meta Probing Agents

There are two critical challenges in designing a dynamic evaluation protocol. First, there is no general principle to guide the evaluation sample generation process for diverse tasks such as knowledge, language understanding, reasoning, and mathematics. Existing literature like DyVal (Zhu et al., 2023a) and NPHardEval (Fan et al., 2023) adopted manually designed principles to generate samples for specific tasks and cannot easily extend to other scenarios. Second, the principle of generating evaluation samples should be fine-grained yet atomic enough to analyze the multifaceted capabilities of LLMs. Our hope is that the evaluation should reflect the primitive abilities, and more fine-grained analysis of their correlations can be conducted.

In this work, we take inspiration from psychometrics (Raykov & Marcoulides, 2011; Burnell et al., 2023) to generate evaluation samples using LLMs as agents. Specifically, instead of relying on specific rules like DyVal (Zhu et al., 2023a), we employ LLMs as agents to automatically generate new problems based on the given evaluation samples. This agent-based evaluation design can potentially fit most tasks. More importantly, psychometric theory categorizes cognitive abilities into three basic ones: language understanding to comprehend and generate texts, problem-solving to deal with complex problems, and domain knowledge. Therefore, as shown in Figure 2(a), we follow these criterion to design principles that not only change the problems but also assist in multifaceted analysis.

Our approach, termed as meta probing agents (MPA), is illustrated in Figure 2(b). Given an original evaluation sample from existing benchmarks, MPA aims to evaluate the ability of LLMs by creating new problems following psychometric theory. We create a set of principles that involve two agents: the probing agent and the judge agent. The probing agent aims to investigate the knowledge of a given question $Q$ and return a new one according to a principle $p_{i}$, and the judge agent is responsible for the validity and consistency check based on the original question. Their interaction features a feedback mechanism: if the judge agent determines that the new question lacks consistency (‘No’), the probing agent is prompted again to generate another version of the question. Conversely, if the answer is ‘Yes’, the question is deemed to have passed the consistency check.

The dynamic nature of MPA lies in two aspects: the dynamic generation of evaluation samples and the dynamic combination of principles. Such a dynamic combination allows for multifaceted analysis of LLMs’ abilities, thus providing more insight for future research. It can also be seen as granting problems with flexible complexities (Zhu et al., 2023a) for a comprehensive evaluation. Now, we present the details of the agents and the psychometric principles.

The probing agent aims to transform a given question into a new one to assess LLMs’ ability required by a question $Q$. The probing is guided by principles inspired by psychometrics, which are encapsulated within a carefully crafted prompt. Unlike generating training samples, one of the most important criteria in probing agents is to ensure that the generated questions maintain the core essence of the original while presenting a different perspective, thus maintaining its correctness. An example is shown in Figure 2(c), where a sample of ARC-C is transformed into a new one by applying different principles. The prompts in our experiments incorporate directives that guide the agent towards creating semantically similar but structurally different questions.

Although we have explicitly restricted the probing agent to maintain the integrity of the original question, there are cases where probing agents unintentionally change the meaning. Thus, the judge agent is designed to provide a clear and unambiguous assessment of whether the generated question maintains the integrity of the original intent and informational content. Its prompt is designed to direct LLMs to compare the original with the rephrased questions, ensuring the preservation of the essence and factual accuracy.

Specifically, unlike traditional evaluation methods that may use a variety of metrics, the judge agent operates in an adversarial manner via a binary response system through prompts. It simply returns a ‘Yes’ or ‘No’ verdict, indicating whether the new question maintains consistency with the original. In this prompt, the judge agent is required to analyze the essence of both the original and rephrased questions. Its goal is not merely to identify superficial similarities or differences in wording, but to delve deeper into whether both versions of the question are aligned in terms of the concept or knowledge area they are assessing.

Psychometric principles guide how we probe the understanding of a question. Inspired by psychometric theory (Raykov & Marcoulides, 2011), we aim to evaluate three basic abilities of LLMs: language understanding, problem solving, and domain knowledge. We have identified five key principles that correspond to these categories, as shown in Table 1. These principles incorporate both a probing agent and a judge agent, as previously discussed, with their functions differing according to the specific principle applied.

Tasks and Datasets. We selected four popular datasets for evaluation: MMLU (Hendrycks et al., 2021), ARC-Challenge (ARC-C) (Clark et al., 2018), GSM8K (Cobbe et al., 2021), and BigBench-Hard (BBH) (Suzgun et al., 2022; Srivastava et al., 2022), encompassing a broad spectrum of computational challenges ranging from knowledge-intensive understanding to complex mathematical and logical reasoning tasks. We adopted only three hard tasks from BBH: Formal Fallacies, Object Counting, and Temporal Sequences. We used their test sets to generate new evaluation samples. The detailed introduction is given in Appendix A.

Evaluated LLMs. We evaluated three proprietary LLMs: GPT-4-Turbo (OpenAI, 2023b), GPT-3.5-Turbo (OpenAI, 2023a), and Gemini-Pro (GeminiTeam, 2023), and three strong open-sourced models: Llama2-70b-chat (Touvron et al., 2023), Yi-34b-chat (01-ai, 2024), and Mixtral-8x7b-Instruct (MistralAITeam, 2023). To ensure a standardized comparison, we set the generation temperature to $0$ for all models, with the generation length as $1000$ tokens.²²2For Gemini-Pro on MMLU dataset, we set the temperature to $0.7$ and omitted some evaluation samples (around 20 samples) to avoid response failures due to Google’s safety constraints.

Agent LLMs in MPA. We utilized GPT-4-Turbo as probing and judging agents, with temperatures of $0.7$ and $0$, respectively. The maximum token generation for each agent is set as $1000$. While GPT-4-Turbo serves as the main agents, we also explored the potential to integrate other LLMs such as GPT-3.5-Turbo and Gemini-Pro as agents in later experiments in Section 4.6. The evaluation prompts are detailed in the Appendix B. Our primary evaluation metric is accuracy.

A bitter reality is that currently, only GPT-4 are capable to generate questions and judge the quality of generated questions. We believe as the field progresses, more cheaper models (such as Claude 3 and Gemini) will become capable of fulfilling both probing and judging roles, thereby reducing dependency on any single model’s API and enhancing the scalability.

In this part, we applied all five principles to generate new evaluation samples for MMLU and ARC-C. For GSM8K and BBH that do not have multiple choices, we restricted our probing to Principle $1$ and $4$. Appendix D shows some examples generated by MPA. We repeated the evaluation three times to reduce randomness. Table 2 presented the test accuracy of different LLMs on the original and our MPA benchmarks. Standard deviation are mostly around $1$ (Table 5), indicating the robustness of the benchmark. As can be seen, all LLMs exhibited performance degradation on our probing benchmarks. Although GPT-4-Turbo demonstrated the strongest data contamination problem, it remains the strongest model. For MMLU, GPT-4-Turbo performed $15.7$% worse than the original benchmark. Furthermore, a notable performance decline is evident in the case of MMLU and ARC-C, which is significantly more pronounced than that observed in GSM8K and BBH. This suggests that LLMs may encounter the memorization of knowledge-based benchmarks, resulting in substantial performance degradation for evaluation on our benchmarks.

We also presented a confusion matrix for analysis as shown in Figure 3. The matrix categorizes the responses into four distinct segments, with four categories to evaluate the model responses: OT (Original True), PT (Probing True), OF (Original False), and PF (Probing False). A notable observation is the high frequency of ‘OT/PF’ instances, which suggests a potential data contamination to specific dataset characteristics. Furthermore, the frequency of open-source models is markedly higher than that of proprietary models like GPT-4. This discrepancy indicates that open-source models might be more susceptible to data contamination.

We further studied the effects of each modular probing principle. To this end, we established a baseline where the combined effect of all principles leads to a decrease in performance, normalized to a value of “1”. This approach enables us to compare the relative effectiveness of each principle in isolation. The performance decrement for each principle, when applied independently, was evaluated and compared with this baseline. The Relative effectiveness (RE) is computed as: $\text{RE}=\frac{\text{Acc}_{p_{i}}-\text{Acc}}{\text{Acc}_{p_{all}}-\text{Acc}},$ where $\text{Acc}_{p_{i}}$ is the accuracy when only apply principle $p_{i}$ to MPA, $\text{Acc}_{p_{\text{all}}}$ denotes the accuracy of MPA when all principles are applied. Acc is the accuracy on the original benchmark.

The results on MMLU and ARC-C datasets are detailed in Figure 4 and Section C.2. It can be observed that, principle $1,2$ and $5$ are the most effective principles. While principle $3$, which randomly permute choices, are less effective. For GPT-4-Turbo and GPT-3.5-Turbo on ARC-C dataset, it can intriguingly increase the performance.

We explored the efficacy of prompt engineering techniques, specifically Chain-of-Thought (CoT) (Wei et al., 2022) and In-Context Learning (ICL) (Brown et al., 2020). For ICL, we selected five examples from the corresponding training set as few-shot examples. We also applied MPA to these five examples, creating a transformed version of few-shot examples. We refer to the use of ICL with the original training examples as ICL_o and the use of ICL with the transformed examples as ICL_t.

As can be observed in Table 3, neither CoT nor ICL can effectively boost the performance of LLMs on our probing benchmarks. Performance enhancements are varied across models and datasets when CoT and ICL are applied. For instance, GPT-4-Turbo showed a modest increase in accuracy on the GSM8K dataset with ICL, improving from $88.50$ to $89.39$. In contrast, GPT-3.5-Turbo’s performance slightly decreased under the same conditions. Note that the significant performance gain observed for Gemini-Pro on the GSM8K dataset when using ICL is attributed to its initial lower effectiveness in a zero-shot context.

We collected $30$ novel reasoning questions from the experts (the same experts in the human evaluation in the general response), and examined the performance of the original questions and the probing questions. The GPT-4’s accuracies of the original questions and the probing questions are both $60$%. The OT/PF ratio is $3$%, which is much lower than those in common public benchmarks, indicating that the newly generated benchmarks are not likely to be memorized compared to the public benchmarks.

In this section, we assess the feasibility of using less advanced LLMs to reduce costs of MPA evaluation. We initially configured GPT-3.5-Turbo and Gemini-Pro as judging agents, while maintaining GPT-4-Turbo as the probing agent for GSM8K dataset. Through meticulous manual examination of the questions transformed by this setup, we observed a significant shortfall in the judging agents’ ability to discern and exclude transformed examples whose meanings deviated from the original questions. highlighting the need for robust and capable LLMs to effectively sieve out appropriately probing questions. Subsequently, we extended our inquiry to assess the potential of employing less capable LLMs like GPT-3.5-Turbo and Gemini-Pro as probing agents. Using GPT-4-Turbo as the judging agent, our experiments revealed that when weaker LLMs served as probing agents, they often altered the essence or subtleties of the original questions, leading to misrepresentations or the introduction of unintended elements. And GPT-4-Turbo struggled to consistently detect these nuanced alterations. This suggests that the sophistication of the probing agent is crucial, as even advanced judging agents like GPT-4-Turbo cannot always offset the limitations of the probing agents.

We first explore the impact of benchmark complexity on the MMLU and ARC-C datasets, as illustrated in LABEL:fig-benchmark-complexity. We construct probing benchmarks with different levels of complexity using (1) language understanding principle $1$, (2) language understanding principle $1$ and problem-solving principle $4$, and (3) language understanding principle $1$, problem-solving principle $4$, and domain knowledge principle $5$. In particular, as the complexity of the benchmarks increases, the performance of all LLMs decreases and GPT-4-Turbo consistently reaches the highest precision. This result shows that there is still much room to improve the abilities of LLMs in complex benchmarks.

To gain a deep understanding of the relationship between the abilities of language understanding (LU), problem solving (PS), and domain knowledge (DK), we constructed three probing benchmarks using the principles that belong to a certain ability and then evaluated LLMs on these benchmarks. The results are presented in Table 8. After obtaining the performance of different LLMs on each probing benchmark, we then calculated the Pearson correlation coefficients (Pearson, 1920), the Spearman correlation coefficients (Spearman, 1904), and the Kendall correlation coefficients (Kendall, 1938), the results are presented in Table 4. It is evident that all abilities are highly correlated, which aligns with the findings of Burnell et al. (2023). Furthermore, it is observed that language understanding and problem solving are more relevant compared to other pairs of abilities. This suggests that the two abilities can predict each other, which has great potential to train and improve LLMs in the future. In the future, further detailed analysis can be performed to gain a deeper insight into the basic abilities of LLMs by constructing MPA evaluation samples based on other existing benchmarks such as HELM (Liang et al., 2023).

We studied the influence of model size on basic abilities. First, LABEL:fig:_model_size_acc shows the correlation with variants of Llama2: 7b, 13b, and 70b. It can be observed that each ability positively correlates with the model size with nearly the same slope, indicating that when the size of the model increases, all abilities are equally enhanced to improve overall performance. Second, we explored the relationship between model size and correlations between different abilities. We roughly divided the models into three sizes: (1) small: Llama2-7b-chat, Llama2-13b-chat; (2) mid: Yi-34b-chat, Mixtral-8x7b-Instruct, Llama2-70b-chat; and (3) Gemini-Pro, GPT-3.5-Turbo, GPT-4-Turbo. The results in LABEL:fig:_model_size_correlation implies an implicit “Matthew Effect” (Merton, 1968): larger (often stronger) models tend to have stronger correlations between basic abilities. This aligns well with existing psychological theory about the $g$ factor of general intelligence (Spearman, 1961). This finding can potentially help explain the emergent abilities of LLMs (Biderman et al., 2023; Schaeffer et al., 2023) and provide insight into the evolution of LLMs.

We conducted an in-depth analysis of LLMs’ failure modes in the three basic abilities. We meticulously selected $50$ instances where GPT-4-Turbo correctly answered the original questions but failed in the transformed questions in the GSM8K dataset. These error modes are shown below.

• •

  Language understanding. (1) Question Understanding Error: GPT-4-Turbo calculates the correct answer but misinterprets the intent of the question, leading to an incorrect response. This error indicates a gap in comprehension of the question. (2) Instruction Following Error: In this mode, GPT-4-Turbo arrives at the correct answer but fails to present it in the required format specified in the prompt, indicating a lack of follow-up instructions.

• •

  Problem solving. Here, GPT-4-Turbo understands the question correctly but errs during the calculation process, resulting in a wrong final answer.

• •

  Domain knowledge. We investigated the distribution of topic error among $57$ tasks of MMLU in Figure 10. Notably, the professional law domain has the highest error rate, followed by moral scenarios and professional psychology, suggesting challenges predominantly in professional and ethical tasks.

Furthermore, we observed two possible reasons for performance degradation: ambiguity in the original questions and inconsistency between the probing and original questions. Some questions in the original dataset were found to be ambiguous (see Section C.5). Despite this, GPT-4-Turbo often provided plausible answers, suggesting potential issues with data memorization. However, certain transformed questions deviated in meaning from their original versions, leading to inconsistencies in responses.