Eliminating Position Bias of Language Models:
A Mechanistic Approach

Abstract

Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias often leads to unexpected model failures and hurts performance, robustness, and reliability across various applications. Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. Specifically, we find that causal attention generally causes models to favor distant content, while relative positional encodings like RoPE Su et al., (2024) prefer nearby ones based on the analysis of retrieval-augmented question answering (QA). Further, our empirical study on object detection reveals that position bias is also present in vision-language models (VLMs). Based on the above analyses, we propose to eliminate position bias caused by different input segment orders (e.g., options in LM-as-a-judge, retrieved documents in QA) in a training-free zero-shot manner. Our method changes the causal attention to bidirectional attention between segments and utilizes model attention values to decide the relative orders of segments instead of using the order provided in input prompts, therefore enabling Position-INvariant inferencE (PINE) at the segment level. By eliminating position bias, models achieve better performance and reliability in downstream tasks where position bias widely exists, such as LM-as-a-judge and retrieval-augmented QA. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: it consistently provides $8$ to $10$ percentage points performance gains in most cases, and makes Llama-3-70B-Instruct perform even better than GPT-4-0125-preview on the RewardBench reasoning subset.

Figure 1: Motivating examples showing how position bias affects model outputs. Upper: LMs are asked to select a more helpful one from two given responses, and are prone to prefer the response positioned at first. Lower Left: LMs (Llama-3-8B-Instruct) are presented with 20 documents to answer a question, with only one document (the gold-standard document) containing the correct answer. The blue curve represents normal inference, while the red curve represents inference without inter-document attention (RoPE position encodings are kept, a concrete implementation is shown in the middle of Figure 2). The height change of the yellow and orange area reflects the position bias brought by causal attention and RoPE: causal attention generally favors distant content but RoPE prefers nearby content. Lower Right: We insert a real-world image to a large black background image at different positions, and prompt VLMs (Fuyu-8B [5]) to compute the loss on the ground truth token. The resulting low-loss (black) and high-loss (white) areas form clusters rather than being evenly distributed.

1 Introduction

Language models (LMs) [6, 9, 32, 1] demonstrate impressive performance in general language tasks such as dialogue [31], reasoning [9], and schema induction [18]. However, they tend to favor content at certain positions [43, 42, 35, 44, 8, 19], which harms complex reasoning [8], long-context understanding [19] and model-based evaluation [43]. For example, LMs perform better in retrieval-augmented QA when the document that contains the correct answer is presented at the beginning or at the end of retrieved documents [19]; and LMs tend to favor the first when it is required to compare the quality of two candidate responses [43], which hurts their reliability when being used as evaluators (Figure 1 upper). Although several works have proposed solutions to the problem [27, 7, 11, 13, 44], they are ad hoc and do not fundamentally tackle the challenge. We seek to understand the causes of position bias from first principles and propose elimination strategies accordingly.

We start by analyzing the key components of state-of-the-art LMs – Casual Attention and Rotary Position Embedding (RoPE) [29] that enable models to understand the order of tokens so that LMs can generate meaningful outputs. We argue that they are also the only two operations in Transformers [33] that will inevitably bring undesirable position bias based on a concise logical proof: RoPE is shown to have recency bias [29, 24] due to its long-form attention weight decay w.r.t. the increase of relative positions, and we hypothesize that causal attention forces unidirectional information propagation, enabling models to pay more attention to distant content. To verify it, we conduct a simple analysis on the retrieval-augmented QA [19] (Figure 1 lower left). The height change of the yellow area and orange area reflects the position bias of causal attention and RoPE. Since the yellow area is mostly wider at the beginning and the orange area generally becomes wider at the end (except for the last data point), showing that the causal attention generally tends to favor distant content, while RoPE generally tends to favor nearby content. To prove generality, we further conduct a semi-synthetic task using Vision-Language Models (VLMs), which also adopt causal attention and RoPE, and would put more attention at certain positions of an image as well (Figure 1 lower right, see more VLM examples in Appendix A).

As a solution, we propose PINE that can eliminate position bias by manipulating causal attention and RoPE to attend to different content equally. For tasks that contain position-agnostic segments while segment orders are not expected to affect results (e.g., retrieved documents and candidate responses in retrieval-augmented QA [19] and LM-as-a-judge [43]), we make the inter-segment attention bidirectional so that attention mask will equally attend to all segments. Next, we compute similarity scores between segments and use them to re-sort segment positions so that those in the original inputs are discarded. The resulting approach enables Position-invariant inference (PINE) in a training-free/zero-shot manner that operates on pre-determined segments.

To justify the effectiveness of PINE, we select two useful tasks where position biases are well-known to benchmark PINE: LM-as-a-judge (RewardBench [17]) that prompts LMs to choose the more helpful one from two given responses to a question (candidate responses as segments), and retrieval-augmented question-answering [19] that prompts models to answer questions based on retrieved documents. We show that PINE eliminates position bias such that PINE reaches greater performance and reliability due to no variance w.r.t. relative segment orders in inputs. Notably, we find our method especially useful when LMs are used to assess reasoning pairs: PINE generally improves model performance by $8$ to $10$ percentage points in most cases and makes Llama-3-70B-Instruct perform even better than GPT-4-0125-preview on the RewardBench [17] reasoning subset.

Our contributions are threefold:

• With a simple logical proof, we pinpoint the causes of position bias in transformers: causal attention and position encoding (Section 3.2). We also give intuitive explanations (Section 3.2) with supporting experimental results (Figure 1 and Section 4.3).

• We propose a training-free approach dubbed PINE that can eliminate the position bias given text segments presumed to be position-invariant (Section 3.3).

• We conduct experiments on two tasks where LMs show severe position biases: retrieval-augmented question answering and LM-as-a-judge and find PINE can consistently improve model performance across model sizes, as well as model reliability (since PINE achieves zero variance), and perform better than other possible design choices. (Section 4)

2 Related Work

Position Encoding and Attention Mask Position encoding is the key component in the Transformer architecture [33]. At first, researchers use absolute positions encoding [33, 14], and then switch to relative position encodings [29, 26, 2, 20]. Recently, researchers have found that incorporating position encoding and attention computation together, instead of doing the two isolatedly, can achieve better language understanding capabilities for LMs [10, 21]. In the rest of the paper, we refer position encoding to RoPE [29] as it is adopted in most modern LMs [2, 4]. Attention mask is another elementary component that affects the fundamental capabilities of LMs. The encoder-only models such as BERT [14] use bi-directional attention. Decoder-only models usually use unidirectional causal attention [25] due to the auto-regressive nature of language generation. Although several works use the encoder-decoder model [26, 30] or prefix-LM [30] that combine both bidirectional attention and causal attention, they suffer from difficulty in scaling up training. In this paper, we investigate causal attention as it is the choice of most modern LMs [2, 4].

Position Bias in LMs. There is a lot of work demonstrating the existence and significance of position bias in LMs [43, 42, 35, 44, 8, 19, 28]. The LM-as-a-judge task offers models two candidate responses to a question and asks models to select the more helpful one. It turns out that LM has a primacy bias that tends to favor the first response [43]. Retrieval-augmented QA asks LM to answer a question based on retrieved documents. [19, 24] find that LMs are prone to answer correctly when the document that contains the correct answer is presented at the beginning and the end of retrieved documents. In the multiple-choice QA where models are required to select the correct answer to a question from multiple candidate answers, [42] points out that models favor options at certain positions (e.g., prefer “A”). In the in-context learning tasks, [40, 38] find that the order of in-context examples affects the final performance. Recently, several papers propose to understand the nature of position bias through prompting [41] and calibration [12]. Our paper analyzes the phenomenon from the computation: the computation must be positional-invariant to order to eliminate position bias.

Position Bias Solutions in LMs. There are many solutions to mitigate position bias (e.g., data augmentation and training [13, 44], content resorting by attention value during inference [24], searching [39], calibration under relatively strong assumptions [12]). Moving one step forward, some other solutions are designed to eliminate position bias. [34] output a compromised result “tie” when position bias happens in the LM-as-a-judge task [34], however, the “tie" result is still suboptimal as it actually “refuses" to give a prediction. [42, 43]) use permutation then average on classification tasks, which will have unacceptable $\mathcal{O}(k!)$ ($k$ is the number of segments) computational overhead when $k$ is large. Different from them, we aim to eliminate the position bias from the angle of transformers’ computation flows while tackling the above issues.

Moreover, we find several methods that are originally designed for other purposes (e.g., long-context understanding) have mathematical guarantees to intrinsically eliminate position bias [27, 7, 11] (Section 3.3). However, these methods obtain poor performance on settings that require language modeling (Section 4), therefore they are limited to certain tasks like in-context learning classification [27, 7, 11]. In contrast, our method is training-free and is shown to be effective in tasks that require language modeling, such as LM-as-a-judge [43] and lost-in-the-middle [19].

3 Methodology

In this section, we aim to propose an approach that can intrinsically eliminate position bias. We start by running an example to illustrate position bias, followed by analyzing the cause of position bias, and end with our own approach PINE.

3.1 Formulation

We take retrieval-augmented QA as an example, where current LMs’ performance may greatly suffer from position bias [19]. The task requires the model to answer a question based on a set of given retrieved documents, where only one of them contains the correct answer. A system prompt SYS for this task looks like this: “Write a high-quality one-sentence answer for the given question using only the provided search results (some of which might be irrelevant).”. Given a question Q: “Who won the first Nobel Prize in Physics?”, and three retrieved documents: $\mathcal{D}_{1}$, $\mathcal{D}_{2}$, and $\mathcal{D}_{3}$, we can formulate several different inputs. For example, $[\textsc{SYS}|\textsc{Q}|\mathcal{D}_{1}|\mathcal{D}_{2}|\mathcal{D}_{3}]$, and $[\textsc{SYS}|\textsc{Q}|\mathcal{D}_{2}|\mathcal{D}_{3}|\mathcal{D}_{1}]$. We expect models to have the same output for these inputs because $\mathcal{D}_{2},\mathcal{D}_{3},\mathcal{D}_{1}$ are position-agnostic input segments: their relative order is not supposed to affect the final result. However, the current LMs answer differently when presented with these different inputs and tend to answer correctly when the document contains the answer at the beginning or at the end of all documents [19]. The systematic differences of model outputs caused by relative positions of position-agnostic input segments reflect the position bias of the model. Therefore, current LMs cannot conduct inter-segment position-invariant inference, and our goal is to make the inference invariant w.r.t. relative segment orders.

3.2 Causal Attention and RoPE Are The Cause of Position Bias

Feed-Forward Networks (FFNs), Query, Key and Value (QKV) projections, and layer normalization in the Transformer architecture do not cause position bias, as they are invariant to relative segment positions. Rather, the attention computation that leads to the position bias:

$$\mathbf{Q}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}=\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}(\mathbf{Q},\mathbf{pos}_{\mathbf{Q}}),\mathbf{K}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}=\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}(\mathbf{K},\mathbf{pos}_{\mathbf{K}})$$ (1)

$$\mathbf{H}=\text{Softmax}\left(\mathbf{Q}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}\mathbf{K}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}^{T}/\sqrt{d}\right)\odot{\color[rgb]{0.21,0.49,0.74}\definecolor[named]{pgfstrokecolor}{rgb}{0.21,0.49,0.74}\mathbbm{1}_{\text{causal}}}\mathbf{V}$$

where $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{n\times d}$ are queries, keys, and values, PE donotes the position encoding (specifically RoPE), $\mathbf{pos}_{\mathbf{Q}}$ and $\mathbf{pos}_{\mathbf{K}}$ denote the position of queries and keys, and $\mathbbm{1}_{\text{causal}}$ denotes the causal attention mask. Eq. 1 reveals that (1) the PE function yields different representations for input segments if their relative order changes, therefore affecting the similarity score $Q_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}K_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}}^{T}$ and hidden states; (2) the $\mathbbm{1}_{\text{causal}}$ generates different attention masks for the input segments if we change their positions, resulting in different hidden states. To achieve inter-segment position-invariant inference, $\mathbf{H}$ needs to remain the same regardless of segment orders.

The effect of position encoding and causal attention. RoPE has been shown to have recency bias due to its mathematical long-form weight decay [29, 24]. However, LMs have also been shown to have primacy bias [19, 35]. Therefore, causal attention is likely the cause for models to favor distant content. Figure 1 (lower left) supports this hypothesis. Intuitively, the causal attention propagates the distant content to the nearby content. Therefore, nearby tokens also contain information about distant tokens, potentially making models favor distant content after the aggregation. Section 4.3 provides more supportive experiment results to our hypothesis.

3.3 PINE: Inter-Segment Position-Invariant Inference via Bidirectional Attention

Figure 2: Previous work PCW [27] eliminates position bias by first masking all inter-segment attention and then assigning all segments the same positions. The attention matrix of the running example in Section 3.1 is at the left of the figure, the orange, different blue, and green colors denote system prompts (1 token), three different documents (2 tokens each) and decoded tokens (1 token), respectively. The number at $(i,j)$ in the figure, $p_{ij}$, denotes the position of a token $j$ when computing the attention from query $\mathbf{q}_{i}$. Therefore, $p_{\cdot j}$ is equal for all $i$ in vanilla inference. Our experiment in Section 4 shows that PCW brings severe performance drop for tasks requiring language modeling capabilities, thus a suboptimal design choice.

Figure 3: PINE: inter-segment position-invariant inference via bidirectional attention. The notions are kept the same as Figure 2. PINE enables inter-segment bidirectional attention and then uses attention scores between segments to compute their similarities. Then, segments are re-sorted by similarities: higher-similarity segments are placed in closer positions. The computation of “Similarity” is introduced in Section 3.3.

Technically, our goal is to obtain an inter-segment position-invariant hidden state $\mathbf{H}_{\text{PINE}}$, which does not change regardless of the segment orders. We can mechanistically eliminate the position bias by equally attending to all position-agnostic input segments. Previous work PCW [27] achieves this goal by masking inter-segment attention and assigning all segments the same position (Figure 2). However, masking all inter-segment attention loses contextual information and introduces out-of-distribution (OOD) operations, which harm model performance (Section 4). For example, models are never trained to compute attention with tokens sharing the same position.

Therefore, we propose PINE, an approach that use bidirectional inter-segment attention and reassigning positions by similarities (computed from attention score) to eliminate position bias (Figure 3). It is worth noting that our goal is to eliminate “inter” position bias among different segments rather than “intra” position bias within each segment. The latter will lose the order information for tokens, and models can degenerate into bag-of-words models, which is not what we expect. For example, the intra-segment attention is causal, but the inter-segment attention is bidirectional in PINE.

Bidirectional Attention. We first change the attention mask so that segments can attend to each other. Specifically, we make the inter-segment attention bidirectional but keep the intra-segment attention causal (the middle figure in Figure 3).

Re-sorting Positions by Attention Weights. It is challenging to assign positions for bidirectional attention masks. First, LMs are trained in a causal way, and each token can only attend to previous tokens. Therefore, it is necessary to manipulate positions so that each segment treats itself as the last segment (the diagonal of the rightmost figure in Figure 3). Second, we determine the positions of other segments based on similarities. Specifically, we first compute the attentions without RoPE involved: $\text{Sim}_{\text{token}}=\text{Softmax}(\mathbf{Q}\mathbf{K}^{T}/\sqrt{d})$, where $d$ is the hidden state dimension. Then, we obtain the similarity between segments by aggregation. For example, $\text{Sim}(\mathcal{D}_{1},\mathcal{D}_{2})=\sum_{i\in\mathcal{D}_{1},j\in\mathcal{D}_{2}}\text{Sim}_{\text{token}}(i,j)/|\mathcal{D}_{2}|$. The length normalization is to prevent assigning higher similarities to longer segments. Lastly, We reassign positions by similarities as shown in the rightmost part of Figure 3. To avoid confusion, we address that we do not actually sort tokens and only assign them to different positions.

The rightmost part of Figure 3 contains the $\mathbf{pos}_{\mathbf{Q}},\mathbf{pos}_{\mathbf{K}}$ and attention mask $\mathbbm{1}_{\text{PINE}}$. Putting them in Equation 1, and we obtain $\mathbf{H}_{\text{PINE}}$. Regardless of the segment orders in inputs, PINE attends each segment to all other segments and uses similarities to decide their relative positions, generating $\mathbf{H}_{\text{PINE}}$ that is not affected by segment orders in inputs. Therefore, PINE achieves inter-segment position-invariant inference in a training-free zero-shot manner.

3.4 Discussion

Alternative options: re-sorting reversely or using the order in inputs. PINE puts segments with higher similarity to a closer relative position. Another option is to put segments with higher similarity to a more distant position. Considering the recency bias brought by RoPE, the latter choice causes a conflict between model judgment and RoPE, whereas the former choice enables RoPE to amplify model judgment further. Therefore, we believe this alternative choice is not optimal, which is justified by our experiments in Section 4.4. Another option is that we do not conduct re-sorting but use the relative order provided in the input. For example, the hidden state of $\mathcal{D}_{1}$ becomes:

$$\mathbf{Q}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}},\mathcal{D}_{1}} =\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}(\mathbf{Q}_{\mathcal{D}_{1}},[6,7])$$ (2)

$$\mathbf{K}^{\prime}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}} =\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}([\mathbf{K}_{\textsc{SYS}},\mathbf{K}_{\mathcal{D}_{2}},\mathbf{K}_{\mathcal{D}_{3}},\mathbf{K}_{\mathcal{D}_{1}}],\mathbf{pos}_{\text{seq}})$$

$$\mathbf{H}_{\mathcal{D}_{1}} =\text{Softmax}(\mathbf{Q}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}},\mathcal{D}_{1}}(\mathbf{K}^{\prime}_{\text{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}PE}}})^{T}/\sqrt{d})\mathbf{V}$$

where $\mathbf{pos}_{\text{seq}}$ denotes sequential positions, e.g., $[1,2,3,4,5,6]$. This option is more computationally efficient due to lack of re-sorting, but it cannot achieve inter-segment position invariant and has worse performance than PINE (Section 4.4).

Inference Cost. PINE incurrs additional computation overhead due to extra operations. Specifically, PINE conducts one more step of attention computation, attention re-sorting, and position reassignment. Practically, the current computation complexity to obtain hidden states is $\mathcal{O}(n^{2}d+nk\log k)$, where $n$, $d$ and $k$ denote input length, hidden state dimension, and the number of input segments, respectively. Compared with the vanilla hidden states computation that requires $\mathcal{O}(n^{2}d)$, PINE has extra $\mathcal{O}(nk\log k)$ cost. The real computation cost is acceptable since $k$ is usually small (e.g., $k=2$ in the LLM-as-a-judge task and $k=20$ in the retrieval-augmented QA). Section 4.5 shows results of real-world wall time and memory cost.

Practical Implications. Although PINE eliminates output variance w.r.t. different input segment orders in principle, the practical solution still has a little variance due to the numerical issue (Softmax operations are prone to produce equal small-value similarities). To ensure zero variance, one can add deterministic rules, such as using length or alphabetic order to determine the order in such a case.

4 Experiment

In the following experiments, we mainly show two aspects of PINE: (1) PINE achieves superior performance without variance when models suffer from position bias. (2) PINE performs the best among other design choices that aim to eliminate position bias intrinsically.

Table 1: The portion of data (%) that models have position bias in RewardBench, i.e., models change answers after swaping candidate responses orders. We color the subsets that have more than $25\%$ data causing position bias with cyan.

Model	Size	Chat	Chat-Hard	Safety	Reasoning	Avg.
LLaMa-3 -Instruct	8B	$10.3$	$21.5$	$11.4$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}27.6}$	$17.7$
	70B	$3.6$	$16.0$	$5.8$	$15.2$	$10.2$
Qwen-1.5 7B-Chat	1.8B	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}33.5}$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}37.9}$	$24.7$	$13.3$	$27.4$
	4B	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}48.0}$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}38.6}$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}57.4}$	$12.7$	$39.2$
	7B	$17.0$	$20.6$	$10.9$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}26.5}$	$18.8$
	32B	$7.8$	$20.0$	$9.6$	${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}26.4}$	$16.0$
	72B	$10.9$	$22.6$	$9.6$	$24.7$	$17.0$
	110B	$8.7$	$16.0$	$11.5$	$23.5$	$14.9$

Table 2: Main results of RewardBench. Vanilla denotes the normal inference, (GT at A) means the ground truth chosen response is presented at the first, and (GT at B) indicates at the second. PINE consistently improves LM’s performance across different model sizes. Consistent to Table 1, we color the subsets with severe position bias cyan. It can be observed that PINE generally improves performance on cyan subsets by a large margin, which is consistent to our motivation and goal.

Model Size Method Chat Chat-Hard Safety Reasoning Avg. LLaMa-3 -Instruct 8B Vanilla (GT at A) $90.1$ $35.2$ $64.6$ $80.3$ $67.5$ Vanilla (GT at B) $85.3$ $48.7$ $65.3$ $66.0$ $66.3$ Vanilla $85.3$ $41.6$ $67.0$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}65.3}$ $64.8$ PINE $85.6$ $41.5$ $66.5$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}73.4}$ $\bm{66.7}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+1.9}}}$ 70B Vanilla (GT at A) $98.6$ $52.0$ $73.6$ $87.8$ $78.0$ Vanilla (GT at B) $93.9$ $62.1$ $69.8$ $80.3$ $76.5$ Vanilla $97.4$ $58.3$ $69.6$ $78.9$ $76.0$ PINE $96.9$ $57.4$ $67.7$ $87.6$ $\bm{77.4}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+1.4}}}$ Qwen-1.5 -Chat 1.8B Vanilla (GT at A) $31.7$ $30.0$ $40.3$ $43.3$ $36.3$ Vanilla (GT at B) $69.4$ $72.6$ $65.7$ $57.2$ $66.2$ Vanilla ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}49.7}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}50.9}$ $52.0$ $48.4$ $50.3$ PINE ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}30.0}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}59.9}$ $61.4$ $60.1$ $\bm{52.9}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+2.6}}}$ 4B Vanilla (GT at A) $32.8$ $24.8$ $17.4$ $42.8$ $29.5$ Vanilla (GT at B) $86.6$ $74.5$ $82.9$ $62.3$ $76.6$ Vanilla ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}58.9}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}48.7}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}50.9}$ $54.1$ $53.1$ PINE ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}73.0}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}45.2}$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}53.7}$ $61.0$ $\bm{58.2}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+5.1}}}$ 7B Vanilla (GT at A) $85.5$ $35.9$ $62.4$ $62.1$ $61.4$ Vanilla (GT at B) $77.1$ $47.4$ $59.5$ $54.3$ $59.6$ Vanilla $77.5$ $44.2$ $62.6$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}59.3}$ $60.9$ PINE $85.8$ $38.7$ $58.6$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}63.0}$ $\bm{61.5}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+0.6}}}$ 32B Vanilla (GT at A) $93.6$ $47.7$ $77.1$ $78.3$ $74.2$ Vanilla (GT at B) $91.9$ $52.2$ $81.6$ $73.6$ $74.8$ Vanilla $92.7$ $51.2$ $80.5$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}66.8}$ $72.8$ PINE $93.0$ $49.8$ $79.7$ ${\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}76.7}$ $\bm{74.8}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+2.0}}}$ 72B Vanilla (GT at A) $95.7$ $59.0$ $80.8$ $83.0$ $79.6$ Vanilla (GT at B) $89.0$ $46.5$ $73.7$ $68.7$ $69.5$ Vanilla $94.0$ $51.4$ $77.8$ $68.2$ $\bm{72.8}$ PINE $93.9$ $46.1$ $78.2$ $69.0$ $71.8_{\bm{{\color[rgb]{0.859375,0.265625,0.21484375}\definecolor[named]{pgfstrokecolor}{rgb}{0.859375,0.265625,0.21484375}-1.1}}}$ 110B Vanilla (GT at A) $98.6$ $70.5$ $89.6$ $90.0$ $87.2$ Vanilla (GT at B) $91.1$ $59.2$ $79.5$ $73.0$ $75.7$ Vanilla $96.2$ $66.7$ $83.7$ $78.0$ $81.1$ PINE $95.5$ $64.8$ $85.0$ $86.2$ $\bm{82.9}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+1.7}}}$

4.1 Settings

Tasks, Data and Configs. We select two representative tasks that pose position bias: LM-as-a-judge [43] and retrieval-augmented question-answering [19]. The former is known for the recency bias, whereas the latter shows both primacy and recency bias. To systematically test the LM-as-a-judge task, we benchmark our method on 23 datasets in the RewardBench [16] that can be categorized into four types: Chat, Chat-Hard, Safety, and Reasoning. LMs are asked to select the more helpful one from two given chat responses to a question. We use the official data split, prompts, and evaluation scripts to ensure reproducibility. We follow the settings and use the prompts, data, and evaluation scripts of [19] to benchmark retrieval-augmented question-answering: only one of the retrieved documents (10 or 20 in total) contains the ground-truth answer for the given question. We list prompts in Appendix B. We use LLaMa-3-Instruct models [2] and Qwen-1.5-Chat models [4] for experiments. More details can be found in Appendix B. Qualitative examples of PINE can be found in Appendix C.

Baselines. The goal of PINE is to eliminate position bias during inference intrinsically. Therefore, we choose methods that have the same design principle as our baselines: (1) Vanilla inference (2) Vanilla inference with no inter-segment attention (NIA for short, i.e., the middle figure in Figure. 2): The latter segments will have no attention to formers. (3) Parallel Context Window (PCW, rightmost in Figure. 2) [27]: PCW extends the baseline (2) by manipulating positions of segments. PCW allows all segments to share the same positions. (4) Structured Prompting (SP, a variant version of PCW) [11]: SP extends (3) by lowering attentions between decoded tokens and input segments to $\frac{1}{N}$ to solve the perplexity exploding problem in PCW. Based on the analysis in Section 3.3, (1) and (2) are not inter-segment position invariant, whereas (3) and (4) are.

Table 3: Baseline performance on RewardBench. PINE achieves superior performance to baseline models, performing $4.8\%$ and $4.7\%$ better than the best performed baseline on two models.

Model	Method	Chat	Chat-Hard	Safety	Reasoning	Avg.
LLaMa-3 8B-Instruct	NIA (GT at A)	$81.0$	$40.7$	$59.7$	$43.7$	$56.3$
	NIA (GT at B)	$81.0$	$49.7$	$65.8$	$66.7$	$65.8$
	NIA	$80.9$	$46.7$	$64.0$	$55.9$	$61.9$
	PCW	$78.6$	$46.8$	$64.8$	$56.5$	$61.7$
	SP	$79.6$	$43.3$	$65.0$	$55.4$	$60.8$
	PINE	$85.6$	$41.5$	$66.5$	$73.4$	$\bm{66.7}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.8}}}$
Qwen-1.5 7B-Chat	NIA (GT at A)	$67.7$	$57.2$	$59.6$	$60.7$	$61.3$
	NIA (GT at B)	$67.9$	$35.9$	$61.0$	$44.1$	$52.2$
	NIA	$74.9$	$43.5$	$57.4$	$51.4$	$56.8$
	PCW	$67.2$	$42.0$	$58.3$	$53.4$	$55.2$
	SP	$69.4$	$41.8$	$58.0$	$52.4$	$55.4$
	PINE	$85.8$	$38.7$	$58.6$	$63.0$	$\bm{61.5}_{\bm{{\color[rgb]{0.05859375,0.6171875,0.34375}\definecolor[named]{pgfstrokecolor}{rgb}{0.05859375,0.6171875,0.34375}+4.7}}}$

4.2 Results on LM-as-a-judge

Table 1 shows the statistics of position bias in RewardBench with different models. Position bias is quite common in RewardBench, and can be up to $48.0\%$. Although the table reflects that generally larger models have less position bias. The position bias could still on average affect up to $10\%$ data. Table 2 shows the main results on RewardBench. We experiment with Llama-3 and Qwen-1.5 across different model sizes. The position of the ground truth chosen option is randomly shuffled. Therefore, the accuracy of the random guess method is expected to be $50\%$. First, the first two rows of each model block reveal that larger models tend to have a primacy bias, whereas smaller models tend to have a recency bias. Second, position bias exists in different model sizes; for example, Qwen1.5-110B-Chat and Qwen1.5-1.8B-Chat models all have obvious position bias due to the large performance gap in the first two rows of each model block ($11.5\%$ and $29.9\%$). By comparing the last two rows of each model size, we conclude that models across different sizes perform better with the help of PINE by eliminating position bias. The only exception is the Qwen-1.5-72B-Chat model. We suspect this model is not well-trained since Qwen-1.5-32B-Chat performs extremely similarly to the 72B model in vanilla inference. PINE consistently improves model performance on the “reasoning" subset by a large margin: from $8$ to $10$ percentage points in most cases. For example, LlaMa-3 Instruct 70B is originally is ranked 8th generative model in the reasoning subset of RewardBench. With PINE, it achieves the 3rd rank ($87.6\%$), outperforming GPT-4-0125-preview (the previous 3rd rank, $86.9\%$), and is only worse than two Google’s most recent models (flame-24b-july-2024, gemini-1.5-pro-0514). Moreover, PINE largely improves subsets with severe position bias (colored with cyan), which is consistent to our motivation and goals. We also observe that PINE sometimes brings an unusual performance drop in small models, which is as expected since position bias is not always a bad phenomenon: if the model capability is worse than random guessing, then position bias can improve the overall performance toward $50\%$; however, if the model is very capable, then the position bias will harm the overall performance toward $50\%$. Overall, PINE improves performance from a statistical perspective and makes models more reliable when as evaluators.

We then compare PINE with baseline models on Llama-3-8B-Instruct and Qwen1.5-7B-Chat model. Since NIA is not inter-segment position invariant, we also apply NIA with two extreme cases: the ground truth chosen response is always in the first or second place. Results on Table 3 show that PINE achieves the best performance and largely outperforms the best baselines by $\sim 5\%$, and outperforms NIA even if NIA is placed in the extreme case. The results reveal that masking inter-segment attention mask is much less effective than bidirectional inter-segment attention mask applied in PINE.

4.3 Results on Retrieval-Augmented Question-Answering

Figure 4: The results of retrieval-augmented QA on Llama-3-70B-Instruct. Dashed lines indicate that the method is either inter-segment position-invariant or the result is obtained on the order-shuffled data (denoted in the legend). (a) shows results of PINE against baselines. (b) shows results of different designs of PINE.

Models tend to perform better when the gold-standard document is at the beginning and the end of all documents in retrieval-augmented question-answers. Figure 4 (a) shows the results on LLaMa-3-70B-Instruct when 10 or 20 documents were presented. First, it is easy to conclude that all baselines are much worse than PINE (the pink line), which is consistent to the previous experiment. Second, PINE  achieves on-par performance on average compared with vanilla inference while being inter-segment position invariant. Specifically, PINE  is slightly better/worse than vanilla inference with the gap +1.2/-2.0 when there are 10 and 20 documents in total. We hypothesize that the slight performance drop of PINE  for the 20 document setting is due to the performance drop of segment similarity computation in PINE when presented with many documents. However, PINE is position-invariant, therefore does not be affected by the worst case (the bottom of blue solid curves). Third, the height generally becomes smaller between blue and brown solid lines in Figure 4 (a), and between the blue and red solid lines in Figure 4 (b) when the gold-standard document position increases, reflecting the causal attention generally prefers distant content, which is consistent to the hypothesis in Section 3.2. The brown line in Figure 4 (a) and red line (b) generally reflect recency bias brought by RoPE, which is consistent to previous works [29, 24].

4.4 Ablation Study

So far, our experiments show that bidirectional inter-segment attention is the better design choice than the masked one. However, there are still several design options for the position assignment, as discussed in Section 3.4. The first option is to re-sort position reversely, and the other is to use PINE without re-sorting. To gain a deeper understanding, we extend the retrieval-augmented QA experiments with the two mentioned alternative position assignment methods, and the results are presented in Figure 4 (b). The figure tells us that PINE is slightly better than PINE  without re-sorting on average (+0.3. The gap becomes larger when 20 documents are presented: +1.5). Besides, segment positions still affect performance without re-sorting. Therefore, PINE without re-sorting may suffice if one does not aim to eliminate the position bias or cares more about efficiency. On the other hand, re-sorting reversely has relatively worse results, showing that PINE is a better design choice, which is consistent with the intuitive analysis mentioned at Section 3.4.

4.5 Computational Overhead

Section 3.4 briefly discusses the computational overhead, with a conclusion that PINE ’s efficiency is still doable. In our experiments, we find the wall time of PINE is $\sim$2x and $\sim$8x of the vanilla inference on the LM-as-a-judge task and retrieval-augmented QA task with 20 documents, which is acceptable at least during experiments. However, we did not specially optimize codes to accelerate PINE, and our implementation still contains a “for” loop. Therefore, we believe there is room to accelerate PINE. Compared with the time overhead, the memory overhead is small and PINE can be run with 70B models on 3x A100 80G on the retrieval-augmented QA task, which requires the same number of GPUs as the vanilla inference. Since efficiency is not the main focus of this paper, we leave this as our future work.

5 Conclusion, Limitations and Future Work

This paper logically analyzes the transformers and attributes the position bias to causal attention and RoPE. Based on the analysis, we propose a novel train-free zero-shot approach to intrinsically eliminate the position bias. The core idea is to make every input segment (e.g., options and retrieved documents) equally affected by the attention mask and RoPE. Experiments on the LM-as-a-judge and retrieval-augmented QA show the effectiveness of our approach. However, PINE requires extra computation, and thus it is not suitable for scenarios that are efficiency-sensitive. We believe there is some room to improve the efficiency with more automated end-to-end implementation, and we leave this as our future work. PINE works only for input segments that are well-defined and structured to be position-invariant. Therefore, users are expected to tell models the parts of inputs that should be position-invariant. Finally, PINE is an early attempt to eliminate position bias that builds on existing attention and position encoding, and we believe that novel designs of position encoding and attention may be a better solution, which is left to be explored in the future.

Appendix A Another Example of Position Bias in VLMs

Figure 5: Image of Andromeda Galaxy. GPT-4V is asked to answer the galaxy of the redbox (M110). (b), (c), (d) images are generated by flipping (a). Results show that GPT-4V can answer correctly on (c) and (d) and incorrectly on (a) and (b), revealing position bias in VLMs.

To show VLMs suffer from position bias, we ask models to identify a small object appeared in the full image. We ask GPT-4V [22] to identify one satellite galaxy M110 of Andromeda Galaxy (M31). The central objective in the Figure 5 (a) is the Andromeda Galaxy and the galaxy in the red box is M110 satellite galaxy. Another satellite galaxy is M32 which is the brightest part near the Andromeda Galaxy.

We flip the image (a) and obtains images (b), (c), and (d). We prompt GPT-4V “What galaxy is in the red box?”, and here are the responses we collected:

(a): The galaxy within the red box is M32, also known as Messier 32. It’s a dwarf elliptical galaxy and is a satellite of the Andromeda Galaxy (M31), which is the prominent galaxy in the image. M32 is one of the more well-studied dwarf elliptical galaxies, largely due to its proximity to our own galaxy and its interactions with the Andromeda Galaxy.

(b): The object inside the red box in the image appears to be a smaller galaxy located near the larger, more prominent galaxy (Andromeda Galaxy, M31). This smaller galaxy is likely M32, a satellite galaxy of the Andromeda Galaxy. M32 is a compact elliptical galaxy and is often visible in images as a smaller, dense, bright spot near Andromeda, as seen in the boxed area of your image.

(c): The galaxy within the red box is M110, also known as NGC 205. M110 is a dwarf elliptical galaxy, which, like M32, is a satellite of the Andromeda Galaxy (M31). It is one of the members of the Local Group of galaxies, which includes the Milky Way, the Andromeda Galaxy, and about 54 other known galaxies.

(d): The galaxy in the red box appears to be a satellite galaxy of the Andromeda Galaxy, which is the prominent galaxy visible in the rest of the image. This smaller galaxy is likely M110 (NGC 205), which is a dwarf elliptical galaxy and a companion to the Andromeda Galaxy, M31. It’s one of the several satellite galaxies gravitationally bound to Andromeda, visible here as a faint, elongated object in the outlined area.

We can find that models answer corrected when M110 is at the top of the image, revealing that VLMs also suffer from the position bias. The position bias may lead unreliable VLMs when fine-grained image analysis are needed (e.g., small object detection [37]).

Appendix B Implementation Details

B.1 Experiment Setting

For reproducibility, the generation temperature is set to 0. We use PyTorch [3, 23], Transformers [36], and vLLM [15] for our experiments. All experiments are launched with a single node of 8x A100 80G with SXM connection. 70B and 110B models are launched with 3x and 4x A100, and other model sizes can be launched with 1x A100.

B.2 Prompts

We use the prompts provided by RewardBench [17] official repo for the LM-as-a-judge task:

System prompt:

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user’s instructions and answers the user’s question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better.

User prompt:

[User Question]

......

[The Start of Assistant A’s Answer]

......

[The End of Assistant A’s Answer]

[The Start of Assistant B’s Answer]

......

[The End of Assistant B’s Answer]

We use the prompts of official repo [19] for the retrieval-augmented QA experiments:

User Prompt:

Write a high-quality one-sentence answer for the given question using only the provided search results (some of which might be irrelevant).

Document (Title: ......): ......

Document (Title: ......): ......

......

Document (Title: ......): ......

Document (Title: ......): ......

Question: ......

Appendix C Qualitative Examples

C.1 LM-as-a-judge

[User Question]

Who created the Superman cartoon character?

[The Start of Assistant A’s Answer]

The Superman cartoon character was created by Jerry Siegel and Joe Shuster in 1938.

[The End of Assistant A’s Answer]

[The Start of Assistant B’s Answer]

Superman, the iconic comic book superhero, was created by writer Jerry Siegel and artist Joe Shuster. Superman first appeared in Action Comics #1, which was published by Detective Comics, Inc. (later DC Comics) in June 1938. The character’s immense popularity established him as one of the most enduring and recognizable figures in the superhero genre.

[The End of Assistant B’s Answer]

Here are outputs of Qwen1.5-7B-Chat:

Vanilla: Both AI assistants provide accurate and informative responses to the user’s question. Assistant A’s answer is brief and directly states the creators’ names and the year of creation, while Assistant B offers additional context by mentioning the first appearance, the publisher, and the character’s significance in the superhero genre. However, since the user asked for the creators, Assistant A’s answer is more directly helpful in this specific context. Final verdict: [[A]]

Vanilla (Swapped): [[B]]. Assistant B’s response is more detailed and informative, providing additional context about the origin of the character in Action Comics #1, the publisher, and its significance in the superhero genre. It also includes the names of the creators, which is crucial information for the question. Assistant A’s response is shorter and lacks the historical context and specific details. Therefore, Assistant B is better in answering the user’s question.

PINE: [[B]]. Assistant B provides a more comprehensive and detailed response to the user’s question. It not only mentions the creators but also provides additional information about the character’s debut and the company that published the comic. This answer is not only accurate but also offers context that highlights the significance of the creation. Assistant A’s response is brief and lacks the depth and historical context provided by Assistant B.

C.2 Retrieval-Augmented Question-Answering

User Prompt:

Document (Title: Pearl) south Pacific and Indian Ocean. The largest pearl oyster is the "Pinctada maxima", which is roughly the size of a dinner plate. South Sea pearls are characterized by their large size and warm luster. Sizes up to 14 mm in diameter are not uncommon. In 2013, Indonesia Pearl supplied 43 percent of South Sea Pearls international market. The other significant producers are Australia, Philippines, Myanmar and Malaysia. In 1914, pearl farmers began growing cultured freshwater pearls using the pearl mussels native to Lake Biwa. This lake, the largest and most ancient in Japan, lies near the city of Kyoto. The

Document (Title: Laccadive Sea) the gulf as most productive in the world. Although extraction of natural pearls is considered too expensive in most parts of the world, it is still conducted in the gulf. Also collected in large numbers are Shankha mollusks ("Xancus pyrum") whose shells are used as a ritual and religious object. Other mollusks of the sea are either too scarce or not popular in the Indian society and therefore have no commercial value. Another traditional occupation in the Laccadive Sea is fishing. The annual fish catch is 2,000 to 5,000 tonnes from the Lakshadweep islands, which is mostly constituted by tuna

Document (Title: Pearl) including the Cook Islands and Fiji are being extensively used for producing cultured pearls. The rarity of the black cultured pearl is now a "comparative" issue. The black cultured pearl is rare when compared to Chinese freshwater cultured pearls, and Japanese and Chinese akoya cultured pearls, and is more valuable than these pearls. However, it is more abundant than the South Sea pearl, which is more valuable than the black cultured pearl. This is simply because the black pearl oyster "Pinctada margaritifera" is far more abundant than the elusive, rare, and larger south sea pearl oyster "Pinctada maxima", which cannot

Document (Title: Pearl powder) Pearl powder Pearl powder () is a preparation of crushed pearls used in China and elsewhere for skin care and in traditional Chinese medicine. Pearl powder is made from freshwater pearls or saltwater pearls below jewellery grade. These are sterilised in boiling water and then milled into a fine powder using stainless steel grinding discs or by milling with small porcelain balls in moist conditions. The powder is sold as such or mixed into creams. Pearl powder is widely believed to help improve the appearance of the skin, and is used as a cosmetic by royal families in Asia. It

Document (Title: Hyderabad pearl) with white pearls. Recently, several pearl makers are exporting processed pearls to markets in Europe and the US. With the capital that they gain from this marketing, they are able to purchase machinery for advanced refinement. In particular, equipment that uses enzymes present in thermophiles is able to substantially improve the process of refining pearls. Hyderabad pearl Hyderabad is considered the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they

Document (Title: Pearl) pearls". The correct definition of a South Sea pearl – as described by CIBJO and GIA – is a pearl produced by the "Pinctada maxima" pearl oyster. South Sea pearls are the color of their host "Pinctada maxima" oyster – and can be white, silver, pink, gold, cream, and any combination of these basic colors, including overtones of the various colors of the rainbow displayed in the pearl nacre of the oyster shell itself. South Sea pearls are the largest and rarest of the cultured pearls – making them the most valuable. Prized for their exquisitely beautiful órientór lustre,

Document (Title: Chandrani Pearls) year 2007–08 Chandrani Pearls imported their pearls from Japan, China or Korea. Chandrani Pearls Chandrani Pearls is a prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls’s headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on 24 January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi Nayar and his father late Mr. N.C. Nayar in Kolkata’s up market Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have 63 showrooms across 9 states. From a modest turnover of Rs.

Document (Title: Pearl) For thousands of years, seawater pearls were retrieved by divers in the Indian Ocean in areas such as the Persian Gulf, the Red Sea and the Gulf of Mannar. Evidence also suggest a prehistoric origin to pearl diving in these regions. Starting in the Han Dynasty (206 BC–220 AD), the Chinese hunted extensively for seawater pearls in the South China Sea. In the 14th-century Arabian Sea, the traveller Ibn Battuta provided the earliest known description of pearl diving by means of attaching a cord to the diver’s waist.

Document (Title: Chandrani Pearls) Chandrani Pearls Chandrani Pearls is a prominent pearl jewelery brand of India. It pioneered the concept of pearls in India. Chandrani Pearls’s headquarters is at Kolkata in West Bengal. Chandrani Pearls was started on 24 January 1985 by Mr. Kuldip Nayar, his wife Mrs. Lakshmi Nayar and his father late Mr. N.C. Nayar in Kolkata’s up market Minto Park area. Chandrani Pearls management is now assisted by Nisheeth Nayar, sons of Mr. Kuldip Nayar. Chandrani Pearls have 63 showrooms across 9 states. From a modest turnover of Rs. 2.75 lakhs in 1990 to a turnover of Rs. 15.00 crores in

Document (Title: Hyderabad pearl) Hyderabad pearl Hyderabad is considered the main pearl trading center in India. The most notable area devoted to the trade is the village called Chandanpet just outside Hyderabad, wherein almost the entire population is engaged in the delicate art of drilling pearls, a skill they have practiced for generations. This practice also makes the city one of the largest pearl drilling locations in India. The pearl industry in Hyderabad flourished due to the patronage of the Qutub Shahi kings and the Asaf Jahis, who were said to have an affinity for sparkling jewels. The pearls were not only part of

Question: in which sea pearl is found in india?

Here are Llama-3-70B-Instruct results:

Vanilla: According to the search results, pearls are found in the Laccadive Sea in India.

PINE: Pearls are found in the Indian Ocean, specifically in the Gulf of Mannar and the Laccadive Sea, off the coast of India.