The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks
Authors:
Anders Giovanni Møller,
Jacob Aarup Dalsgaard,
Arianna Pera,
Luca Maria Aiello
Abstract:
In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks…
▽ More
In the realm of Computational Social Science (CSS), practitioners often navigate complex, low-resource domains and face the costly and time-intensive challenges of acquiring and annotating data. We aim to establish a set of guidelines to address such challenges, comparing the use of human-labeled data with synthetically generated data from GPT-4 and Llama-2 in ten distinct CSS classification tasks of varying complexity. Additionally, we examine the impact of training data sizes on performance. Our findings reveal that models trained on human-labeled data consistently exhibit superior or comparable performance compared to their synthetically augmented counterparts. Nevertheless, synthetic augmentation proves beneficial, particularly in improving performance on rare classes within multi-class tasks. Furthermore, we leverage GPT-4 and Llama-2 for zero-shot classification and find that, while they generally display strong performance, they often fall short when compared to specialized classifiers trained on moderately sized training sets.
△ Less
Submitted 5 February, 2024; v1 submitted 26 April, 2023;
originally announced April 2023.
The Danish Gigaword Project
Authors:
Leon Strømberg-Derczynski,
Manuel R. Ciosici,
Rebekah Baglini,
Morten H. Christiansen,
Jacob Aarup Dalsgaard,
Riccardo Fusaroli,
Peter Juel Henrichsen,
Rasmus Hvingelby,
Andreas Kirkedal,
Alex Speed Kjeldsen,
Claus Ladefoged,
Finn Årup Nielsen,
Malte Lau Petersen,
Jonathan Hvithamar Rystrøm,
Daniel Varab
Abstract:
Danish language technology has been hindered by a lack of broad-coverage corpora at the scale modern NLP prefers. This paper describes the Danish Gigaword Corpus, the result of a focused effort to provide a diverse and freely-available one billion word corpus of Danish text. The Danish Gigaword corpus covers a wide array of time periods, domains, speakers' socio-economic status, and Danish dialect…
▽ More
Danish language technology has been hindered by a lack of broad-coverage corpora at the scale modern NLP prefers. This paper describes the Danish Gigaword Corpus, the result of a focused effort to provide a diverse and freely-available one billion word corpus of Danish text. The Danish Gigaword corpus covers a wide array of time periods, domains, speakers' socio-economic status, and Danish dialects.
△ Less
Submitted 12 May, 2021; v1 submitted 7 May, 2020;
originally announced May 2020.
