-
Don't mention it: An approach to assess challenges to using software mentions for citation and discoverability research
Authors:
Stephan Druskat,
Neil P. Chue Hong,
Sammie Buzzard,
Olexandr Konovalov,
Patrick Kornek
Abstract:
Datasets collecting software mentions from scholarly publications can potentially be used for research into the software that has been used in the published research, as well as into the practice of software citation. Recently, new software mention datasets with different characteristics have been published. We present an approach to assess the usability of such datasets for research on research s…
▽ More
Datasets collecting software mentions from scholarly publications can potentially be used for research into the software that has been used in the published research, as well as into the practice of software citation. Recently, new software mention datasets with different characteristics have been published. We present an approach to assess the usability of such datasets for research on research software. Our approach includes sampling and data preparation, manual annotation for quality and mention characteristics, and annotation analysis. We applied it to two software mention datasets for evaluation based on qualitative observation. Doing this, we were able to find challenges to working with the selected datasets to do research. Main issues refer to the structure of the dataset, the quality of the extracted mentions (54% and 23% of mentions respectively are not to software), and software accessibility. While one dataset does not provide links to mentioned software at all, the other does so in a way that can impede quantitative research endeavors: (1) Links may come from different sources and each point to different software for the same mention. (2) The quality of the automatically retrieved links is generally poor (in our sample, 65.4% link the wrong software). (3) Links exist only for a small subset (in our sample, 20.5%) of mentions, which may lead to skewed or disproportionate samples. However, the greatest challenge and underlying issue in working with software mention datasets is the still suboptimal practice of software citation: Software should not be mentioned, it should be cited following the software citation principles.
△ Less
Submitted 22 February, 2024;
originally announced February 2024.
-
Understanding Equity, Diversity and Inclusion Challenges Within the Research Software Community
Authors:
Neil P. Chue Hong,
Jeremy Cohen,
Caroline Jay
Abstract:
Research software -- specialist software used to support or undertake research -- is of huge importance to researchers. It contributes to significant advances in the wider world and requires collaboration between people with diverse skills and backgrounds. Analysis of recent survey data provides evidence for a lack of diversity in the Research Software Engineer community. We identify interventions…
▽ More
Research software -- specialist software used to support or undertake research -- is of huge importance to researchers. It contributes to significant advances in the wider world and requires collaboration between people with diverse skills and backgrounds. Analysis of recent survey data provides evidence for a lack of diversity in the Research Software Engineer community. We identify interventions which could address challenges in the wider research software community and highlight areas where the community is becoming more diverse. There are also lessons that are applicable, more generally, to the field of software development around recruitment from other disciplines and the importance of welcoming communities.
△ Less
Submitted 4 April, 2021;
originally announced April 2021.
-
Addressing Research Software Sustainability via Institutes
Authors:
Daniel S. Katz,
Jeffrey C. Carver,
Neil P. Chue Hong,
Sandra Gesing,
Simon Hettrick,
Tom Honeyman,
Karthik Ram,
Nicholas Weber
Abstract:
Research software is essential to modern research, but it requires ongoing human effort to sustain: to continually adapt to changes in dependencies, to fix bugs, and to add new features. Software sustainability institutes, amongst others, develop, maintain, and disseminate best practices for research software sustainability, and build community around them. These practices can both reduce the amou…
▽ More
Research software is essential to modern research, but it requires ongoing human effort to sustain: to continually adapt to changes in dependencies, to fix bugs, and to add new features. Software sustainability institutes, amongst others, develop, maintain, and disseminate best practices for research software sustainability, and build community around them. These practices can both reduce the amount of effort that is needed and create an environment where the effort is appreciated and rewarded. The UK SSI is such an institute, and the US URSSI and the Australian AuSSI are planning to become institutes, and this extended abstract discusses them and the strengths and weaknesses of this approach.
△ Less
Submitted 5 March, 2021;
originally announced March 2021.
-
Software Sustainability & High Energy Physics
Authors:
Daniel S. Katz,
Sudhir Malik,
Mark S. Neubauer,
Graeme A. Stewart,
Kétévi A. Assamagan,
Erin A. Becker,
Neil P. Chue Hong,
Ian A. Cosden,
Samuel Meehan,
Edward J. W. Moyse,
Adrian M. Price-Whelan,
Elizabeth Sexton-Kennedy,
Meirin Oan Evans,
Matthew Feickert,
Clemens Lange,
Kilian Lieret,
Rob Quick,
Arturo Sánchez Pineda,
Christopher Tunnell
Abstract:
New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will b…
▽ More
New facilities of the 2020s, such as the High Luminosity Large Hadron Collider (HL-LHC), will be relevant through at least the 2030s. This means that their software efforts and those that are used to analyze their data need to consider sustainability to enable their adaptability to new challenges, longevity, and efficiency, over at least this period. This will help ensure that this software will be easier to develop and maintain, that it remains available in the future on new platforms, that it meets new needs, and that it is as reusable as possible. This report discusses a virtual half-day workshop on "Software Sustainability and High Energy Physics" that aimed 1) to bring together experts from HEP as well as those from outside to share their experiences and practices, and 2) to articulate a vision that helps the Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP) to create a work plan to implement elements of software sustainability. Software sustainability practices could lead to new collaborations, including elements of HEP software being directly used outside the field, and, as has happened more frequently in recent years, to HEP developers contributing to software developed outside the field rather than reinventing it. A focus on and skills related to sustainable software will give HEP software developers an important skill that is essential to careers in the realm of software, inside or outside HEP. The report closes with recommendations to improve software sustainability in HEP, aimed at the HEP community via IRIS-HEP and the HEP Software Foundation (HSF).
△ Less
Submitted 16 October, 2020; v1 submitted 10 October, 2020;
originally announced October 2020.
-
Software Citation Implementation Challenges
Authors:
Daniel S. Katz,
Daina Bouquin,
Neil P. Chue Hong,
Jessica Hausman,
Catherine Jones,
Daniel Chivvis,
Tim Clark,
Mercè Crosas,
Stephan Druskat,
Martin Fenner,
Tom Gillespie,
Alejandra Gonzalez-Beltran,
Morane Gruenpeter,
Ted Habermann,
Robert Haines,
Melissa Harrison,
Edwin Henneken,
Lorraine Hwang,
Matthew B. Jones,
Alastair A. Kelly,
David N. Kennedy,
Katrin Leinweber,
Fernando Rios,
Carly B. Robinson,
Ilian Todorov
, et al. (2 additional authors not shown)
Abstract:
The main output of the FORCE11 Software Citation working group (https://www.force11.org/group/software-citation-working-group) was a paper on software citation principles (https://doi.org/10.7717/peerj-cs.86) published in September 2016. This paper laid out a set of six high-level principles for software citation (importance, credit and attribution, unique identification, persistence, accessibilit…
▽ More
The main output of the FORCE11 Software Citation working group (https://www.force11.org/group/software-citation-working-group) was a paper on software citation principles (https://doi.org/10.7717/peerj-cs.86) published in September 2016. This paper laid out a set of six high-level principles for software citation (importance, credit and attribution, unique identification, persistence, accessibility, and specificity) and discussed how they could be used to implement software citation in the scholarly community. In a series of talks and other activities, we have promoted software citation using these increasingly accepted principles. At the time the initial paper was published, we also provided guidance and examples on how to make software citable, though we now realize there are unresolved problems with that guidance. The purpose of this document is to provide an explanation of current issues impacting scholarly attribution of research software, organize updated implementation guidance, and identify where best practices and solutions are still needed.
△ Less
Submitted 21 May, 2019;
originally announced May 2019.
-
Sustaining Research Software: an SC18 Panel
Authors:
Daniel S. Katz,
Patrick Aerts,
Neil P. Chue Hong,
Anshu Dubey,
Sandra Gesing,
Henry J. Neeman,
David E. Pearah
Abstract:
Many science advances have been possible thanks to the use of research software, which has become essential to advancing virtually every Science, Technology, Engineering and Mathematics (STEM) discipline and many non-STEM disciplines including social sciences and humanities. And while much of it is made available under open source licenses, work is needed to develop, support, and sustain it, as un…
▽ More
Many science advances have been possible thanks to the use of research software, which has become essential to advancing virtually every Science, Technology, Engineering and Mathematics (STEM) discipline and many non-STEM disciplines including social sciences and humanities. And while much of it is made available under open source licenses, work is needed to develop, support, and sustain it, as underlying systems and software as well as user needs evolve.
In addition, the changing landscape of high-performance computing (HPC) platforms, where performance and scaling advances are ever more reliant on software and algorithm improvements as we hit hardware scaling barriers, is causing renewed tension between sustainability of software and its performance. We must do more to highlight the trade-off between performance and sustainability, and to emphasize the need for sustainability given the fact that complex software stacks don't survive without frequent maintenance; made more difficult as a generation of developers of established and heavily-used research software retire. Several HPC forums are doing this, and it has become an active area of funding as well.
In response, the authors organized and ran a panel at the SC18 conference. The objectives of the panel were to highlight the importance of sustainability, to illuminate the tension between pure performance and sustainability, and to steer SC community discussion toward understanding and addressing this issue and this tension. The outcome of the discussions, as presented in this paper, can inform choices of advance compute and data infrastructures to positively impact future research software and future research.
△ Less
Submitted 24 February, 2019;
originally announced February 2019.
-
Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained
Authors:
Daniel S. Katz,
Lois Curfman McInnes,
David E. Bernholdt,
Abigail Cabunoc Mayes,
Neil P. Chue Hong,
Jonah Duckles,
Sandra Gesing,
Michael A. Heroux,
Simon Hettrick,
Rafael C. Jimenez,
Marlon Pierce,
Belinda Weaver,
Nancy Wilkins-Diehr
Abstract:
Software is the key crosscutting technology that enables advances in mathematics, computer science, and domain-specific science and engineering to achieve robust simulations and analysis for science, engineering, and other research fields. However, software itself has not traditionally received focused attention from research communities; rather, software has evolved organically and inconsistently…
▽ More
Software is the key crosscutting technology that enables advances in mathematics, computer science, and domain-specific science and engineering to achieve robust simulations and analysis for science, engineering, and other research fields. However, software itself has not traditionally received focused attention from research communities; rather, software has evolved organically and inconsistently, with its development largely as by-products of other initiatives. Moreover, challenges in scientific software are expanding due to disruptive changes in computer hardware, increasing scale and complexity of data, and demands for more complex simulations involving multiphysics, multiscale modeling and outer-loop analysis. In recent years, community members have established a range of grass-roots organizations and projects to address these growing technical and social challenges in software productivity, quality, reproducibility, and sustainability. This article provides an overview of such groups and discusses opportunities to leverage their synergistic activities while nurturing work toward emerging software ecosystems.
△ Less
Submitted 7 December, 2018; v1 submitted 20 November, 2018;
originally announced November 2018.
-
Software Citation in Theory and Practice
Authors:
Daniel S. Katz,
Neil P. Chue Hong
Abstract:
In most fields, computational models and data analysis have become a significant part of how research is performed, in addition to the more traditional theory and experiment. Mathematics is no exception to this trend. While the system of publication and credit for theory and experiment (journals and books, often monographs) has developed and has become an expected part of the culture, how research…
▽ More
In most fields, computational models and data analysis have become a significant part of how research is performed, in addition to the more traditional theory and experiment. Mathematics is no exception to this trend. While the system of publication and credit for theory and experiment (journals and books, often monographs) has developed and has become an expected part of the culture, how research is shared and how candidates for hiring, promotion are evaluated, software (and data) do not have the same history. A group working as part of the FORCE11 community developed a set of principles for software citation that fit software into the journal citation system, allow software to be published and then cited, and there are now over 50,000 DOIs that have been issued for software. However, some challenges remain, including: promoting the idea of software citation to developers and users; collaborating with publishers to ensure that systems collect and retain required metadata; ensuring that the rest of the scholarly infrastructure, particularly indexing sites, include software; working with communities so that software efforts "count" and understanding how best to cite software that has not been published.
△ Less
Submitted 21 July, 2018;
originally announced July 2018.
-
Top Tips to Make Your Research Irreproducible
Authors:
Neil P. Chue Hong,
Tom Crick,
Ian P. Gent,
Lars Kotthoff,
Kenji Takeda
Abstract:
It is an unfortunate convention of science that research should pretend to be reproducible; our top tips will help you mitigate this fussy conventionality, enabling you to enthusiastically showcase your irreproducible work.
It is an unfortunate convention of science that research should pretend to be reproducible; our top tips will help you mitigate this fussy conventionality, enabling you to enthusiastically showcase your irreproducible work.
△ Less
Submitted 8 April, 2015; v1 submitted 31 March, 2015;
originally announced April 2015.
-
Best Practices for Scientific Computing
Authors:
Greg Wilson,
D. A. Aruliah,
C. Titus Brown,
Neil P. Chue Hong,
Matt Davis,
Richard T. Guy,
Steven H. D. Haddock,
Katy Huff,
Ian M. Mitchell,
Mark Plumbley,
Ben Waugh,
Ethan P. White,
Paul Wilson
Abstract:
Scientists spend an increasing amount of time building and using software. However, most scientists are never taught how to do this efficiently. As a result, many are unaware of tools and practices that would allow them to write more reliable and maintainable code with less effort. We describe a set of best practices for scientific software development that have solid foundations in research and e…
▽ More
Scientists spend an increasing amount of time building and using software. However, most scientists are never taught how to do this efficiently. As a result, many are unaware of tools and practices that would allow them to write more reliable and maintainable code with less effort. We describe a set of best practices for scientific software development that have solid foundations in research and experience, and that improve scientists' productivity and the reliability of their software.
△ Less
Submitted 26 September, 2013; v1 submitted 30 September, 2012;
originally announced October 2012.
