Software articles should describe a tool likely to be of broad utility that represents a significant advance over previously published software (usually demonstrated by direct comparison with available related software).
Software articles are intended for software developers and advanced analysts who have developed innovative or useful ways of implementing existing methods. Manuscripts should generally be around 5,000 words and describe implementations of models and methods to analyze large-scale assessment data using software tools. It is important that manuscripts include the necessary background, theory, and relevant concepts to understand the implementation or tool and the importance of the submission. Workflows should be well-documented, ideally with a completely worked example (see the Applications section below). All submissions should include examples that are fully reproducible, including data and source code. Alternatively, for relevant tools, detailed supporting documentation of performance (e.g. comprehensive test files and results) should be provided or made available on a free, open-source repository.
Review criteria should include applicability to large-scale assessment, scientific merit, a fully described background, evidence of feasibility and performance, and full reproducibility.
The Introduction section should explain the relevant context and the specific issue that the software described is intended to address, what the software does to solve it, what competing software already exists, and how this tool is different.
This section should present underlying theories or necessary background to understand the problem in sufficient detail to appreciate what the tool offers.
About the tool
This should include a short explanation of the basic features of the tool (e.g., a list of functions in an R package, a short description of the steps to use the package, etc), and a description of the overall architecture of the software implementation, along with details of any critical issues and how they were addressed.
This section contains a basic overview and demonstration of some of the tool’s key features. This would include input (data, .in files, other relevant inputs), a “prototypical” worked example that illustrates how to use the tool, and the results.
The user interface should be described and a discussion of the intended uses of the software, and the benefits that are envisioned, should be included, together with data on how its performance and functionality compare with, and improve, on functionally similar existing software. A case study of the use of the software may be presented. The planned future development of new features, if any, should be mentioned.
This should state clearly the main conclusions and provide an explanation of the importance and relevance of the software reported.