At FMCAD 2025 we introduce a standardized artifact evaluation as an initiative to encourage authors to share supplementary evidence that enhances the transparency, usability, and reproducibility of their work. This process is designed to be flexible, minimally intrusive, and supportive of authors in their efforts to publish high-quality artifacts.
Why Submit an Artifact?
An artifact is any additional material (software, data sets, …) that offers a way to support and extend the claims of a paper, providing transparency and usability for future research. A well-prepared artifact demonstrates the robustness of the research. While submission is not mandatory, presenting a strong artifact can positively influence how reviewers perceive the clarity, reproducibility, and impact of the paper. Even artifacts that are not fully polished may offer valuable insights and could still be beneficial compared to not submitting any artifact at all.
Reviewing Process
Artifacts will be reviewed alongside the main paper by members of the PC, leveraging the reviewers’ expertise. Each paper will have at least one selected reviewer, who is responsible for evaluating the artifact.
The timeline of artifact reviewing will align with the paper review process timeline, enabling authors to address artifact-related questions during the rebuttal phase.
Accepted artifacts will be visibly labeled on the title page of the published paper.
Submission
The artifact has to be submitted together with the paper in EasyChair. Artifact submissions are single-blind. The deadline for submitting the artifact is the same as for paper submission.
An artifact submission in EasyChair consists of:
- a link from which your artifact package (.zip file) can be downloaded. At the point of submission the .zip file does not yet require a DOI. It can also be provided at an institutional web page. However, a DOI for the artifact has to be provided together with the camera-ready version of accepted papers.
- a description of the system on which the artifact was tested
- a description of any special requirements beyond a Docker or virtual machine image
- the SHA256 checksum of the .zip file to ensure the integrity of the artifact
You can generate the checksum using the following command-line tools.
Linux: sha256sum
Windows: CertUtil -hashfile SHA256
MacOS: shasum -a 256
Packaging Guidelines
While transparency is key, authors retain flexibility in how artifacts are published, without rigid guidelines on formats or platforms. It is up to the authors to choose the most appropriate way to publish their artifact, e.g., using a Virtual Machine, Docker image, or even plain source code.
However, the packed artifact must contain the following elements:
- A LICENSE file describing the rights. The license must at least allow the PC members to download, use, and execute the artifact. Please refer to typical open-source licenses. For quick help about possible licenses, visit www.choosealicense.com.
- A README that introduces the artifact to the user. It should describe and provide
- the structure and content of the artifact
- the resource requirements (RAM, number of cores) needed to execute the artifact
- detailed steps to set up the artifact. Do not assume any prior knowledge of the user, e.g., do not assume that reviewers are familiar with using Docker.
- example inputs that can be used to test the artifact together with a description of the expected output.
Ideally, an artifact also contains log-files and benchmarks that verify the results presented in the paper, without requiring full experiment reruns. Add a description how the tables, graphs, etc in the paper can be generated from the log-files.
Authors are of course welcome to prepare easy-to-use scripts that allow to reproduce the experimental results of the paper. In this case, please provide a time estimate for the task.
For any questions regarding the artifact evaluation process, please contact the PC Chairs.