Open Science in Automotive User Research
Openness and transparency are critical not only for judging the quality of empirical research, but also for accelerating scientific progress and promoting an inclusive scientific community. This initiative reflects our commitment to driving forward the development of open, user-centered automotive technologies and services, ensuring that knowledge and findings are shared openly to benefit the broader community. Therefore, we provide practical guidelines for authors on how to make their research open and transparent and provide a platform for authors to promote the research artifacts they have made available to the broader automotive user research community.
Why Open Science?
The pursuit of Open Science is essential for several reasons:
- Accelerating Knowledge Dissemination: Open Science practices ensure that research findings are shared more broadly and quickly, facilitating faster advancements in the field.
- Enhancing Collaboration: By making research data and methodologies openly available, we foster an environment where researchers can easily collaborate, building upon each other’s work to push the boundaries of knowledge.
- Promoting Transparency and Reproducibility: Open access to research findings and data supports the verification of results, enhancing the credibility and reliability of research within our community.
Toward Open Science
Every step toward greater transparency in research is valuable, contributing incrementally to the overall integrity and reproducibility of scientific work. It’s important to recognize and celebrate each step taken, even if complete openness is not immediately achieved. Here are three levels of engagement with open science practices:
- Level 1:
- Document methods and results: Even if data cannot be shared, detailed documentation of research methods and results can be invaluable. If you can’t share the raw data, maybe you can share aggregated data or the statistics of the data.
- Acknowledge limitations: Clearly state any restrictions on data sharing and transparency, explaining the reasons behind them.
- Level 2:
- Share data when possible: When permissible, share data in a trusted repository, ensuring it is de-identified and secure.
- Use open-access platforms: Publish findings open-access and on FAIR (findable, accessible, interoperable, and reusable) platforms to make the research accessible to a broader audience.
- Level 3:
- Full data and code availability: Share all research data, code, and materials necessary to replicate the study and reproduce the results.
- Engage in pre-registration: Register study protocols in advance to enhance the credibility of the research.
Practical Tips for Authors
Following the guidelines we shared for the AutomotiveUI ‘24 conference, here are practical tips to incorporate Open Science into your research workflow:
Data Sharing and Management
There are many places where we can publish open access artifacts. It is good practice to keep your data on a FAIR platform/repository. Here are some examples (updated June 2024):
- ACM Digital Library (FAIR): AutoUI is an ACM conference, and the platform allows the sharing of (small) supplementary material with the publication.
- OSF (FAIR): Allows to make repos anonymous for review, which can be handy for the submission process. It is a ‘swiss knife’ for publishing artifacts in open access, and the platform allows a great level of flexibility. File versioning is straightforward, and OSF also allows you to pre-register your study. It is also easy to make your material anonymous for the review process.
- Zenodo (FAIR): Data is stored at CERN, which is reliable. It is becoming a common place to share data, which may make it user-friendly. They accept up to 50GB per dataset with an option to have multiple datasets.
- GitHub (not FAIR; see here) Everyone knows how to navigate around a repo on GitHub, which can be an advantage for outreach. But it is not FAIR as no globally unique and persistent identifier is created. What can be practiced is to provide a link to the active project in the manuscript, and then still upload materials to a FAIR platform for reproducibility.
- 4TU.ResearchData (FAIR): Local storage for the technical universities of the Netherlands. They do quite a good job of making the management of datasets easy. Likely, there is a similar portal in your institution/country.
Open Access Publications
- Choose Open Publishers: When publishing your findings, opt for conferences and journals that offer open access, making your research available to all. Avoid Predatory Journals.
- Preprints: Consider posting preprints of your research to platforms such as arXiv, bioRxiv, or OSF. This is a good way of sharing your results either prior to publication or in case you cannot or do not want to pay the open access fees of publishers.
Transparency in Methodology
- Code Sharing: Use FAIR platforms/repositories (see above) to share the code used in your research, including analysis scripts and software developed as part of your work.
- Detailed Documentation: Ensure that your methodologies are thoroughly documented and shared, allowing others to replicate or build upon your work.
Openly Available Research Artifacts
Here, we provide a dynamically updated table of related works, categorized by the criteria datasets, simulation software and models to facilitate easy access to a wealth of resources that can support your research and inspire your Open Science endeavors.
We encourage authors of papers with openly available materials who want to link their work on this site to create a Pull Request on GitHub to add their work. Add your work here: Related Works CSV. The category must be one of [Dataset, Software, Model]. Please make sure your pull requests complies with the format. We will check the entry and approve it as soon as possible.
Dataset
| Title | Author | Year | Paper-Link | Repo-Link |
|---|---|---|---|---|
| AutoTherm: A Dataset and Benchmark for Thermal Comfort Estimation Indoors and in Vehicles | Mark Colley and Sebastian Hartwig and Albin Zeqiri and Timo Ropinski and Enrico Rukzio | 2024 | https://dl.acm.org/doi/abs/10.1145/3678503 | https://huggingface.co/datasets/kopetri/AutoTherm |
Software
| Title | Author | Year | Paper-Link | Repo-Link |
|---|---|---|---|---|
| PedSUMO: Simulacra of Automated Vehicle-Pedestrian Interaction Using SUMO To Study Large-Scale Effects | Mark Colley and Julian Czymmeck and Mustafa Kücükkocak and Pascal Jansen and Enrico Rukzio | 2024 | https://dl.acm.org/doi/10.1145/3610977.3637478 | https://github.com/M-Colley/pedsumo |
| AutoTherm: A Dataset and Benchmark for Thermal Comfort Estimation Indoors and in Vehicles | Mark Colley and Sebastian Hartwig and Albin Zeqiri and Timo Ropinski and Enrico Rukzio | 2024 | https://dl.acm.org/doi/abs/10.1145/3678503 | https://github.com/az16/thermal-comfort-classification |
| Portobello: Extending Driving Simulation from the Lab to the Road | Fanjun Bu and Stacey Li and David Goedicke and Mark Colley and Gyanendra Sharma and Wendy Ju | 2024 | https://dl.acm.org/doi/abs/10.1145/3613904.3642341 | https://github.com/FAR-Lab/Portobello |
| Hey What's Going On?: Conveying Traffic Information to People with Visual Impairments in Highly Automated Vehicles: Introducing OnBoard | Luca-Maxim Meinhardt and Maximilian Rück Julian Zähnle and Maryam Elhaidary and Mark Colley and Michael Rietzler and Enrico Rukzio | 2024 | https://dl.acm.org/doi/abs/10.1145/3659618 | https://github.com/luca-maxim/hey_whats_going_on |
| Introducing VAMPIRE – Using Kinaesthetic Feedback in Virtual Reality for Automated Driving Experiments | Philipp Hock and Mark Colley and Ali Askari and Tobias Wagner and Martin Baumann and Enrico Rukzio | 2022 | https://dl.acm.org/doi/abs/10.1145/3543174.3545252 | https://dl.acm.org/doi/suppl/10.1145/3543174.3545252/suppl_file/source-code.zip |
| Visualizing imperfect situation detection and prediction in automated vehicles: Understanding users' perceptions via user-chosen scenarios | Pascal Jansen and Mark Colley and Tim Pfeifer and Enrico Rukzio | 2024 | https://www.sciencedirect.com/science/article/pii/S1369847824001141 | https://github.com/Pascal-Jansen/VisualizingAutomatedVehicleFunctionalities |
| Coupled simulator for research on the interaction between pedestrians and (automated) vehicles | Pavlo Bazilinskyy and Lars Kooijman and Dimitra Dodou and Joost De Winter | 2020 | https://pure.tudelft.nl/ws/portalfiles/portal/95761905/CoupledsimulatorinUnity_2.pdf | https://github.com/bazilinskyy/coupled-sim |
Model
| Title | Author | Year | Paper-Link | Repo-Link |
|---|---|---|---|---|
| Example Model 1 | Author C | 2023 | https://dl.acm.org/ | https://example.com/model1 |
Paper and Workshop
For more information on the current state of the art and the motivators and barriers to open science in the automotive user research community, see the paper or its materials published on OSF. If you find this information useful, please consider citing it:
@inproceedings{ebel2024changing,
title={Changing Lanes Toward Open Science: Openness and Transparency in Automotive User Research},
author={Ebel, Patrick and Bazilinskyy, Pavlo and Colley, Mark and Goodridge, Courtney and Hock, Philipp and Janssen, Christian P. and Sandhaus, Hauke and Srinivasan, Aravinda Ramakrishnan and Wintersberger, Philipp},
booktitle={16th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '24)},
pages={17},
year={2024},
address={Stanford, California, USA},
month={sep},
publisher={ACM},
location={New York, NY, USA},
url = {https://doi.org/10.1145/3640792.3675730},
doi = {10.1145/3640792.3675730},
series = {AutomotiveUI '24}
}
Also, see the 2023 workshop website for more information.