In this workshop lab, we would like to bring together IR researchers interested in the online evaluation of academic search systems. The goal is to foster knowledge on improving the search for academic resources like literature (ranging from short bibliographic records to full-text papers), research data, and the interlinking between these resources. The employed online evaluation approach in this workshop allows the direct connection to existing academic search systems from the Life Sciences and the Social Sciences.
The motivation behind this lab is to
- bring together interested researchers
- advertise the online evaluation campaign idea, and
- develop ideas, best practices, and guidelines for a full online evaluation campaign at CLEF 2021.
We see academic search as a broader term for scientific and especially domain-specific retrieval tasks which comprises Document as well as Dataset Retrieval. As huge platforms like Google Scholar (or Google Dataset Search) are not open to public research and do not offer any domain-specific features, we focus on mid-size scientific search systems that offer domain-specific resources and use cases. This focus allows for using many specific information types like bibliographic metadata, usage data, download rates, citations, etc., in order to develop and evaluate innovative search applications.
We would like to move beyond the traditional offline evaluation setup and bring together evaluation techniques from industry and practice into the academic realm. Therefore, utilizing online evaluations, as take the actual user into account, would be a step forward towards improving the evaluation situation. The details of the online experiments and the metrics are to be discussed at the workshop, but right now we favor a setup which is based on a Docker container infrastructure that is briefly described in Breuer et al. (2019) and would incorporate usage feedback like click-through rates.
As this is a workshop lab we don’t have any explicit tasks like regular evaluation labs at CLEF.
Nevertheless we would like to release some sample data sets from the scientific search systems LIVIVO and GESIS-wide Search and some Docker templates to allow early adopters to implement first prototypes for an online evaluation. At the workshop we would like to have these early adopters who took part in this open beta phase to present their first-hand experiences to lay a foundation for 2021.
We plan to have a half-day workshop that is split up in two parts.
The first part will consist of an overview presentation by the organizers, invited talks (e.g. from the organizers of the former CLEF labs LL4IR or NewsREEL to present lessons learned from previous campaigns) and some lightning talks from early adopters who implemented first prototypes based on the data and APIs made available or that describe an interesting use case or idea that could lead into a task in 2021. We invite participants of the workshop to submit short technical or position papers that would lead into the lightning talks.
The second half will consist of break-out sessions which should be inspired by the talks of the first half and might focus on topics like data sets, technical setups, evaluation metrics, interfaces or task design. The break-out groups will organize themselves using collaborative text editors like Google Docs to allow a later documentation of the discussion and idea tracking.
The workshop will end with some short presentations of the results of the break-out sessions.
LiLAS 2020 Chairs
- Philipp Schaer, TH Köln, Germany
- Johann Schaible, GESIS, Germany
- Leyla Jael Garcia-Castro, ZB MED, Germany
LiLAS ist part of CLEF 2020 held in Thessaloniki.