Living Labs for Academic Search (LiLAS)

Evaluation Lab at CLEF 2021, 21-24 September 2021

LiLAS @ CLEF2021

The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems’ content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.

Schedule for 21 September 2021

All times are for the main conference location Bucharest (GMT+3). For more detailed information regarding registration and the official programme of CLEF 2021 check the official CLEF 2021 website. Don’t forget to register for CLEF 2021!

Time Topic  
15:30 Introduction and Welcome, Overview and Tasks (Philipp Schaer)
15:45 STELLA Infrastructure and Tech Details of LiLAS (Timo Breuer)
16:00 Presentations of Participating Teams
16:30 Results (Timo Breuer)
16:45 Discussion and Outlook (Leyla Jael Garcia)
17:00 End of Session 

Updates

Tasks for CLEF 2021

LiLAS offers two different evaluation tasks:

  1. Ad-hoc retrieval of scientific documents for the multi-lingual and multi-source Life Science search portal LIVIVO.
  2. Research dataset recommendation within the Social Science portal GESIS Search: Given a scientific publication find relevant research data sets (out of a list of candidates).

For both tasks, participants are invited to submit

For type A, participants pre-compute result files following TREC run file syntax and submit them for integration into the live systems. For type B, participants encapsulate their retrieval system into a Docker container following some simple implementation rules inspired by the OSIRRC workshop at SIGIR 2019.

The details of the two tasks are described seperately.

Data Sets

We publish two datasets to allow participants to train and compile their systems for the two platforms LIVIVO and GESIS Search. We offer a list of candidate documents and candidate research data for each query and seed document, respectively, so participants focus on the actual ranking approaches behind the ad-hoc search and recommendation task: https://th-koeln.sciebo.de/s/OBm0NLEwz1RYl9N

The data sets share a common struture:

├── gesis-search
│   ├── candidates
│   ├── datasets
│   ├── documents
├── livivo
│   ├── candidates
│   ├── documents

For both platforms we release the documents/research data sets and a precompiled set of candidate documents. For further data set documentation, please refer to the documentation included in the repository.

Feedback and Evaluation Metrics

For both we use an interleaving between the experimental systems and the baseline productive system. Within GESIS Search we track clicks on the dataset records. Within LIVIVO we track the following click events:

Using these events we compute wins, ties and losses against the production baseline system.

STELLA Evaluation Framework

We use the STELLA framework to include your rankings and recommendations!

Currently, the infrastructure supports two different types of submission. Experimenters can choose to submit pre-computed runs with TREC run file syntax OR use this repository in order to integrate their system as a micro-service into the STELLA App. In contrast to pre-computed results, these dockerized systems can deliver more comprehensive search result since they are not limited to pre-selected queries or items.

Ressources

Dates

For further details, please refer to the CLEF 2021 schedule

Organization

We will have a half-day workshop that is split up in two parts.

LiLAS 2020 Chairs

Follow us

Google Groups | GitHub | lilas@stella-project.org

LiLAS is part of CLEF 2021.