ELRA - E0022 : EQueR Evaluation Package

Home Catalogue

Language Resources

Bug reports

Send us your bug reports.

Search Catalogue

Languages

Informations

Purchase procedure & Conditions

Pricing & user licences

How to promote your resources ?

Catalog Reference : E0022

EQueR Evaluation Package

The EQueR Evaluation Package was produced within the French national project EQueR (Evaluation campaign for Question-Answering systems), as part of the Technolangue programme funded by the French Ministry of Research and New Technologies (MRNT). The EQueR project enabled to carry out a campaign for the evaluation of Question-Answering systems in French.

This package includes the material that was used for the EQueR evaluation campaign. It includes resources, protocols, scoring tools, results of the campaign, etc., that were used or produced during the campaign. The aim of these evaluation packages is to enable external players to evaluate their own system and compare their results with those obtained during the campaign itself.

The campaign is distributed over two actions:
1) Generic task: it consists in evaluating the performances of question-answering systems on a collection of heterogeneous texts.
2) Specialised task: it consists in evaluating the performances of question-answering systems on a collection of texts from the medical domain.

The EQueR evaluation package contains the following data and tools:
1) Two text collections:
- General corpus: about 1.5 Gb of data consisting of news articles of several years from Le Monde and Le Monde Diplomatique, press releases and information reports from the French Senate dealing with various subjects.
- Medical corpus: about 140 Mb of data mainly consisting of scientific articles and guidelines for good medical practice, selected by the CISMeF (Catalogue et Index des Sites Médicaux Francophones) from the University Hospital Centre of Rouen.
1) Two corpora of questions :
- 500 questions for the generic task and 200 questions for the specialised task.
- For each question in the two corpora, the first 100 identifiers are provided (from Pertimm’s search engine).
2) Two Pertimm’ sub-corpora, created from the document identifiers and returned by the search engine.
3) The whole results provided by the participants.
4) A help software for the evaluation of results within the evaluation of question-answering systems (with detailed documentation).

A description of the project is available at the following address:
http://www.technolangue.net/article.php3?id_article=195 (in French language)

Production

Project : EVALDA

Contents

Click on the arrow to display content.