Evaluation Module

  • Version 1

Evaluation of QA systems typically involve having a set of pre-defined questions, example queries, and answers to the questions. The Question Answering over Linked Data (QALD) is an evaluation campaign on multilingual question answering over linked data. QALD-4 provides a set of biomedical questions drawn from DrugBank, SIDER and Diseaseome. We wanted to extend this evaluation to a broader set of questions of increasing complexity, and to consider the data from DBpedia, Bio2RDF and BioGateway. Towards this goal, we

  1. established a set of descriptors to characterize questions [document][spreadsheet]
  2. created questions, queries, and answers over our KB.

At the moment, evaluation module uses NLP-1 dataset for the evaluation of the full module.


Input is a language tag with configuration containing the address of the module and the number of maximum answers.

    "language": "en",
    "conf": {
        "tgm" : [""],
        "dm"  : [""],
        "qgm": [""], 
        "kb"   : ["http://dbpedia.org/sparql"], 
        "answer_num": "5"


Output includes accuracy of the full system applying it to the full NLP-1 dataset.