Ontology Matching Tools Evaluation Campaign 2011.5

 Organizer:

• Jérôme Euzenat (INRIA)
• Heiner Stuckenschmidt (UMA)
• Christian Meilicke (UMA)
• Cássia Trojahn (INRIA)

The SEALS evaluation campaign, built on the previous matching evaluation initiatives (Ontology Alignment Evaluation Initiative - OAEI), aims at evaluating the competence of matching systems with respect to different evaluation criteria. The evaluation will focus on demonstrating the feasibility and benefits of automating matching evaluation. Hereby, we focus on the following criteria:

• Conformance against a reference alignment (precision, recall, f-measure);

• Efficiency in terms of runtime (elapsed runtime).

This evaluation will be fully running into the SEALS platform.

Evaluation scenarios and datasets

The evaluation campaign contains at least three five scenarios, where the tools will be evaluated according to common datasets and criteria.

• Scenario 1: Benchmark
  The goal of this benchmark series is to identify the areas in which each matching algorithm is strong or weak. The test is based on one particular ontology dedicated to the very narrow domain of bibliography and a number of alternative ontologies on the same domain for which alignments are provided.

  • Criteria: conformance with expected results.

  • URL in SEALS test repository: http://seals.sti2.at/tdrs-web/testdata/persistent/Benchmark+Testsuite/2010full/suite

• Scenario 2: Anatomy
  The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy. Both ontologies are relatively large biomedical ontologies described in a typical medical language.
  

  • Criteria: conformance with expected results, efficiency in terms of execution time.

  • URL in SEALS test repository: http://seals.sti2.at/tdrs-web/testdata/persistent/Anatomy+Testsuite/2010/suite

• Scenario 3: Conference
  The conference dataset consists of a collection of conference organization ontologies. This effort was expected to materialize in alignments as well as in interesting individual correspondences ('nuggets'), aggregated statistical observations and/or implicit design patterns.

  • Criteria: conformance with expected results and alignment coherence.

  • URL in SEALS test repository: http://seals.sti2.at/tdrs-web/testdata/persistent/Conference+Testsuite/2010/suite
    

• Scenario 4: Multilingual Conference
  This data set consists of a set of a subset of conference ontologies translated into eight languages: Spanish, Germany, French, Russian, Portuguese, Czech, Dutch and Chinese.

• Scenario 5: Scalability tests (to be announced).

Timeline and Tutorial

More information can be found on the relevant OAEI pages.

• OAEI 2011.5 Webpage - Informs about all tracks of OAEI 2011.5 and contains a timeline binding also for the tracks supported by SEALS. This page links also to track specific information.
  
• SEALS platform evaluation modalities for OAEI 2011 - Contains specific instructions for participants, in particular, a tutorial for wrapping a matching tool and software that supports the process is offered for download. The concrete steps for participating are described in detail.

How do I get involved?

Participation is open to all developers interested in evaluating their tool. The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.