Ontology Matching Tools Evaluation Campaign 2010

• Jérôme Euzenat (INRIA)
• Heiner Stuckenschmidt (UMA)
• Christian Meilicke (UMA)
• Cássia Trojahn (INRIA)

------------------------------------------------------------------------------------------------

Click here for the latest evaluation campaign news

Overview of the Evaluation Campaign

The first SEALS evaluation campaign, built on the previous matching evaluation intitiatives (Ontology Alignment Evaluation Initiative - OAEI), aims at evaluating the competence of matching systems with respect to different evaluation criteria. The evaluation will focus on demonstrating the feasibility and benefits of automating matching evaluation.

In this first evaluation a limited set of criteria will be considered:

• Conformance: standard precision and recall, restricted semantic precision and recall, coherence;

• Efficiency: runtime, memory consumption.

Notice that efficiency metrics are for the first time part of the evaluation compared to previous OAEI evaluations.

Evaluation scenarios

The evaluation campaign contains 3 different scenarios, where the tools will be evaluated according to common datasets and criteria:

• Scenario 1:

  • Test data: Benchmark

  • Criteria: conformance with expected results.

• Scenario 2:

  • Test data: Anatomy

  • Criteria: conformance with expected results, efficiency in terms of memory consumption and execution time.

• Scenario 3:

  • Test data: Conference

  • Criteria: conformance with expected results and alignment coherence.

Evaluation datasets

We have selected a subset of the OAEI datasets. The datasets were selected based on the existence of reliable reference alignments and experiences with using the datasets in evaluation campaigns:

• systematic benchmark: the goal of this benchmark series is to identify the areas in which each matching algorithm is strong or weak. The test is based on one particular ontology dedicated to the very narrow domain of bibliography and a number of alternative ontologies on the same domain for which alignments are provided.

• conference: collection of conference organization ontologies. This effort was expected to materialize in alignments as well as in interesting individual correspondences ('nuggets'), aggregated statistical observations and/or implicit design patterns.

• anatomy: the anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.

Evaluation materials

Each matching tool should at least be able to match two input ontologies with any kind of fixed parameters and any kind of fixed and general purpose resources.

These resources can be part of the installation files of the tool or will, after consultation with the SEALS support team, be installed on the SEALS platform. It is not required to process any additional input alignment, nor is any kind of user input or training examples provided. Each tool should generate an output format that conforms to the RDF Alignment format used in previous OAEI campaigns.

To enable tools to be processed within the SEALS platform, each tool must implement a minimal API. This minimal API consists in a Web Service interface.This interface can and should be tested prior to the evaluation campaign via a simple web-form. The web-based evaluation service will also be helpful during developing/testing/verifying the functionality of a matching system.

How do I get involved?

Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.

Once the tool is connected to the SEALS Platform participants will also be able to run the evaluations on their own with these and future test data.

The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.

Timeline and events

The evaluation campaign will take place during 2010. The concrete timeline of the whole evaluation campaign is the following: