Click here for the latest evaluation campaign news

The OAEI 2011 evaluation campaign is over! The preliminary results can be found at OAEI web page. Tool developers that could not participate in this campaign will have the opportunity to evaluate their tools in the 2011.5 campaign.

Ontology Matching Tools Evaluation Campaign 2011

 Organizer:

• Jérôme Euzenat (INRIA)
• Heiner Stuckenschmidt (UMA)
• Christian Meilicke (UMA)
• Cássia Trojahn (INRIA)

Important modifications for 2011: SEALS is supporting the OAEI organizers in running several OAEI 2011 tracks on the SEALS platform. This year, participants are required to upload their tools into the SEALS platform instead of implementing web service interfaces. To that extend, participants must provide their tools in a ZIP file format. A tutorial is provided (details can be found below) in order to explain how to prepare, upload, and package the tools in the required format. The web based evaluation service used in the 2010 campaign, now integrated into the SEALS portal (https://www.seals-project.eu/ontology-matching-evaluation-ui), is still available for testing purposes but it will be not be used in the campaign to generate the official results.

Overview of the Evaluation Campaign

The SEALS evaluation campaign, built on the previous matching evaluation initiatives (Ontology Alignment Evaluation Initiative - OAEI), aims at evaluating the competence of matching systems with respect to different evaluation criteria. The evaluation will focus on demonstrating the feasibility and benefits of automating matching evaluation. Hereby, we focus on the following criteria:

• Conformance against a reference alignment (precision, recall, f-measure);

• Efficiency in terms of runtime (elapsed runtime)
  

Notice that efficiency metrics are for the first time part of the evaluation compared to previous OAEI evaluations. Measuring efficiency was already planned for 2010, however, we finally had to postpone this issue to 2011. Running tools on the platform in 2011 allows us now to include runtime efficiency criteria (and also to take care of the reproducibility of results in a controlled environment).

Evaluation scenarios and datasets

The evaluation campaign contains at least three different scenarios, where the tools will be evaluated according to common datasets and criteria. Note also that, due to supporting OAEI 2011, the timeline of the ontology matching campaign differs from the timeline of the other SEALS campaigns.

• Scenario 1: Benchmark
  The goal of this benchmark series is to identify the areas in which each matching algorithm is strong or weak. The test is based on one particular ontology dedicated to the very narrow domain of bibliography and a number of alternative ontologies on the same domain for which alignments are provided.

  • Criteria: conformance with expected results.

  • URL in SEALS test repository: http://seals.sti2.at/tdrs-web/testdata/persistent/Benchmark+Testsuite/2010full/suite

• Scenario 2: Anatomy
  The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy. Both ontologies are relatively large biomedical ontologies described in a typical medical language.
  

  • Criteria: conformance with expected results, efficiency in terms of execution time.

  • URL in SEALS test repository: http://seals.sti2.at/tdrs-web/testdata/persistent/Anatomy+Testsuite/2010/suite

• Scenario 3: Conference
  The conference dataset consists of a collection of conference organization ontologies. This effort was expected to materialize in alignments as well as in interesting individual correspondences ('nuggets'), aggregated statistical observations and/or implicit design patterns.

  • Criteria: conformance with expected results and alignment coherence.

  • URL in SEALS testrepository: http://seals.sti2.at/tdrs-web/testdata/persistent/Conference+Testsuite/2010/suite

We are currently investigating the possibility of adding more scenarios to be executed in the SEALS modality. As soon as we know more, we will forward the information via a SEAL news article.

Timeline and Tutorial

More information can be found on the relevant OAEI pages.

• OAEI 2011 Webpage - Informs about all tracks of OAEI 2011 and contains a timeline binding also for the tracks supported by SEALS. This page links also to track specific information.
  
• SEALS platform evaluation modalities for OAEI 2011 - Contains specific instructions for participants of tracks in SEALS modality. In particular, a tutorial for wrapping a matching tool and software that supports the process is offered for download. The concrete steps for participating are described in detail.

How do I get involved?

Participation is open to all developers interested in evaluating their tool. The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.