SEALS Yardsticks for Ontology Management

Click here for the latest evaluation campaign news

The evaluation campaign is open for participation! 

Overview of the Evaluation Campaign

When dealing with the many existing semantic technologies, some questions arise sooner or later:

• Tool A is able of managing OWL DL ontologies but, up to what extent can it manage OWL Full ontologies?
• I am using an OWL Full ontology in Tool B and I want to use it in Tool C, which only supports OWL Lite. Can I make it with a minimal loss of information?
• Someone recommended me Tool D, but I need to manage very big ontologies. Can this tool make it efficiently? If not, which one can?

The SEALS Yardsticks For Ontology Management is an evaluation campaign that comprises a set of evaluations defined with the goal of evaluating the ontology management capabilities of ontology engineering tools (both ontology editors and ontology management frameworks and APIs) in order to answer those questions.

Evaluation scenarios

The evaluation campaign contains three different scenarios that cover the evaluation of three characteristics of these tools: conformance, interoperability and scalability.

• OET Conformance 2010. We will evaluate tool conformance with regards to different ontology languages with the goal of analysing up to what extent the different ontology constructors are supported by tools.
• OET Interoperability 2010. We will evaluate the interoperability of tools when interchanging ontologies using an interchange language with the goal of knowing the effects of these ontology interchanges.
• OET Scalability 2010. We will evaluate the scalability of tools when managing ontologies of increasing size with the goal of checking up to what extent tools are able of dealing with big ontologies while maintaining their efficiency. 

Evaluation datasets

To evaluate the conformance and interoperability of ontology engineering tools with regards to a certain ontology language, we will use four test suites to ensure enough coverage of the RDF(S) and OWL 1 specifications:

• RDF(S) Import Test Suite
• OWL Lite Import Test Suite 
• OWL DL Import Test Suite
• OWL Full Import Test Suite

Details about the datasets used in the scalability evaluation can be found here.

How do I get involved?

Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.

The only requirement for performing the evaluation on a tool is that the tool is able of programmatically importing and exporting ontologies in the ontology language.

Participants are just expected to collaborate in the connection of their tool with the SEALS Platform, which will be the infrastructure that will run all the evaluations automatically. Besides checking their results and comparing with others, once the tool is connected to the SEALS Platform participants will also be able to run the evaluations on their own with these and future test data.

The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.

Evaluation materials

A detailed description of the evaluation scenarios and test data can be found in SEALS deliverable D10.1. 

Timeline and events

The evaluation campaign will take place during 2010. The concrete timeline of the whole evaluation campaign is the following:

• June 2010: Registration opens
• Mid-June 2010: Evaluation materials and documentation are provided to participants
• July 2010: Participants upload their tools
• August 2010: Evaluation scenarios are executed
• September 2010: Evaluation results are analysed
• November 2010: Evaluation results are discussed in a workshop

Contact

• Raúl García Castro, Universidad Politécnica de Madrid, Spain (rgarcia AT fi DOT upm DOT es)
• Stephan Grimm, FZI Karlsruhe, Germany (grimm AT fzi DOT de)