Storage and Reasoning Systems Evaluation Campaign 2011

Storage and Reasoning Systems Evaluation Campaign 2011

Organizers:

• Mikalai Yatskevich, Oxford University, UK (mikalai.yatskevich AT comlab DOT ox DOT ac DOT uk) 
  

------------------------------------------------------------------------------------------------

Click here for the latest evaluation campaign news

Updates to this evaluation campaign will be announced here and through RSS. 

Overview of the Evaluation Campaign

Semantic technologies are at the heart of the future Web providing ways to express knowledge and data so that it can be properly exploited. These technologies will empower a new class of Information and Communication Technologies much more scalable, interoperable, and with a higher degree of process automation support that will fulfil the needs of an emergence market that will exceed $10 billion by 2010.

Description logic based systems (DLBSs) evaluation aims at assessing interoperability, performance and scalability of one of the core building blocks of semantic technologies vision namely description logic reasoners.
Resource description framework reasoning systems (RDF RS) evaluation is targeted at assessing interoperability and performance of RDF reasoners.
Resource description framework storage systems (RDF SS) evaluation assesses efficiency and effectiveness of RDF storage systems.

This evaluation, thus, serves several purposes:
- Helps researches and users to select appropriate technologies;
- Gives an objective comparison among existing technologies;
- Stimulates technologies advancement through continuous evaluation.

Evaluation scenarios

The overall objective of evaluation campaign is to evaluate DLBSs standard inference services: classification, class satisfiability, ontology satisfiability, logical entailment and instance retrieval. The challenge uses a set of state of the art ontologies for evaluation. The set include OWL 2 test cases repository, ontologies from Gardiner suite, various versions of the GALEN ontology, ontologies that have been created in EU funded projects SEMINTEC, VICODI, AEO and SNOMED CT ontology. DLBSs are expected to support OWL 2 language and provide interface to their functionalities through a set of evaluation interfaces defined in the SEALS evaluation platform.

The RDF RS evaluation includes inconsistency checking and entailment checking inference services. Test data includes OWL 2 Conformance suites with about 1000 of specification-level and feature-level tests and artificially generated RDF datasets,

The RDF SS evaluation includes dataset loading and query assessment inference services. The real-world datasets include DBpedia, YAGO, linkedCT, US congress votes and set of SPARQL queries for each of these datasets. Artificially generated datasets include LUBM and Berlin SPARQL benchmark. RDF SS are expected to support SPARQL query language.

How do I get involved?

Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.

Participants are just expected to collaborate in the connection of their tool with the SEALS Platform, which will be the infrastructure that will run all the evaluations automatically. Besides checking their results and comparing with others, once the tool is connected to the SEALS Platform participants will also be able to run the evaluations on their own with these and future test data.

The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.