1st SEALS Tutorial – Semantic evaluation at large scale

at the 8th Extended Semantic Web Conference ESWC 2011,
Heraklion, Greece.
May 29th, 2011
Duration: Full Day

Abstract

Semantic Web technologies have become a well-established area of computer science research. With the increasing number of technologies being developed, the problem of how to compare and evaluate the various approaches gains more and more importance. Such evaluation is critical not only for future scientific progress, but also for the industrial adoption of the developed technologies. A still open challenge in this area is the lack of standard benchmarks and evaluation resources that can be easily reused by the community.

The objective of the SEALS (Semantic Evaluation at Large Scale) European project is to create a lasting reference infrastructure for semantic technology evaluation (the SEALS Platform) and enable the continuous benchmarking of semantic technologies at a large scale via public evaluation campaigns. In October 2010 the SEALS project completed its first phase where different evaluation services for semantic technologies were implemented in the SEALS Platform and were used to evaluate several tools in the first set of SEALS Evaluation Campaigns.

Automated evaluation of semantic technologies is a challenge in the Semantic Web area. SEALS concentrates on providing an infrastructure for the evaluation of five key areas of semantic technologies: ontology engineering tools, storage and reasoning systems, matching tools, semantic search tools, and semantic web service tools. The SEALS Platform provides an integrated way for evaluating these technologies through a set of evaluation services and test suites. This kind of infrastructure is crucial for researches, developers and industrial adopters of these technologies.

This tutorial aims at presenting the SEALS Platform and is targeted at researchers and practitioners interested in learning how to use this platform for evaluating their semantic technologies.

Previous tutorials at ESWC 2009 and ESWC 2010 presented the state of the art of Semantic Web technology evaluation, giving an overview on the methodological aspects of evaluations and community-driven evaluation initiatives. The SEALS Tutorials aim to jump from theory to practice, showing how the SEALS Platform can support anyone interested in performing their own evaluations or reusing existing ones.

Tutorial Description

Aims and Target Audience

The tutorial presents the SEALS infrastructure for automated evaluation of semantic technologies. The target audience includes researchers, developers and industrial adopters that either use or develop such technology. They will learn how to prepare test data, define and run evaluation experiments and create reports on evaluation results using the services provided by the SEALS Platform.

We also expect to have attendants who have already defined their own evaluations and would like to include them into the SEALS Platform. During the tutorial these people will learn how to insert their evaluations into the platform step by step.

Prerequisite Knowledge and Technical Requirements

A basic understanding for the terms and technologies of the Semantic Web (the Semantic Web stack, RDF, SPARQL, OWL, etc.) will be assumed as well as minimal Java programming expertise. Other than those, no special skills are required for the tutorial.

Tutorial Content and Schedule

The tutorial is divided in half day focused on people who want to use the SEALS Platform to evaluate semantic tools using existing evaluations and another half day focused on people who want to use the SEALS Platform to define their own evaluations.

Morning. Tutorial for users

1. (30 min) Introduction to the SEALS Platform. The session will present the SEALS Platform and its evaluation services, which will be used along the tutorial, and will give an overview of the first set of SEALS Evaluation Campaigns and of the evaluation scenarios that were executed in each of them.

2. (30 min) Connecting tools to the SEALS Platform. In order to evaluate a tool using the SEALS Platform, it has to be previously connected to it. This session will include a step-by-step guide that will allow attendants to connect their tools (or any other tool they want to evaluate) to the platform so they can run evaluations over them.

The rest of the sessions in this part of the tutorial will include step-by-step guides on how to evaluate different types of semantic tools and how to analyse the obtained results and compare them with those of other tools.

3. (30 min) Evaluating ontology engineering tools.

4. (30 min) Evaluating ontology reasoners.

5. (30 min) Evaluating ontology matching tools.

6. (30 min) Evaluating semantic search tools.

7. (30 min) Evaluating semantic web service tools.

Afternoon. Tutorial for evaluators

8. (60 min) SEALS repositories management. This session will cover the definition of test data, test data generators, evaluation results, and interpretations, the insertion of these entities into the SEALS Platform repositories, and their later access.

10. (60 min) Evaluation description definition. This session will deal with the definition and implementation of evaluation descriptions, that is, of the workflows that will automatically execute the evaluations over the tools. During this learn how to implement their evaluation descriptions in the SEDL language (a BPEL extension).

11. (60 min) Evaluation description insertion. During this session, attendants will learn how to insert evaluation descriptions into the SEALS Platform and how to run them.

Presenters

Raúl García-Castro (primary contact) ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is an Interim Associate Professor at the Computer Science School at the Universidad Politécnica de Madrid (UPM), where he obtained his Ph.D. titled "Benchmarking Semantic Web technology". His research focuses on the evaluation and benchmarking of Semantic Web technologies. He led three activities for benchmarking the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange language. He co-organized the ISWC2007 and ESWC2008 workshops on Evaluation of Ontology Tools (EON2007, EON-SWSC2008) and co-organized the ESWC2009 and ESWC2010 tutorials on Evaluation of Semantic Technologies. He currently teaches at UPM and has given several tutorials on the topics of ontologies and the Semantic Web.

Stuart N. Wrigley ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is a post-doctoral research scientist at the University of Sheffield, UK. He received his PhD in 2002 and has been involved in a number of large EU-funded projects focussed on the provision of technologies for assisting multiparty interactions such as meetings. Most recently he acted as work package manager for the EU Framework 6 project Augmented Multi-party Interaction with Distance Access (AMIDA) with responsibility for all audio and video processing. He is currently work package manager for a number of work packages within the EU Framework 7 project Semantic Evaluation at Large Scale (SEALS). Within the context of SEALS, he led the 2010 semantic search evaluation campaign. Dr Wrigley has taught undergraduate and postgraduate courses and supervised students within the Department of Computer Science, University of Sheffield; he has also been a lecturer and tutor at a number of European summer schools.

Miguel Esteban( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is a member of the Ontology Engineering Group at Universidad Politécnica de Madrid (UPM). His research activities are focused in Ontological Engineering issues, especially in ontology usage (access and management) in Grid environments, and ontology evolution in distributed and heterogeneous environments. He is leading the development of WS-DAIOnt-RDF(S), a specification for RDF(S) model-based access in the Open Grid Services Architecture (OGSA). This work is the basis of the DAIS RDF(S) Access activity that is taking place in the DAIS WG of the OGF, in which he is also participating, taking the leadership in the development of the model-based part of the specification. Nowadays, in the context of the SEALS project is co-leading the development of the SEALS Platform, an infrastructure for the evaluation of semantic technologies.

Mikalai Yatskevich ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) joined University of Trento Data & Knowledge Management research group as Research Assistant In March of 2003. In November of 2003 he joined Information and Telecommunication Technologies International Doctoral School in Trento where he completed his doctoral research on knowledge management systems and techniques for their large-scale evaluation. In March 2008 he defended his PhD dissertation. After short-term postdoctoral assignments at University of Trento and University of Rennes I he joined the Computing Laboratory at the University of Oxford as a Postdoctoral Research Assistant. He is a member of the Information Systems group led by Prof. Georg Gottlob and Prof. Ian Horrocks.

Liliana Cabral ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is a Research Fellow at the Open University, UK. She obtained her PhD from this university in the topic of Semantic Web Services. She has developed technologies and co-organized a number of tutorials on Semantic Web Services in previous ESWC and ISWC conferences. She has also been a member of several projects from the European Commission concerning Semantic Web Services, including DIP and SUPER. She is currently the chair of the SWS Challenge initiative and has co-organized its workshop in 2009. She is also a member of the SEALS (Semantic Evaluation at Large Scale) project (EC FP7).

Jürgen Bock ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is working as a research assistant at FZI Research Center for Information Technologies in Karlsruhe, Germany, since March 2007. He is working in the projects SEALS and THESEUS, where his research work focuses on the area of ontology management, in particular ontology mapping and reasoning with an emphasis on scalability. He graduated from the University of Ulm, Germany in March 2007 as "Diplom Informatiker" and in December 2006 from the Griffith University in Brisbane, Australia as Bachelor of Information Technology with Honours. During his studies he focused on Knowledge Representation and Dialogue Systems. Besides his studies he completed internships in the area of database programming, documentation in the EDPC area, and dialogue systems.

Christian Meilicke ( This e-mail address is being protected from spambots. You need JavaScript enabled to view it ) is a research assistant at the Chair of Artificial Intelligence at the University of Mannheim, Germany. He graduated from the University of Mannheim as a Master in Philosophy and Educational Science in 2003. After moving from the field of humanities to in the field of computer science, he is working now for his PhD on the topic of reasoning-based debugging of ontology alignments.  For several years he is organizing together with Heiner Stuckenschmidt the anatomy track of the Ontology Alignment Evaluation Initiative (OAEI). As a member of the SEALS project (Semantic Evaluation at Large Scale), he made first experiences in using new evaluation technologies to improve current state of the art evaluation techniques at the OAEI.

Registration

Register at the ESWC 2011 conference website (mark the box for Tutorial 6 on the registration form)