|
Semantic Web Service Tools Evaluation Campaign 2010
Updates to this evaluation campaign will be announced here and per RSS. Events
The SWS discovery activity consists of finding Web Services based on their semantic descriptions. Tools for SWS discovery/matchmaking use formal semantics of both requests and offers, written in the same description language, in order to retrieve appropriate services. Evaluation of these tools and corresponding retrieval algorithms will be the focus of the SWS track of this SEALS Campaign. Our goal is to bring together different SWS evaluation initiatives such as the SWS Challenge and S3 along with the SEALS SWS evaluation, focusing on the discovery activity so that we can share results and lessons learned. Presentation of results for each event will be held by the respective organisers. Attendance of participants are encouraged but not required. For more information about the events please see: Evaluation results for the three events will be discussed at ISWC2010 (IWEST workshop) in a parallel session on November 7th, 2010. SEALS Semantic Web Service Discovery Evaluation Organizers: · Liliana Cabral, KMI, The Open University, UK· Ioan Toma, University of Innsbruck, Austria· Adrian Marte, University of Innsbruck, Austria
Evaluation Scenario OverviewIn the SEALS Semantic Web Service Discovery evaluation scenario we will test and compare SWS discovery tools/matchmakers on retrieval performance using the SEALS platform. SEALS services for applying standard IR measures such as Precision and Recall over registered datasets will be available. Retrieval results will be generated and stored for each registered tool. In addition, we will compare different sets of tools according to datasets or language via pre-defined evaluation descriptions (workflows). Evaluation datasetsTo evaluate the retrieval performance of SWS discovery tools/matchmakers, we will use: - OWL-S TC
- SAWSDL TC
- WSML TC
- WSMO-LITE TC
Evaluation materials In the future you will find here the documentation and materials needed to connect your tool to the SEALS Platform and include it in the evaluation campaign. The expected way of connecting a tool to the platform is by implementing the SWS plugin API. How do I get involved?Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.Participants are just expected to collaborate in the connection of their tool with the SEALS Platform, which will be the infrastructure that will run all the evaluations automatically. Besides checking their results and comparing with others, once the tool is connected to the SEALS Platform participants will also be able to run the evaluations on their own with these and future test data.The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available. Timeline and eventsThe evaluation campaign will take place during 2010. The concrete timeline of the whole evaluation campaign is the following: May 2010 Registration opens
August 2010 Evaluation materials and documentation are provided to participants
September 2010 Participants upload their tools
September2010 Evaluation scenarios are executed
October2010 Evaluation results are analysed
November 2010 Evaluation results are discussed at ISWC2010 workshop (tbc)
|
SEALS evaluation campaigns 
