|
Semantic Web Service Tools Evaluation Campaign 2011
SEALS Semantic Web Service Discovery Evaluation
Organizers:
· Liliana Cabral, KMI, The Open University, UK
· Ning Li, KMI, The Open University, UK
Evaluation Scenario Overview
The SWS discovery activity consists of finding Web Services based on their semantic descriptions. Tools for SWS discovery or matchmaking can be evaluated on retrieval performance, where for a given goal, i.e. a semantic description of a service request, and a given set of service descriptions, i.e. semantic descriptions of service offers, the tool returns the match degree between the goal and each service, and the platform measures the rate of matching correctness based on a number of metrics.
In the SEALS Semantic Web Service Discovery evaluation scenario we will test and compare SWS discovery tools (matchmakers) on retrieval performance using the SEALS platform. SEALS services for applying standard IR measures such as Precision and Recall over registered datasets will be available. Retrieval results will be generated and stored in the Results Repository.
Evaluation datasets
To evaluate the retrieval performance of SWS discovery tools/matchmakers, we will use:
Evaluation materials
If you are a tool provider and would like to evaluate your tool, the instructions will be provided here soon.
How do I get involved?
Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.Participants are just expected to collaborate in the connection of their tool with the SEALS Platform, which will be the infrastructure that will run all the evaluations automatically. Besides checking their results and comparing with others, once the tool is connected to the SEALS Platform participants will also be able to run the evaluations on their own with these and future test data.The first step is to join the SEALS Community. Once you have your community login, you will be able to register your tools for the evaluation campaign. Users with tools registered to the evaluation campaign will receive notifications as further details and services of the SEALS platform become available.
Timeline and events
The evaluation campaign will be launched at ESWC 2011. The concrete timeline of the whole evaluation campaign will be available soon.
June 2011 Campaign is launched
TBA Evaluation materials and documentation are provided to participants
TBA Participants upload their tools
TBA Evaluation scenario is executed and results are analysed
TBA Evaluation results are discussed at workshop
|
SEALS evaluation campaigns 
