Introduction

At the heart of the SEALS initiative is the organisation of two public world-wide evaluation campaigns (one in summer 2010 and the second in late 2011 / early 2012). The results of these evaluation campaigns will be employed in creating semantic technology roadmaps which will identify sets of efficient and compatible tools for developing large-scale semantic applications.

Tool developers are cordially invited to participate in the SEALS Evaluation Campaign 2010 in one or more of the five core areas. Participation is open to developers interested in evaluating their tool or to anyone who wants to evaluate a certain tool.

From the tool provider's perspective, the SEALS Evaluation Campaign has been designed to require minimal effort on their part. For the majority of SEALS Evaluation Campaigns, the evaluation process is fully automated from the point at which the tool has been uploaded to the SEALS Platform.

 The SEALS Platform, which will be available late July 2010, will manage all aspects of the evaluation. Once the evaluation has been completed, the results and analyses will be available from the SEALS Portal.

In order to facilitate interaction between your tool and the SEALS Platform, the SEALS teams from each technology area have defined an easy to use API. Details about each technology area's API is available from their SEALS Evaluation Campaign's Portal page listed on the menu to the left.

Organisation

The organisation and execution of the SEALS campaigns is overseen by two types of committee: 

Evaluation Campaign Organizing Committee (E.C.O.C.)

The E.C.O.C. is in charge of the general organization and monitoring of the evaluation campaign across all technology areas. Currently, E.C.O.C. membership is:

Evaluation Campaign Executing Committee (E.C.E.C.)

The five E.C.E.C.s - one per technology area - are in charge of organizing the evaluation scenarios that are performed in the evaluation campaign and of taking them to a successful end. Details of membership can be found on the respective technology area's page (listed on the menu to the left).

If you are interested in being involved in the definition and organisation of future SEALS evaluation campaigns and/or sitting on either the E.C.O.C. or one or more E.C.E.C.s, please get in touch - we are keen to involve members of the community in the SEALS initiative.

Timeline

May 2010 Registration opens
May-June 2010 Evaluation materials and documentation are provided to participants
July 2010 Participants upload their tools
August 2010 Evaluation scenarios are executed
September 2010 Evaluation results are analysed
November 2010 Evaluation results are discussed at ISWC2010 (IWEST workshop)

Workshop

We will hold an evaluation workshop at ISWC 2010 in which we will invite participants to present their systems and evaluation results. The workshop will also provide an opportunity to discuss the direction SEALS ought to take with respect to the second evaluation campaign which is scheduled for 2011/2012.

Find out more about the workshop at its own page, IWEST 2010. 

How do I find out more or get involved?

The easiest way to find out more is to join the SEALS Community. Joining the SEALS Community is easy and poses no obligations. On the other hand, by being a member of the community you will have first-hand information about the semantic technology evaluation campaigns and results. Here you can join the SEALS Community.

To learn more about the future activities of the SEALS community, you can read about the evaluation campaigns for each technology area in their own section of the SEALS portal (see the sub-menu items under Evaluation Campaigns). Each campaign also has its own RSS news feed (see the sub-menu items under News). In addition to this, you can also follow some E.C.E.C.s on twitter - check out their pages for more information.

Download the evaluation campaigns flyer. 

Evaluation campaigns contact: Stuart Wrigley, University of Sheffield   s DOT wrigley AT dcs DOT shef DOT ac DOT uk