The SEALS campaign has been conducted in 2010 in collaboration with the Ontology Alignment Evaluation Initiative (OAEI). The OAEI has been the basis for evaluation over the last five years. It is an annual evaluation campaign that offers several data sets and corresponding evaluations (called "tracks") organized by different groups of researchers. SEALS has supported the OAEI by providing software components for automizing the most important aspects of this years campaign. In particular, a web service interface wrapping the functionality of a matching tool has been proposed. Tool developers, thus, had to wrap their service in the proposed way. As a result, an evaluation service developed by SEALS could be used for both preliminary tests and the registration of the final and official OAEI results.

We were not sure that switching to an automated evaluation would preserve the success of OAEI, given that the effort of implementing a web service interface was required from participants. For that reason members of the SEALS project stayed in contact with most of the participants during the process of implementing and testing the interface. Contrary to our assumptions, the number of participants is similar to the last years of OAEI and the feedback from participants has been positive.

An overview about participation can be found at http://oaei.ontologymatching.org/2010/results/. The tracks conducted in the SEALS modality have been the anatomy, benchmark, and conference track. The precision-recall-graph depicted here as an example shows an analysis conducted as part of evaluating the benchmark results. The graph cut the results given by the participants under a threshold necessary for achieving n% recall and compute the corresponding precision. Systems for which these graphs are not meaningful (because they did not provide graded confidence values) are drawn in dashed lines. Track specific, detailed results and interpretations can be found on the respective webpage.

We thank all the participants who used the SEALS service for participating at the OAEI. Cheers go out to AgreementMaker, AROMA, ASMOV, BLOOMS, CODI, Eff2Mat, Falcon-AO, GeromeSMB, MapPSO, NBJLM, RiMOM, SOBOM & TaxoMap. We would also like to thank all the people visiting our poster at the ISWC-2010 Poster & Demo session for fruitful discussions regarding further steps in the development of an evaluation infrastructure.

We will be happy to see your tool next year running on the SEALS platform!