|
Thursday, 27 October 2011 16:38 |
|
As in the last years the Ontology Matching (OM) workshop as ISWC has been a big success with more than 50 listeners and around 20 posters. The workshop day started in the morning with the presentation of technical papers. Amongst other interesting papers, Jerome Euzenat presented the paper "Ontology matching benchmarks: generation and evaluation". It explains a new way of generating benchmarks for ontology matching. In addition to the results discussed in the paper, some new results were included in the presentation that have been generated over the last weeks with help of SEALS technology.
In the afternoon, results of the Ontology Alignment Evaluation Initiative were presented. Cassia Trojahn started with an overview on this years procedure, showed preliminary results of OAEI 2011 (assisted by Christian Meilicke and Ondrej Svab-Zamazal) and talked about plans for the future, using the SEALS platform to its full extent. After this talk of the organizers, three tool developers had been chosen to present the architecture of their systems and talked about implemented matching techniques.
The workshop ended with a discussion and wrap-up that touched several interesting points, such as new datasets that might be used for further evaluations. Ernesto Jiminez Ruiz presented some ideas related to the use of large biomedical ontologies in a poster. A group of people from http://www.dmo-foundry.org/ raised the idea to match several ontologies from the domain of Data Mining. We will stay in contact with both groups to support them in increasing the quality of further OAEI campaigns. ... and note also that there will be an OAEI 2011.5 at ESWC 2012.
Final deadline for tool updates or new tool submissions will probably be
March 2012, more details will follow! ![Precision/recall triangular graph for benchmark2 (best five matchers, triangulars), anatomy (best four matchers, circles) and conference (best eight matchers, squares). Horizontal line depicts level of precision/recall while values of F-measure are depicted by areas bordered by corresponding lines for F-measure=0.[5|6|7].](/images/stories/triangle2.png "Precision/recall triangular for three SEALS supported tracks") Precision/recall triangular graph for three SEALS supported tracks: benchmark2 (best five matchers, triangulars), anatomy (best four matchers, circles) and conference (best eight matchers, squares). Horizontal line depicts level of precision/recall while values of F-measure are depicted by areas bordered by corresponding lines for F-measure=0.[5|6|7].
|
|
|
New datasets for OAEI 2011 available |
|
Tuesday, 23 August 2011 15:51 |
We have been working on improving existing and adding new datasets to OAEI during the last weeks. As one of the results we propose a new test suite, referred to as benchmark2. This dataset has been generated based on one of the ontologies from the OntoFarm dataset. Similar to the old benchmark test suite it consists of one reference ontology and many modified variants of this ontology. The dataset and more information can be found at http://oaei.ontologymatching.org/2011/benchmarks2/.
Further, we have improved the reference alignment of the OAEI Anatomy track by removing and adding in total around 50 correspondences. We made some evaluation experiments with last years submissions. Compared to the old reference alignment matching systems have now slightly higher precision and recall. We like to thank Elena Beisswanger from JULIE Lab for her suggestions based on a comprehensive analysis of the dataset. The dataset is available at http://web.informatik.uni-mannheim.de/oaei/anatomy11/.
Both datasets are also stored as test suites in the SEALS test repository. Just search for 'anatomy' or 'benchmark2' in the Test Data Repository after logging in.
|
|
Online Ontology Matching Evaluation Service into the SEALS Portal |
|
Monday, 28 March 2011 13:27 |
|
The online evaluation service for ontology matching, that has been used for the first SEALS/OAEI evaluation campaign, is now integrated into the SEALS portal: https://www.seals-project.eu/ontology-matching-evaluation-ui
Developers can continue using the service for testing their systems, with one of three OAEI test data sets (Anatomy, Benchmark and Conference). The only restriction is that now developers have to join the SEALS community for accessing the service.
|
|
Reference alignment of the OAEI anatomy track available |
|
Tuesday, 15 February 2011 10:48 |
|
 After discussing pros and cons of publishing the reference alignment of the OAEI anatomy track, we decided finally that it is time for this step. We had several requests over the last years. However, in the past we have been committed to the advantages of having a blind test instead of having a dataset that is completely free and allows for doing comprehensive tests to improve the matching algorithm and / or to find a good configuration. While it is hard to avoid any kind of overfitting - especially when the number of available datasets with a gold standard is limited - it is clear that an open dataset can help a developer of a matching systems a lot to increase recall and precision of the system in a way that will also work in general or at least for ontologies from a similar domain.
The reference alignment is available now via the OAEI webpages of the anatomy track. We would like to gratefully thank Martin Ringwald and Terry Hayamizu (Mouse Genome Informatics), who provided us not only with an initial reference alignment for the matching task of the anatomy track, but also allowed us to publish the material now. Best regards,
The organizers of the OAEI anatomy track.
|
|
Results of SEALS Ontology Matching campaign |
|
Monday, 15 November 2010 16:27 |
|
The SEALS campaign has been conducted in 2010 in collaboration with the Ontology Alignment Evaluation Initiative (OAEI). The OAEI has been the basis for evaluation over the last five years. It is an annual evaluation campaign that offers several data sets and corresponding evaluations (called "tracks") organized by different groups of researchers. SEALS has supported the OAEI by providing software components for automizing the most important aspects of this years campaign. In particular, a web service interface wrapping the functionality of a matching tool has been proposed. Tool developers, thus, had to wrap their service in the proposed way. As a result, an evaluation service developed by SEALS could be used for both preliminary tests and the registration of the final and official OAEI results. 
We were not sure that switching to an automated evaluation would preserve the success of OAEI, given that the effort of implementing a web service interface was required from participants. For that reason members of the SEALS project stayed in contact with most of the participants during the process of implementing and testing the interface. Contrary to our assumptions, the number of participants is similar to the last years of OAEI and the feedback from participants has been positive. An overview about participation can be found at http://oaei.ontologymatching.org/2010/results/. The tracks conducted in the SEALS modality have been the anatomy, benchmark, and conference track. The precision-recall-graph depicted here as an example shows an analysis conducted as part of evaluating the benchmark results. The graph cut the results given by the participants under a threshold necessary for achieving n% recall and compute the corresponding precision. Systems for which these graphs are not meaningful (because they did not provide graded confidence values) are drawn in dashed lines. Track specific, detailed results and interpretations can be found on the respective webpage. We thank all the participants who used the SEALS service for participating at the OAEI. Cheers go out to AgreementMaker, AROMA, ASMOV, BLOOMS, CODI, Eff2Mat, Falcon-AO, GeromeSMB, MapPSO, NBJLM, RiMOM, SOBOM & TaxoMap. We would also like to thank all the people visiting our poster at the ISWC-2010 Poster & Demo session for fruitful discussions regarding further steps in the development of an evaluation infrastructure. We will be happy to see your tool next year running on the SEALS platform!
|
|
|
|
|
<< Start < Prev 1 2 Next > End >>
|
|
Page 1 of 2 |
Get newsfeed!
