The ontology engineering evaluation campaign was organized along three scenarios that covered the conformance, interoperability and scalability of these tools. A detailed description of these evaluation scenarios and of the test data used can be found in SEALS Deliverables D10.1 and D10.2.

Tools evaluated

Sometimes tools use ontology management frameworks for processing ontologies, which has an effect in their conformance and interoperability. The next table shows the tools evaluated and the ontology management frameworks (i.e., APIs) that they use, including in both cases their version numbers.

Ontology Management frameworks
Tool	Version
Jena	2.6.3
OWL API	3.0.1 1592
Sesame	2.3.1
Ontology editors
Tool	Version	API	API version
NeOn Toolkit	2.3.2	OWL API	3.0.0 1310
Protégé 4	4.1 beta 209	OWL API	3.1.0 1602
Protégé OWL	3.4.4 build 579	Protégé OWL API	3.4.4 build 579

 

Conformance results

The following tables present the tool conformance results for RDF(S), OWL Lite, OWL DL and OWL Full, respectively. The tables show the number of tests in each category in which the results of a test can be classified, depending on whether the original and the resultant ontologies are the same (SAME), whether they are different (DIFF), or whether the tool execution fails (FAIL).

 

RDF(S) Conformance results.

Category	JE	NT	OA	P4	PO	SE
SAME	82	0	0	0	68	82
DIFF	0	82	82	82	14	0
FAIL	0	0	0	0	0	0
TOTAL	82	82	82	82	82	82

 

OWL Lite Conformance results.

Category	JE	NT	OA	P4	PO	SE
SAME	82	78	80	80	73	82
DIFF	0	2	2	2	9	0
FAIL	0	2	0	0	0	0
TOTAL	82	82	82	82	82	82

OWL DL Conformance results.

Category	JE	NT	OA	P4	PO	SE
SAME	561	549	549	549	429	561
DIFF	0	8	11	11	132	0
FAIL	0	4	1	1	0	0
TOTAL	561	561	561	561	561	561

 

OWL Full Conformance results.

Category	JE	NT	OA	P4	PO	SE
SAME	90	6	6	6	73	90
DIFF	0	78	83	83	13	0
FAIL	0	6	1	1	4	0
TOTAL	90	90	90	90	90	90

 

Interoperability results

The tables show the percentage of tests in which the original and the resultant ontologies in an interchange are the same. For each cell, the row indicates the tool origin of the interchange, whereas the column indicates the tool destination of the interchange.

 

RDF(S) Interoperability results.

	JE	SE	PO	NT	OA	P4
JE	100	100	83	0	0	0
SE	100	100	83	0	0	0
PO	83	83	83	0	0	0
NT	0	0	0	0	0	0
OA	0	0	0	0	0	0
P4	0	0	0	0	0	0

 

OWL Lite Interoperability results.

	JE	SE	OA	P4	NT	PO
JE	100	100	98	98	95	89
SE	100	100	98	98	95	89
OA	98	98	98	98	95	89
P4	98	98	98	98	95	89
NT	95	95	95	95	95	87
PO	89	89	89	89	87	89

OWL DL Interoperability results.

	JE	SE	OA	P4	NT	PO
JE	100	100	98	98	98	76
SE	100	100	98	98	98	76
OA	98	98	98	98	98	75
P4	98	98	98	98	98	75
NT	98	98	98	98	98	75
PO	76	76	75	75	75	76

 

OWL Full Interoperability results.

	JE	SE	OA	P4	NT	PO
JE	100	100	81	7	7	7
SE	100	100	81	7	7	7
PO	82	82	83	7	7	7
OA	0	7	0	7	7	7
P4	0	7	0	7	7	7
NT	0	7	0	7	7	7

 

Summary of conformance and interoperability results

In the results we can see that all those tools that manage ontologies at the RDF level (Jena and Sesame) have no problems in processing ontologies regardless of the ontology language. Since the rest of the tools evaluated are based in OWL or in OWL 2, their conformance and interoperability is clearly better when dealing with OWL ontologies.

From the results presented in chapter 3 we can see that conformance and interoperability are highly influenced by development decisions. For example, the decision of the OWL API developers (propagated to all the tools that use it for ontology management) of converting all the ontologies into OWL 2 makes the RDF(S) conformance and interoperability of these tools quite low.

Since the OWL Lite language is a subset of the OWL DL one, there is a dependency between the results obtained using the test suites for OWL Lite and OWL DL. In the results we can also see that, since the OWL DL test suite is more exhaustive than the OWL Lite one, the OWL DL evaluation unveiled more problems in the tools than the OWL Lite evaluation. These problems included not only issues related to the OWL DL language, but also issues related to OWL Lite ontologies included in the OWL DL test suite.

The results also show the dependency between the results of a tool and those of the ontology management framework that it uses; using a framework does not isolate a tool from having conformance or interoperability problems. Besides inheriting existing problems in the framework (if any), a tool may have more problems if it requires further ontology processing (e.g., its representation formalism is different from that of the framework or an extension of it) or if it affects the correct working of the framework.

However, using ontology management frameworks may help increasing the conformance and interoperability of the tools, since developers do not have to deal with the problems of low-level ontology management. Nevertheless, as observed in the results, this also requires being aware of existing defects in these frameworks and regularly updating the tools to use their latest versions.

 

Scalability results

An overview of the scalability tests, the size of the corresponding ontologies and time spent on importing and exporting the ontology is available in table

 

Scalability results with real-world ontologies.

ScalTest	OntSize	JE	NT	OA	P4	PO	SE
ScalTest010	4KB	less 1	-	-	-	27 sec	less 1
ScalTest007	29KB	less 1	8 sec	6 sec	6 sec	19 sec	less 1
ScalTest004	33KB	less 1	less 1	1 sec	less 1	less 1	less 1
ScalTest001	82KB	1 sec	2 sec	3 sec	3 sec	5 sec	1 sec
ScalTest003	119KB	less 1	1 sec	less 1	less 1	1 sec	less 1
ScalTest009	164KB	less 1	-	-	-	81 sec	less 1
ScalTest008	233KB	less 1	10 sec	11 sec	9 sec	14 sec	1 sec
ScalTest005	274KB	less 1	1 sec	1 sec	1 sec	1 sec	1 sec
ScalTest012	430KB	1 sec	2 sec	less 1	1 sec	1 sec	1 sec
ScalTest013	476KB	1 sec	2 sec	1 sec	less 1	1 sec	less 1
ScalTest002	758KB	2 sec	3 sec	3 sec	2 sec	4 sec	2 sec
ScalTest015	881KB	less 1	3 sec	3 sec	3 sec	8 sec	less 1
ScalTest011	1.3MB	1 sec	3 sec	2 sec	1 sec	4 sec	2 sec
ScalTest016	1.3MB	2 sec	-	-	-	99 sec	1 sec
ScalTest014	4.1MB	4 sec	-	-	-	38 sec	2 sec
ScalTest018	12.6MB	12 sec	-	-	-	23 sec	5 sec
ScalTest020	13.8MB	12 sec	-	-	-	26 sec	6 sec
ScalTest019	13.9MB	14 sec	-	-	-	27 sec	5 sec
ScalTest017	14.6MB	17 sec	-	-	-	30 sec	6 sec
ScalTest006	37.7MB	-	-	-	-	-	6 sec

 

Scalability results with LUBM data generator ontologies.

ScalTest	OntSize	JE	NT	OA	P4	PO	SE
ScalTest001	8212KB	10 sec	8 sec	7 sec	8 sec	22 sec	4 sec
ScalTest002	18907KB	15 sec	11 sec	10 sec	11 sec	50 sec	5 sec
ScalTest003	27745KB	23 sec	16 sec	14 sec	14 sec	82 sec	7 sec
ScalTest004	39380KB	37 sec	23 sec	21 sec	20 sec	-	11 sec
ScalTest005	51511KB	57 sec	36 sec	31 sec	31 sec	-	13 sec

 

Summary of scalability results

The current scenario, in which the duration of ImportExport operation is measured, is appropriate for the analysis; however, more dimensions should be added. As an example the amount of memory can be measured. In this way the extended scenario will be more suitable for a more complete testing of scalability. With respect to the results of the scalability tests, an extended analysis of the results should be provided in order to analyze why there are problems with some ontologies, how do these problems correlate with the size of the ontologies. It would be relevant to graphically analyze the relationship between the ontology size and the duration of the ImportExport operation.