The scalability evaluation has the goal of evaluating the scalability of semantic technologies in terms of their time and space characteristics. More concretely, the scalability evaluation will evaluate the ability of semantic technologies to handle large ontologies and will measure the amount of time it takes to perform particular operations on these semantic technologies and how much system memory is consumed in the process of conducting these operations.

In the evaluation we need an automated and uniform way to access semantic tools. Due to the high heterogeneity in semantic tools, ontology management APIs vary from one tool to another. Therefore, in the same way as in the conformance and interoperability evaluations, the way chosen to automatically access the tools is through the following two operations commonly supported by most semantic tools: to import an ontology from a file, and to export an ontology to a file.

During the experiment, a common group of tests are executed and each test describes one input ontology that has to be either imported and exported. As with the conformance testing, test executions comprise either two steps. In the case of evaluating the scalability of a tool, an ontology import operation needs to be executed on a file containing a single ontology, followed by an export operation to another file. The evaluation of the scalability of the import operation involves measuring the time in milliseconds before the import operation is executed, the memory usage of the semantic tool before the import operation is executed, the time in milliseconds after the operation is executed, and the memory usage of the semantic tool after the operation is executed. This evaluation is conducted in a similar way for the export operation, producing other four values for memory and time.

The scalability evaluation does not ensure that the process of importation or exportation is performed correctly, as the scalability test will show the scalability results for the import and export operations in the context of the conformance and interoperability results generate by the conformance and interoperability evaluations described in conformance and interoperability evaluations.

The only requirement for performing the evaluation on a tool is that the tool is able of importing and exporting ontologies in the ontology language. Therefore, this evaluation can be performed not only on ontology engineering tools but also on other types of semantic technologies (e.g., ontology repositories). The tool features covered in this evaluation are the tool importers and exporters.