There are a few aspects of the evaluation in which we request your cooperation.  

1. The implementation of Java wrapper class which will handle all the communication between your tool and the SEALS evaluation software.
2. Bundle the wrapper and the tool ready for it to be deployed on the SEALS Platform.
3. Convert the questions which will be used for the automated phase into your tool's internal representation.
4. Running of a small-scale subject-based experiment (for which all materials will be provided).

1. The Java wrapper

The first step is to implement the SEALS Search Evaluation Campaign Java API which will enable two-way communication between your tool and the various software we will use to execute the evaluation. This API exposes a small number of core functions such as provided the URI of the ontology to use, submitting a query and requesting the results.

Note: There is a tutorial covering steps 1 and 2 (this is the recommended approach). Feel free to contact us for help or advice in relation to creating the API-compliant wrapper. 

2. Bundle your tool and upload it

The wrapper code should be bundled into a single jar called toolWrapper.jar; the executable code (preferably in the form of one or more jars) and any associated dependency jars should also be included. More information about how to bundle the tool itself is available on pages 23-26 of SEALS Deliverable 9.1.

Note: There is also a tutorial covering steps 1 and 2 (this is the recommended approach). Feel free to contact us for help or advice in relation to creating the API-compliant wrapper. 

Once you have completed the wrapping and bundle process, log in to the community area of the SEALS portal and register your tool and then upload the bundle.

3. Question conversion 

Since participating tools can use varied forms of input, the natural language questions that have been created will have to be transformed into a String representation which your tool, when receiving it via the API will be able to understand and execute. For EvoOnt, there are 50 questions in total and they are split equally across the 5 EvoOnt datasets (10 per set). All the questions are based upon the T-Box specified in som.owl. You are allowed to convert the original questions into your tool's internal representation (serialised as Strings) and we will use those for the evaluation of your tool.  

For more information on this aspect of the evaluation, contact Stuart Wrigley.

4. Experiment

The second aspect of the SEALS evaluation of semantic search tools concerns a small subject-based experiment which focusses on the usability of your tool. All materials (instruction sheets, debriefing questionnaires, control software) will be provided. For instance, the control software will interact with your tool using the wrapper you created in Step 1. Watch this space to find out when the new experiment pack is ready for download - alternatively, email us for a pre-release version!

The experiment involves a human subject using your tool to answer a small number of questions based upon an easy to understand data set (e.g., geography or music, etc.). The data set has been chosen such that no specialist knowledge is required of the field to take part - the emphasis is on assessing the usability of the tool itself. The experiment should take no longer than an hour for each subject to complete. We suggest approximately 10 subjects to give good reliability to the results. Ideally, these subjects should not be directly associated with the development of the tool itself.

Note, it may be possible for SEALS to undertake the user evaluation study on your behalf should you not have the time or resources (thus reducing your participation overhead). Get in touch to discuss your needs.