A Human-centric Ontology Evaluation Process (HERO)

As ontologies enable advanced intelligent applications, ensuring their correctness is crucial. Nevertheless, the lack of a standardized methodology and tool support available for human-centric evaluation of ontologies leads to high organizational efforts of such evaluation campaigns. 

As part of the FWF Honest Project, we established HERO.

HERO is a Human-centric ontology Evaluation pROcess that guides ontology engineers through the relevant steps to be taken in human-in-the-loop ontology evaluation campaigns and can thus reduce the likelihood of crowd-sourced errors or biases and improve the reliability of the results at the design phase of the campaign. The activities, part of HERO can be divided into three main stages: preparation, execution, and follow-up analysis of the crowd-sourced campaign. 

In addition, we implemented a platform based on HERO, which automates a portion of the steps thus further reducing the effort of ontology engineers preparing batch-style human-in-the-loop evaluation campaigns. The reference architecture relies on a core, capturing the general logic and plugins, which extend the platform functionalities and allow for personalizations.



  • Tsaneva, S., Käsznar, K., Sabou, M. (2022). Human-Centric Ontology Evaluation: Process and Tool Support. In: Corcho, O., Hollink, L., Kutz, O., Troquard, N., Ekaputra, F.J. (eds) Knowledge Engineering and Knowledge Management. EKAW 2022. Lecture Notes in Computer Science(), vol 13514. Springer, Cham. https://doi.org/10.1007/978-3-031-17105-5_14
  • Käsznar, K. (2022). A process and tool support for human-centred ontology verification [Diploma Thesis, Technische Universität Wien]. reposiTUm. https://doi.org/10.34726/hss.2022.97603