This is the first of a two-part blog post on the past and future of automated evaluation of predicted semantic mappings. I describe some of the challenges and opportunities for challenges like the Ontology Alignment Evaluation Initiative (OAEI) and how they could be addressed with Simple Standard for Sharing Ontological Mappings (SSSOM), and its ecosystem. In the second part, I present a concrete implementation of a new automated evaluation system that address some of those challenges.

Brief Background on OAEI

While there have been many manual, automated, and semi-automated efforts towards ontology alignment, this post will focus on the Ontology Alignment Evaluation Initiative (OAEI) as a well-established, yearly competition occurring since 2004. It has produced and evaluated benchmarks for ontology mapping software across biology, medicine, ecology, digital humanities, archaeology, and other disciplines.

Caution

I found navigating the OAEI site difficult and its artifacts hard to find, so it’s possible I missed something. If you find that I did, please let me know with the comment box at the bottom of the post and I will update the post accordingly.

The OAEI documents on their website that ontology mapping software should consume ontologies in the Web Ontology Language (OWL) format and outputs predicted semantic mappings the XML-based Expressive and Declarative Ontology Alignment Language (EDOAL) format.

Originally, the results were to be submitted to the OAEI’s Alignment API and Alignment Server, but there are now three additional frameworks for preparing code for submission including the HOBBIT, MELT, and SEALS1. I even found in the OAEI’s 2024 results paper that the MELT framework can now accept mappings in the Simple Standard for Sharing Ontological Mappings (SSSOM).

During its two-decade runtime, the OAEI consistently reuses the same benchmarks. For example, the largebio task for mapping between the Foundational Model of Anatomy (FMA) ontology, Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT), and United States National Cancer Institute Thesaurus (NCIT) ran between 2011 and 2022 before being incorporated into the Bio-ML task, which still runs as of writing in 2026.

Challenges and Opportunities

There are several opportunities for improving the way the evaluation of predicted semantic mappings is done. Even though I’m using OAEI as an example along with its format and technology stack, these ideas are widely applicable to any automated evaluation of predicted semantic mappings.

Improved Standards and Software

The revelation that the OAEI can now accept semantic mappings in SSSOM came late into the process of writing this post, so the main point to make on the software is that the proliferation and complexity of four frameworks for submitting to OAEI (and their need to support the legacy EDOAL format) is overwhelming and not obviously documented from their website.

expand to show why SSSOM is great anyway

SSSOM presents an alternative for the automated evaluation of predicted semantic mappings whose:

  • specification is defined with modern tooling (i.e., LinkML)
  • primary exchange format is simpler (TSV instead of XML)
  • documentation is more approachable and complete
  • software ecosystem ( sssom-pydantic, sssom-py, and sssom-java) is still under active development (the Alignment API project has been inactive since 2021) and available in Python and does not require containerization

Further, the concept of evaluation could be greatly simplified. Ideally, evaluation would just be the comparison of semantic mappings from two sources: manual curation from experts and predictions from software. SSSOM has the benefit of storing a mapping justification that tracks exactly this, enabling the ingestion of arbitrary number of self-documenting SSSOM files, rather than relying on external configuration, file identify, file naming conventions, or other fragile mechanisms to differentiate between mapping types.

As an aside, sssom-py implements a preliminary EDOAL importer, however, this would need revisiting and re-reimplementing in SSSOM-Pydantic with unit tests, examples, and better documentation. Establishing interoperability between data formats makes transitioning much easier, and allows for more abstract thinking about the problem space rather than worrying about the politics and history of the formats themselves.

Improved Transparency and Longevity

The OAEI does not have an obvious mechanism through which it stores and shares predicted semantic mappings and their metadata. In some cases, I was able to find links from the published results to files that contained the results from the challenges, but not consistently.

This is an openness and transparency issue that poses challenges in the reuse of previously results, such as for reproduction. It also hinders historical analyses such as the elucidation trends in software performance over time.

Community repositories for semantic mappings like Biomappings demonstrated how an open data, open code, and open infrastructure (O3) approach democratizes the storage and curation of semantic mappings. O3 resources achieve transparency and longevity by making their static results available through long-term archival systems like Zenodo and manually curated results available through version control systems like Git via a forge like GitHub.

Biomappings spun out its code into the stand-alone SSSOM Curator software package to enable the creation and management of new semantic mapping repositories beyond Biomappings, which might be created for various tasks, projects, or domains.

Making the Results Actionable

If the predicted semantic mappings were stored and shared, then expert curators could review them and submit them to the upstream resources that they concern. This would be an opportunity for OAEI to have a meaningful impact on the downstream scientific tasks that consume mappings, as the goal of predicting mappings is to complete alignments between pairs of ontologies (and other semantic spaces).

Again, the Biomappings repository pioneered combining the semantic mapping prediction, storage, and curation workflows into a single software stack. Such a workflow could be adopted by OAEI (and similar mapping challenges) to increase its downstream impact. The use of SSSOM also feeds into ontology curation environments like the Ontology Development Kit (ODK), which have implemented SSSOM support in order to enable the external curation of semantic mappings. I wrote about this in detail in a previous post.

The OAEI could achieve more meaningful impact by storing the predictions in the SSSOM a git repository where they could be more easily reviewed, either with SSSOM Curator or any other similar workflow that maintains full provenance from prediction to review/curation.

Making Benchmarks More Dynamic

The benchmark datasets in OAEI appear to be static, i.e., they do not appear to be updated as the resources they concern are updated with new terms and new first-party semantic mappings. Ideally, the benchmarks would be updated each year to incorporate not only first party curations from the ontologies, but also third party reviews/curations of predicted mappings from previous years’ competitions. Benchmark tasks could then be retired when the alignment between two or more ontologies are completed.

Further, a workflow that can (semi-)automatically maintain a benchmark should also be able to easier produce new benchmarking tasks based on first-party semantic mappings available in ontologies and third-party semantic mappings from repositories like Biomappings.

Conclusions

Competitions like the Critical Assessment of Structure Prediction (CASP) work because they ask for predictions to be made before performing an expensive, specialized experiment in the laboratory. Conversely, the prediction of semantic mappings exists in a very different space, where predictions can be easily reviewed and validated. Therefore, I think that the automated evaluation of predicted semantic mappings is long overdue for a perspective change, where the goal shouldn’t be to focus on the development of new tools, but rather to focus on the prediction and curation of novel mappings that serve downstream scientific applications.

In short: make your predictions, curate them, contribute them to Biomappings or directly upstream, then everyone benefits, and we don’t have to keep playing this game.

In the second part of this post, I present an implementation of a new workflow for constructing and evaluating benchmarks that closes this loop and addresses many of the challenges and opportunities described above.

Appendix

For reference, here are the OAEI Calls and Publications between 2004-2026:

Year Call Publication
2026 Call  
2025 Call  
2024 Call https://inria.hal.science/hal-04892635/
2023 Call https://ora.ox.ac.uk/objects/uuid:e167c7dc-72cd-476a-ba23-d4bcc86e0b60
2022 Call https://hal.science/hal-04351729/
2021 Call https://openaccess.city.ac.uk/id/eprint/27602/
2020 Call https://hal.science/hal-04312966/
2019 Call https://openaccess.city.ac.uk/id/eprint/23708/
2018 Call https://hal.science/hal-02089249/
2017 Call https://air.unimi.it/handle/2434/550707
2016 Call https://inria.hal.science/hal-01421833/
2015 Call https://hal.science/hal-01254907/
2014 Call https://hal.science/hal-01180915/
2013 Call https://inria.hal.science/hal-01140027/
2012 Call https://inria.hal.science/hal-00768409/
2011 Call https://inria.hal.science/hal-00781022/
2010 Call https://inria.hal.science/hal-00793276/
2009 Call https://inria.hal.science/hal-00794918/
2008 Call https://inria.hal.science/hal-00793535/
2007 Call https://inria.hal.science/hal-00822893/
2006 Call https://ceur-ws.org/Vol-225/paper7.pdf
2005 Call https://inria.hal.science/hal-00922283/
2004 Call https://inria.hal.science/hal-04892635/

  1. The most recent report from OAEI in 2024 points to https://seals-project.eu, which appears to have been hijacked by a cryptocurrency scam. ↩