Cultural heritage professionals aim to improve the way we understand paintings by generating descriptions of them. However, since millions of cultural objects have been created throughout history, completing such a task seems impossible, but only for humans.

The goal of the Saint George on a Bike (SGoaB) project is to provide high-performance metadata enrichment capability by using High Performance Computing (HPC) resources in the cultural heritage domain. The project trains natural language processing and deep learning algorithms in culture, symbols, and historical context to automatically generate rich metadata for hundreds of thousands of images from various European cultural heritage repositories. Within the project, researchers and engineers are creating rich descriptions for paintings to train AI models, so these can in turn analyse many more thousands of paintings automatically.

In this regard the project has launched an engaging and inspiring video that aims to show how AI can help the cultural heritage sector and its potential to soon recognize the context of artwork and generate accurate annotations automatically. The audience of this video are professionals working in the GLAM sector and also the general public.

Among other advantages of applying AI in the cultural heritage sector the video highlights the improved experience visually impaired people could have with better descriptions; the possibility to study hidden relations between thousands of items simultaneously and the opportunity to curate virtual exhibitions with related paintings from around the globe.

Saint George on a Bike: Training AI to be aware of cultural heritage contexts

Automatic image captioning is a process that allows already trained models running on commodity computers to generate textual descriptions from an image. It is a burgeoning reality in a handful of other areas such as classifying image contents on social media. However, to date, no AI system has been built and trained to help in the description of cultural heritage images, while factoring in the time-period and scene composition rules for sacred iconography from the 14th to the 18th centuries.

As part of the Saint George on a Bike project, researchers at the Barcelona Supercomputing Center and Europeana Foundation build and train AI systems to help cultural heritage institutions describe and classify their art pieces automatically.  In the end both casual users and cultural heritage professionals will benefit from a better access to collections and also a better experience navigating through collection catalogs. They will owe this to richer artwork annotations, leading to improved image scene indexation and search capabilities, obtained with the help of a specialized AI system.

“Our project will allow quick access to enriched cultural information, which can serve equally well for cultural and social ends, education, tourism, and possibly for historians or anthropologists. Indirectly the citizens can benefit from better public services, when these are based on the insight that the richer metadata we produce offers – such as web accessibility for the visually impaired or narratives that can expose social injustice or integration and gender issues through cultural heritage corpora and help create a more tolerant European identity”, says Maria-Cristina Marinescu, coordinator of the Saint George on a Bike project.

To learn more about the Saint George on a Bike project, visit https://saintgeorgeonabike.eu/.