GANDR - Georelating Dataset, Metrics, and Evaluation

Kai Moltzen; Ricardo Usbeck

doi:10.1145/3764912.3770819

GANDR - Georelating Dataset, Metrics, and Evaluation

Professur für Wirtschaftsinformatik, insbesondere Künstliche Intelligenz und Erklärbarkeit

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

Abstract

Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.

Originalsprache	Englisch
Titel	GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery
Redakteure/-innen	Shawn Newsam, Lexie Yang, Song Gao, Di Zhu, Dalton Lunga, Gengchen Mai, Bruno Martins, Samantha Arundel
Seitenumfang	11
Erscheinungsort	New York, NY, USA
Herausgeber (Verlag)	Association for Computing Machinery
Erscheinungsdatum	19.12.2025
Seiten	61-71
ISBN (Print)	9798400721793
ISBN (elektronisch)	9798400721793
DOIs	https://doi.org/10.1145/3764912.3770819
Publikationsstatus	Erschienen - 19.12.2025

Bibliographische Notiz

Publisher Copyright:
© 2025 Copyright held by the owner/author(s)

UN SDGs

Dieser Output leistet einen Beitrag zu folgendem(n) Ziel(en) für nachhaltige Entwicklung

SDG 11 – Nachhaltige Städte und Gemeinschaften

ASJC Scopus Sachgebiete

Artificial intelligence
Geografie, Planung und Entwicklung

Zugriff auf Dokument

10.1145/3764912.3770819Lizenz: CC BY-SA

Dieses zitieren

Moltzen, K., & Usbeck, R. (2025). GANDR - Georelating Dataset, Metrics, and Evaluation. in S. Newsam, L. Yang, S. Gao, D. Zhu, D. Lunga, G. Mai, B. Martins, & S. Arundel (Hrsg.), GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (S. 61-71). (GeoAI '25). Association for Computing Machinery. https://doi.org/10.1145/3764912.3770819

Moltzen, Kai ; Usbeck, Ricardo. / GANDR - Georelating Dataset, Metrics, and Evaluation. GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. Hrsg. / Shawn Newsam ; Lexie Yang ; Song Gao ; Di Zhu ; Dalton Lunga ; Gengchen Mai ; Bruno Martins ; Samantha Arundel. New York, NY, USA : Association for Computing Machinery, 2025. S. 61-71 (GeoAI '25).

@inbook{6aab3a7c29a4417b8293acf74439553d,

title = "GANDR - Georelating Dataset, Metrics, and Evaluation",

abstract = "Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.",

keywords = "DGGS, Dataset, Georelating, LLMs, Metrics, Spatial Reasoning",

author = "Kai Moltzen and Ricardo Usbeck",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s)",

year = "2025",

month = dec,

day = "19",

doi = "10.1145/3764912.3770819",

language = "English",

isbn = "9798400721793",

series = "GeoAI '25",

publisher = "Association for Computing Machinery",

pages = "61--71",

editor = "Shawn Newsam and Lexie Yang and Song Gao and Di Zhu and Dalton Lunga and Gengchen Mai and Bruno Martins and Samantha Arundel",

booktitle = "GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery",

address = "United States",

}

Moltzen, K & Usbeck, R 2025, GANDR - Georelating Dataset, Metrics, and Evaluation. in S Newsam, L Yang, S Gao, D Zhu, D Lunga, G Mai, B Martins & S Arundel (Hrsg.), GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. GeoAI '25, Association for Computing Machinery, New York, NY, USA, S. 61-71. https://doi.org/10.1145/3764912.3770819

GANDR - Georelating Dataset, Metrics, and Evaluation. / Moltzen, Kai ; Usbeck, Ricardo.
GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. Hrsg. / Shawn Newsam; Lexie Yang; Song Gao; Di Zhu; Dalton Lunga; Gengchen Mai; Bruno Martins; Samantha Arundel. New York, NY, USA: Association for Computing Machinery, 2025. S. 61-71 (GeoAI '25).

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

TY - CHAP

T1 - GANDR - Georelating Dataset, Metrics, and Evaluation

AU - Moltzen, Kai

AU - Usbeck, Ricardo

PY - 2025/12/19

Y1 - 2025/12/19

N2 - Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.

AB - Georelating has been introduced to learn geospatial representations of events from textual reports, which requires the interpretation of spatial relations. To foster the development and evaluation of Georelating systems, we construct the silver-standard Georelating Annotated Natural Disaster Reports dataset GANDR and benchmark our LLM agent architecture as a baseline (areal F1 = 0.609, fuzzy cell match score = 0.833) for this new task.GANDR comprises synthetic disaster reports referencing 1,000 US and 1,000 EU cities, annotated with Discrete Global Grid System (DGGS) cells for efficient geospatial integration. We propose a set of five complementary metrics capitalizing on the DGGS annotations for efficient and comprehensive evaluation.Analysis reveals the potential of reasoning LLMs integrated with geographical knowledge bases to address variation across spatial relations such as (inter-)cardinal directions. We highlight the estimation of the impact area's size as a key challenge of Georelating.

KW - DGGS

KW - Dataset

KW - Georelating

KW - LLMs

KW - Metrics

KW - Spatial Reasoning

UR - https://www.scopus.com/pages/publications/105026337752

U2 - 10.1145/3764912.3770819

DO - 10.1145/3764912.3770819

M3 - Article in conference proceedings

SN - 9798400721793

T3 - GeoAI '25

SP - 61

EP - 71

BT - GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery

A2 - Newsam, Shawn

A2 - Yang, Lexie

A2 - Gao, Song

A2 - Zhu, Di

A2 - Lunga, Dalton

A2 - Mai, Gengchen

A2 - Martins, Bruno

A2 - Arundel, Samantha

PB - Association for Computing Machinery

CY - New York, NY, USA

ER -

Moltzen K , Usbeck R. GANDR - Georelating Dataset, Metrics, and Evaluation. in Newsam S, Yang L, Gao S, Zhu D, Lunga D, Mai G, Martins B, Arundel S, Hrsg., GeoAI 2025 - Proceedings of the 8th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. New York, NY, USA: Association for Computing Machinery. 2025. S. 61-71. (GeoAI '25). doi: 10.1145/3764912.3770819

GANDR - Georelating Dataset, Metrics, and Evaluation

Abstract

Bibliographische Notiz

UN SDGs

ASJC Scopus Sachgebiete

Zugriff auf Dokument

Fingerprint

Dieses zitieren