TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering

Andrey Sakhovskiy; Mikhail Salnikov; Irina Nikishina; Aida Usmanova; Angelie Kraft; Cedric Möller; Debayan Banerjee; Junbo Huang; Longquan Jiang; Rana Abdullah; Xi Yan; Dmitry Ustalov; Elena Tutubalina; Ricardo Usbeck; Alexander Panchenko

TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering

Andrey Sakhovskiy
, Mikhail Salnikov
, Irina Nikishina
, Aida Usmanova
, Angelie Kraft
, Cedric Möller
, Debayan Banerjee
, Junbo Huang
, Longquan Jiang
, Rana Abdullah
, Xi Yan
, Dmitry Ustalov
, Elena Tutubalina
, Ricardo Usbeck
, Alexander Panchenko

Professur für Wirtschaftsinformatik, insbesondere Künstliche Intelligenz und Erklärbarkeit

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

9 Zitate (Scopus)

Abstract

This paper describes the results of the Knowledge Graph Question Answering (KGQA) shared task that was co-located with the TextGraphs 2024 workshop. In this task, given a textual question and a list of entities with the corresponding KG subgraphs, the participating system should choose the entity that correctly answers the question. Our competition attracted thirty teams, four of which outperformed our strong ChatGPT-based zero-shot baseline. In this paper, we overview the participating systems and analyze their performance according to a large-scale automatic evaluation. To the best of our knowledge, this is the first competition aimed at the KGQA problem using the interaction between large language models (LLMs) and knowledge graphs.

Originalsprache	Englisch
Titel	Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing : Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics
Redakteure/-innen	Dmitry Ustalov, Yanjun Gao, Alexander Pachenko, Elena Tutubalina, Irina Nikishina, Arti Ramesh, Andrey Sakhovskiy, Ricardo Usbeck, Gerald Penn, Marco Valentino
Seitenumfang	10
Erscheinungsort	Kerrville
Herausgeber (Verlag)	Association for Computational Linguistics (ACL)
Erscheinungsdatum	01.08.2024
Seiten	116-125
ISBN (elektronisch)	979-8-89176-145-2
Publikationsstatus	Erschienen - 01.08.2024
Veranstaltung	TextGraphs-17: Graph-based Methods for Natural Language Processing - Bangkok, Thailand Dauer: 15.08.2024 → 15.08.2024 Konferenznummer: 17 https://sites.google.com/view/textgraphs2024

Bibliographische Notiz

Publisher Copyright:
© 2024 Association for Computational Linguistics.

Fachgebiete und Schlagwörter

Wirtschaftsinformatik

Zugriff auf Dokument

https://aclanthology.org/2024.textgraphs-1.9Lizenz: CC BY

Dieses zitieren

Sakhovskiy, A., Salnikov, M., Nikishina, I., Usmanova, A., Kraft, A., Möller, C., Banerjee, D., Huang, J., Jiang, L., Abdullah, R., Yan, X., Ustalov, D., Tutubalina, E., Usbeck, R., & Panchenko, A. (2024). TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. in D. Ustalov, Y. Gao, A. Pachenko, E. Tutubalina, I. Nikishina, A. Ramesh, A. Sakhovskiy, R. Usbeck, G. Penn, & M. Valentino (Hrsg.), Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing: Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics (S. 116-125). Association for Computational Linguistics (ACL). https://aclanthology.org/2024.textgraphs-1.9

Sakhovskiy, Andrey ; Salnikov, Mikhail ; Nikishina, Irina et al. / TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing: Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics. Hrsg. / Dmitry Ustalov ; Yanjun Gao ; Alexander Pachenko ; Elena Tutubalina ; Irina Nikishina ; Arti Ramesh ; Andrey Sakhovskiy ; Ricardo Usbeck ; Gerald Penn ; Marco Valentino. Kerrville : Association for Computational Linguistics (ACL), 2024. S. 116-125

@inbook{8b2f000ca41a47ed9231e8ac28940620,

title = "TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering",

abstract = "This paper describes the results of the Knowledge Graph Question Answering (KGQA) shared task that was co-located with the TextGraphs 2024 workshop. In this task, given a textual question and a list of entities with the corresponding KG subgraphs, the participating system should choose the entity that correctly answers the question. Our competition attracted thirty teams, four of which outperformed our strong ChatGPT-based zero-shot baseline. In this paper, we overview the participating systems and analyze their performance according to a large-scale automatic evaluation. To the best of our knowledge, this is the first competition aimed at the KGQA problem using the interaction between large language models (LLMs) and knowledge graphs.",

keywords = "Business informatics",

author = "Andrey Sakhovskiy and Mikhail Salnikov and Irina Nikishina and Aida Usmanova and Angelie Kraft and Cedric M{\"o}ller and Debayan Banerjee and Junbo Huang and Longquan Jiang and Rana Abdullah and Xi Yan and Dmitry Ustalov and Elena Tutubalina and Ricardo Usbeck and Alexander Panchenko",

note = "Publisher Copyright: {\textcopyright} 2024 Association for Computational Linguistics.; TextGraphs-17 : Graph-based Methods for Natural Language Processing ; Conference date: 15-08-2024 Through 15-08-2024",

year = "2024",

month = aug,

day = "1",

language = "English",

pages = "116--125",

editor = "Dmitry Ustalov and Yanjun Gao and Alexander Pachenko and Elena Tutubalina and Irina Nikishina and Arti Ramesh and Andrey Sakhovskiy and Ricardo Usbeck and Gerald Penn and Marco Valentino",

booktitle = "Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing",

publisher = "Association for Computational Linguistics (ACL)",

address = "United States",

url = "https://sites.google.com/view/textgraphs2024",

}

Sakhovskiy, A, Salnikov, M, Nikishina, I, Usmanova, A, Kraft, A, Möller, C, Banerjee, D, Huang, J, Jiang, L, Abdullah, R, Yan, X, Ustalov, D, Tutubalina, E, Usbeck, R & Panchenko, A 2024, TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. in D Ustalov, Y Gao, A Pachenko, E Tutubalina, I Nikishina, A Ramesh, A Sakhovskiy, R Usbeck, G Penn & M Valentino (Hrsg.), Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing: Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics (ACL), Kerrville, S. 116-125, TextGraphs-17, Bangkok, Thailand, 15.08.24. <https://aclanthology.org/2024.textgraphs-1.9>

TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. / Sakhovskiy, Andrey; Salnikov, Mikhail; Nikishina, Irina et al.
Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing: Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics. Hrsg. / Dmitry Ustalov; Yanjun Gao; Alexander Pachenko; Elena Tutubalina; Irina Nikishina; Arti Ramesh; Andrey Sakhovskiy; Ricardo Usbeck; Gerald Penn; Marco Valentino. Kerrville: Association for Computational Linguistics (ACL), 2024. S. 116-125.

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

TY - CHAP

T1 - TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering

AU - Sakhovskiy, Andrey

AU - Salnikov, Mikhail

AU - Nikishina, Irina

AU - Usmanova, Aida

AU - Kraft, Angelie

AU - Möller, Cedric

AU - Banerjee, Debayan

AU - Huang, Junbo

AU - Jiang, Longquan

AU - Abdullah, Rana

AU - Yan, Xi

AU - Ustalov, Dmitry

AU - Tutubalina, Elena

AU - Usbeck, Ricardo

AU - Panchenko, Alexander

N1 - Conference code: 17

PY - 2024/8/1

Y1 - 2024/8/1

N2 - This paper describes the results of the Knowledge Graph Question Answering (KGQA) shared task that was co-located with the TextGraphs 2024 workshop. In this task, given a textual question and a list of entities with the corresponding KG subgraphs, the participating system should choose the entity that correctly answers the question. Our competition attracted thirty teams, four of which outperformed our strong ChatGPT-based zero-shot baseline. In this paper, we overview the participating systems and analyze their performance according to a large-scale automatic evaluation. To the best of our knowledge, this is the first competition aimed at the KGQA problem using the interaction between large language models (LLMs) and knowledge graphs.

AB - This paper describes the results of the Knowledge Graph Question Answering (KGQA) shared task that was co-located with the TextGraphs 2024 workshop. In this task, given a textual question and a list of entities with the corresponding KG subgraphs, the participating system should choose the entity that correctly answers the question. Our competition attracted thirty teams, four of which outperformed our strong ChatGPT-based zero-shot baseline. In this paper, we overview the participating systems and analyze their performance according to a large-scale automatic evaluation. To the best of our knowledge, this is the first competition aimed at the KGQA problem using the interaction between large language models (LLMs) and knowledge graphs.

KW - Business informatics

UR - https://www.scopus.com/pages/publications/85204909366

M3 - Article in conference proceedings

SP - 116

EP - 125

BT - Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing

A2 - Ustalov, Dmitry

A2 - Gao, Yanjun

A2 - Pachenko, Alexander

A2 - Tutubalina, Elena

A2 - Nikishina, Irina

A2 - Ramesh, Arti

A2 - Sakhovskiy, Andrey

A2 - Usbeck, Ricardo

A2 - Penn, Gerald

A2 - Valentino, Marco

PB - Association for Computational Linguistics (ACL)

CY - Kerrville

T2 - TextGraphs-17

Y2 - 15 August 2024 through 15 August 2024

ER -

Sakhovskiy A, Salnikov M, Nikishina I, Usmanova A, Kraft A, Möller C et al. TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering. in Ustalov D, Gao Y, Pachenko A, Tutubalina E, Nikishina I, Ramesh A, Sakhovskiy A, Usbeck R, Penn G, Valentino M, Hrsg., Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing: Graph-Based Methods for Natural Language Processing, 62nd Annual Meeting of the Association of Computational Linguistics. Kerrville: Association for Computational Linguistics (ACL). 2024. S. 116-125