Measuring Gender Bias in German Language Generation

Angelie Kraft; Hans Peter Zorn; Pascal Fecht; Judith Simon; Chris Biemann; Ricardo Usbeck

doi:10.18420/inf2022_108

Measuring Gender Bias in German Language Generation

Angelie Kraft
, Hans Peter Zorn
, Pascal Fecht
, Judith Simon
, Chris Biemann
, Ricardo Usbeck

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

1 Zitat (Scopus)

Abstract

Most existing methods to measure social bias in natural language generation are specified for English language models. In this work, we developed a German regard classifier based on a newly crowd-sourced dataset. Our model meets the test set accuracy of the original English version. With the classifier, we measured binary gender bias in two large language models. The results indicate a positive bias toward female subjects for a German version of GPT-2 and similar tendencies for GPT-3. Yet, upon qualitative analysis, we found that positive regard partly corresponds to sexist stereotypes. Our findings suggest that the regard classifier should not be used as a single measure but, instead, combined with more qualitative analyses.

Originalsprache	Englisch
Titel	INFORMATIK 2022 - Informatik in den Naturwissenschaften
Redakteure/-innen	Daniel Demmler, Daniel Krupka, Hannes Federrath
Seitenumfang	18
Erscheinungsort	Bonn
Herausgeber (Verlag)	Gesellschaft für Informatik e.V.
Erscheinungsdatum	2022
Seiten	1257-1274
ISBN (elektronisch)	978-3-88579-720-3
DOIs	https://doi.org/10.18420/inf2022_108
Publikationsstatus	Erschienen - 2022
Extern publiziert	Ja
Veranstaltung	52. Jahrestagung der Gesellschaft für Informatik - INFORMATIK 2022: Informatik in den Naturwissenschaften - UHH Gebäude in der Edmund-Siemers-Allee 1, Hamburg, Deutschland Dauer: 26.09.2022 → 30.09.2022 Konferenznummer: 52 https://informatik2022.gi.de/ https://doi.org/10.25798/pmfp-py55

Bibliographische Notiz

Funding Information:
This work presents and extends the results of Angelie Kraft's Master thesis at Universität Hamburg and inovex GmbH. Regarding any additional research and experimentation, we acknowledge the financial support from the Federal Ministry for Economic Affairs and Energy of Germany in the project CoyPu (project number 01MK21007[G]).

Funding Information:
Acknowledgments This work presents and extends the results of Angelie Kraft’s Master thesis at Universitčt Hamburg and inovex GmbH. Regarding any additional research and experimentation, we acknowledge the financial support from the Federal Ministry for Economic Affairs and Energy of Germany in the project CoyPu (project number 01MK21007[G]).

Publisher Copyright:
© 2022 Gesellschaft fur Informatik (GI). All rights reserved.

UN SDGs

Dieser Output leistet einen Beitrag zu folgendem(n) Ziel(en) für nachhaltige Entwicklung

SDG 5 – Gleichberechtigung der Geschlechter

Fachgebiete und Schlagwörter

Wirtschaftsinformatik
Informatik

Zugriff auf Dokument

10.18420/inf2022_108Lizenz: CC BY-SA

Dieses zitieren

Kraft, A., Zorn, H. P., Fecht, P., Simon, J., Biemann, C., & Usbeck, R. (2022). Measuring Gender Bias in German Language Generation. in D. Demmler, D. Krupka, & H. Federrath (Hrsg.), INFORMATIK 2022 - Informatik in den Naturwissenschaften (S. 1257-1274). (Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI); Band P-326). Gesellschaft für Informatik e.V.. https://doi.org/10.18420/inf2022_108

Kraft, Angelie ; Zorn, Hans Peter ; Fecht, Pascal et al. / Measuring Gender Bias in German Language Generation. INFORMATIK 2022 - Informatik in den Naturwissenschaften. Hrsg. / Daniel Demmler ; Daniel Krupka ; Hannes Federrath. Bonn : Gesellschaft für Informatik e.V., 2022. S. 1257-1274 (Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)).

@inbook{c32ae1abb1bf466f85bb3380d5d01a7a,

title = "Measuring Gender Bias in German Language Generation",

abstract = "Most existing methods to measure social bias in natural language generation are specified for English language models. In this work, we developed a German regard classifier based on a newly crowd-sourced dataset. Our model meets the test set accuracy of the original English version. With the classifier, we measured binary gender bias in two large language models. The results indicate a positive bias toward female subjects for a German version of GPT-2 and similar tendencies for GPT-3. Yet, upon qualitative analysis, we found that positive regard partly corresponds to sexist stereotypes. Our findings suggest that the regard classifier should not be used as a single measure but, instead, combined with more qualitative analyses.",

keywords = "gender bias, german, gpt-2, gpt-3, natural language generation, regard, stereotypes, Business informatics, Informatics",

author = "Angelie Kraft and Zorn, \{Hans Peter\} and Pascal Fecht and Judith Simon and Chris Biemann and Ricardo Usbeck",

note = "This work presents and extends the results of Angelie Kraft's Master thesis at Universit{\"a}t Hamburg and inovex GmbH. Regarding any additional research and experimentation, we acknowledge the financial support from the Federal Ministry for Economic Affairs and Energy of Germany in the project CoyPu (project number 01MK21007[G]). Publisher Copyright: {\textcopyright} 2022 Gesellschaft fur Informatik (GI). All rights reserved.; 52. Jahrestagung der Gesellschaft f{\"u}r Informatik : Informatik in den Naturwissenschaften, INFORMATIK 2022 ; Conference date: 26-09-2022 Through 30-09-2022",

year = "2022",

doi = "10.18420/inf2022\_108",

language = "English",

series = "Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)",

publisher = "Gesellschaft f{\"u}r Informatik e.V.",

pages = "1257--1274",

editor = "Daniel Demmler and Daniel Krupka and Hannes Federrath",

booktitle = "INFORMATIK 2022 - Informatik in den Naturwissenschaften",

address = "Germany",

url = "https://informatik2022.gi.de/, https://doi.org/10.25798/pmfp-py55",

}

Kraft, A, Zorn, HP, Fecht, P, Simon, J, Biemann, C & Usbeck, R 2022, Measuring Gender Bias in German Language Generation. in D Demmler, D Krupka & H Federrath (Hrsg.), INFORMATIK 2022 - Informatik in den Naturwissenschaften. Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI), Bd. P-326, Gesellschaft für Informatik e.V., Bonn, S. 1257-1274, 52. Jahrestagung der Gesellschaft für Informatik - INFORMATIK 2022, Hamburg, Hamburg, Deutschland, 26.09.22. https://doi.org/10.18420/inf2022_108

Measuring Gender Bias in German Language Generation. / Kraft, Angelie; Zorn, Hans Peter; Fecht, Pascal et al.
INFORMATIK 2022 - Informatik in den Naturwissenschaften. Hrsg. / Daniel Demmler; Daniel Krupka; Hannes Federrath. Bonn: Gesellschaft für Informatik e.V., 2022. S. 1257-1274 (Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI); Band P-326).

Publikation: Beiträge in Sammelwerken › Aufsätze in Konferenzbänden › Forschung › Begutachtung

TY - CHAP

T1 - Measuring Gender Bias in German Language Generation

AU - Kraft, Angelie

AU - Zorn, Hans Peter

AU - Fecht, Pascal

AU - Simon, Judith

AU - Biemann, Chris

AU - Usbeck, Ricardo

N1 - Conference code: 52

PY - 2022

Y1 - 2022

N2 - Most existing methods to measure social bias in natural language generation are specified for English language models. In this work, we developed a German regard classifier based on a newly crowd-sourced dataset. Our model meets the test set accuracy of the original English version. With the classifier, we measured binary gender bias in two large language models. The results indicate a positive bias toward female subjects for a German version of GPT-2 and similar tendencies for GPT-3. Yet, upon qualitative analysis, we found that positive regard partly corresponds to sexist stereotypes. Our findings suggest that the regard classifier should not be used as a single measure but, instead, combined with more qualitative analyses.

AB - Most existing methods to measure social bias in natural language generation are specified for English language models. In this work, we developed a German regard classifier based on a newly crowd-sourced dataset. Our model meets the test set accuracy of the original English version. With the classifier, we measured binary gender bias in two large language models. The results indicate a positive bias toward female subjects for a German version of GPT-2 and similar tendencies for GPT-3. Yet, upon qualitative analysis, we found that positive regard partly corresponds to sexist stereotypes. Our findings suggest that the regard classifier should not be used as a single measure but, instead, combined with more qualitative analyses.

KW - gender bias

KW - german

KW - gpt-2

KW - gpt-3

KW - natural language generation

KW - regard

KW - stereotypes

KW - Business informatics

KW - Informatics

UR - https://www.scopus.com/pages/publications/85139747989

U2 - 10.18420/inf2022_108

DO - 10.18420/inf2022_108

M3 - Article in conference proceedings

AN - SCOPUS:85139747989

T3 - Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)

SP - 1257

EP - 1274

BT - INFORMATIK 2022 - Informatik in den Naturwissenschaften

A2 - Demmler, Daniel

A2 - Krupka, Daniel

A2 - Federrath, Hannes

PB - Gesellschaft für Informatik e.V.

CY - Bonn

T2 - 52. Jahrestagung der Gesellschaft für Informatik

Y2 - 26 September 2022 through 30 September 2022

ER -

Kraft A, Zorn HP, Fecht P, Simon J, Biemann C, Usbeck R. Measuring Gender Bias in German Language Generation. in Demmler D, Krupka D, Federrath H, Hrsg., INFORMATIK 2022 - Informatik in den Naturwissenschaften. Bonn: Gesellschaft für Informatik e.V. 2022. S. 1257-1274. (Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)). doi: 10.18420/inf2022_108

Measuring Gender Bias in German Language Generation

Abstract

Bibliographische Notiz

UN SDGs

Fachgebiete und Schlagwörter

Zugriff auf Dokument

Fingerprint

Dieses zitieren