Construct relation extraction from scientific papers: Is it automatable yet?

The process of identifying relevant prior research
articles is crucial for theoretical advancements, but
often requires significant human effort. This study
examines the feasibility of using large language
models (LLMs) to support this task by extracting
tested hypotheses, which consist of related constructs,
moderators or mediators, path coefficients, and
p-values, from empirical studies using structural
equation modeling (SEM). We combine state-of-the-art
LLMs with a variety of post-processing measures
to improve the relation extraction quality. An
extensive evaluation yields recall scores of up to
79.2% in construct entity extraction, 58.4% in
construct-mediator/moderator-construct extraction,
and 39.3% in extracting the full tested hypotheses.
We provide a manually annotated dataset of 72 SEM
articles and 749 construct relations to facilitate future
research. Our findings offer critical insights and
suggest promising directions for advancing the field of
automated construct relation extraction from scholarly
documents.