Metrics for Experimentation Programs: Categories, Benefits and Challenges

Nils Stotz; Paul Drews

doi:10.1007/978-3-031-94544-1_15

Metrics for Experimentation Programs: Categories, Benefits and Challenges

Nils Stotz^*
, Paul Drews

^*Corresponding author for this work

Professorship of Information Systems, in particular Digital Transformation and Information Management

Leuphana University of Lüneburg

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

Abstract

Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

Original language	English
Title of host publication	Agile Processes in Software Engineering and Extreme Programming : 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings
Editors	Sibylle Peter, Martin Kropp, Ademar Aguiar, Craig Anslow, Maria Ilaria Lunesu, Andrea Pinna
Number of pages	16
Place of Publication	Cham
Publisher	Springer Nature
Publication date	2025
Pages	210-225
ISBN (Print)	978-3-031-94543-4
ISBN (Electronic)	978-3-031-94544-1
DOIs	https://doi.org/10.1007/978-3-031-94544-1_15
Publication status	Published - 2025
Event	26th International Conference on Agile Software Development - XP 2025 - Brugg-Windisch, Switzerland Duration: 02.06.2025 → 05.06.2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2025.

Research areas and keywords

A/B Testing
Continuous Experimentation
Experimentation Metrics
Program-Level Evaluation
Business informatics

ASJC Scopus Subject Areas

Management Information Systems
Control and Systems Engineering
Business and International Management
Information Systems
Modelling and Simulation
Information Systems and Management

Access to Document

10.1007/978-3-031-94544-1_15Licence: CC BY

Cite this

Stotz, N., & Drews, P. (2025). Metrics for Experimentation Programs: Categories, Benefits and Challenges. In S. Peter, M. Kropp, A. Aguiar, C. Anslow, M. I. Lunesu, & A. Pinna (Eds.), Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings (pp. 210-225). ( Lecture Notes in Business Information Processing; Vol. 545). Springer Nature. https://doi.org/10.1007/978-3-031-94544-1_15

Stotz, Nils ; Drews, Paul. / Metrics for Experimentation Programs: Categories, Benefits and Challenges. Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. editor / Sibylle Peter ; Martin Kropp ; Ademar Aguiar ; Craig Anslow ; Maria Ilaria Lunesu ; Andrea Pinna. Cham : Springer Nature, 2025. pp. 210-225 ( Lecture Notes in Business Information Processing).

@inbook{63d2eb4c93a447a4a2b9e81b2a41c7a6,

title = "Metrics for Experimentation Programs: Categories, Benefits and Challenges",

abstract = "Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.",

keywords = "A/B Testing, Continuous Experimentation, Experimentation Metrics, Program-Level Evaluation, Business informatics",

author = "Nils Stotz and Paul Drews",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.; 26th International Conference on Agile Software Development - XP 2025, XP 2025 ; Conference date: 02-06-2025 Through 05-06-2025",

year = "2025",

doi = "10.1007/978-3-031-94544-1\_15",

language = "English",

isbn = "978-3-031-94543-4",

series = " Lecture Notes in Business Information Processing",

publisher = "Springer Nature",

pages = "210--225",

editor = "Sibylle Peter and Martin Kropp and Ademar Aguiar and Craig Anslow and Lunesu, \{Maria Ilaria\} and Andrea Pinna",

booktitle = "Agile Processes in Software Engineering and Extreme Programming",

address = "Switzerland",

}

Stotz, N & Drews, P 2025, Metrics for Experimentation Programs: Categories, Benefits and Challenges. in S Peter, M Kropp, A Aguiar, C Anslow, MI Lunesu & A Pinna (eds), Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. Lecture Notes in Business Information Processing, vol. 545, Springer Nature, Cham, pp. 210-225, 26th International Conference on Agile Software Development - XP 2025, Brugg-Windisch, Switzerland, 02.06.25. https://doi.org/10.1007/978-3-031-94544-1_15

Metrics for Experimentation Programs: Categories, Benefits and Challenges. / Stotz, Nils; Drews, Paul.
Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. ed. / Sibylle Peter; Martin Kropp; Ademar Aguiar; Craig Anslow; Maria Ilaria Lunesu; Andrea Pinna. Cham: Springer Nature, 2025. p. 210-225 ( Lecture Notes in Business Information Processing; Vol. 545).

Research output: Contributions to collected editions/works › Article in conference proceedings › Research › peer-review

TY - CHAP

T1 - Metrics for Experimentation Programs: Categories, Benefits and Challenges

AU - Stotz, Nils

AU - Drews, Paul

N1 - Publisher Copyright: © The Author(s) 2025.

PY - 2025

Y1 - 2025

N2 - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

AB - Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

KW - A/B Testing

KW - Continuous Experimentation

KW - Experimentation Metrics

KW - Program-Level Evaluation

KW - Business informatics

UR - https://www.scopus.com/pages/publications/105007523247

U2 - 10.1007/978-3-031-94544-1_15

DO - 10.1007/978-3-031-94544-1_15

M3 - Article in conference proceedings

SN - 978-3-031-94543-4

T3 - Lecture Notes in Business Information Processing

SP - 210

EP - 225

BT - Agile Processes in Software Engineering and Extreme Programming

A2 - Peter, Sibylle

A2 - Kropp, Martin

A2 - Aguiar, Ademar

A2 - Anslow, Craig

A2 - Lunesu, Maria Ilaria

A2 - Pinna, Andrea

PB - Springer Nature

CY - Cham

T2 - 26th International Conference on Agile Software Development - XP 2025

Y2 - 2 June 2025 through 5 June 2025

ER -

Stotz N, Drews P. Metrics for Experimentation Programs: Categories, Benefits and Challenges. In Peter S, Kropp M, Aguiar A, Anslow C, Lunesu MI, Pinna A, editors, Agile Processes in Software Engineering and Extreme Programming: 26th International Conference on Agile Software Development, XP 2025, Brugg-Windisch, Switzerland, June 2–5, 2025, Proceedings. Cham: Springer Nature. 2025. p. 210-225. ( Lecture Notes in Business Information Processing). Epub 2025 May 29. doi: 10.1007/978-3-031-94544-1_15

Metrics for Experimentation Programs: Categories, Benefits and Challenges

Abstract

Bibliographical note

Research areas and keywords

ASJC Scopus Subject Areas

Access to Document

Fingerprint

Cite this