Understanding metric-related pitfalls in image analysis validation
Autoři
REINKE, Annika, Minu D TIZABI, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NÖTZEL, A Emre KAVUR, Tim RÄDSCH, Carole H SUDRE, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Florian BUETTNER, M Jorge CARDOSO, Veronika CHEPLYGINA, Jianxu CHEN, Evangelia CHRISTODOULOU, Beth A CIMINI, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, Bram van GINNEKEN, Ben GLOCKER, Patrick GODAU, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Fabian ISENSEE, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Jens KLEESIEK, Florian KOFLER, Thijs KOOI, Annette KOPP-SCHNEIDER, Michal KOZUBEK (203 Česká republika, garant, domácí), Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Erik MEIJERING, Bjoern MENZE, Karel GM MOONS, Henning MÜLLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Susanne M RAFELSKI, Nasir RAJPOOT, Mauricio REYES, Michael A RIEGLER, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SÁNCHEZ, Shravya SHETTY, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Ben Van CALSTER, Gaël VAROQUAUX, Ziv R YANIV, Paul F JÄGER a Lena MAIER-HEIN
Vydání
NATURE METHODS, UNITED STATES, NATURE PORTFOLIO, 2024, 1548-7091
Další údaje
Jazyk
angličtina
Typ výsledku
Článek v odborném periodiku
Obor
10201 Computer sciences, information science, bioinformatics
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.
Návaznosti
LM2018129, projekt VaV
Název: Národní infrastruktura pro biologické a medicínské zobrazování Czech-BioImaging
Investor: Ministerstvo školství, mládeže a tělovýchovy ČR, National research infrastructure for biological and medical imaging
REINKE, Annika, Minu D TIZABI, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NÖTZEL, A Emre KAVUR, Tim RÄDSCH, Carole H SUDRE, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Florian BUETTNER, M Jorge CARDOSO, Veronika CHEPLYGINA, Jianxu CHEN, Evangelia CHRISTODOULOU, Beth A CIMINI, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, Bram van GINNEKEN, Ben GLOCKER, Patrick GODAU, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Fabian ISENSEE, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Jens KLEESIEK, Florian KOFLER, Thijs KOOI, Annette KOPP-SCHNEIDER, Michal KOZUBEK, Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Erik MEIJERING, Bjoern MENZE, Karel GM MOONS, Henning MÜLLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Susanne M RAFELSKI, Nasir RAJPOOT, Mauricio REYES, Michael A RIEGLER, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SÁNCHEZ, Shravya SHETTY, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Ben Van CALSTER, Gaël VAROQUAUX, Ziv R YANIV, Paul F JÄGER a Lena MAIER-HEIN. Understanding metric-related pitfalls in image analysis validation. NATURE METHODS. UNITED STATES: NATURE PORTFOLIO, 2024, roč. 21, č. 2, s. 182-194, 20 s. ISSN 1548-7091. Dostupné z: https://dx.doi.org/10.1038/s41592-023-02150-0.
@article{2453698, author = {Reinke, Annika and Tizabi, Minu D and Baumgartner, Michael and Eisenmann, Matthias and HeckmannandNötzel, Doreen and Kavur, A Emre and Rädsch, Tim and Sudre, Carole H and Acion, Laura and Antonelli, Michela and Arbel, Tal and Bakas, Spyridon and Benis, Arriel and Buettner, Florian and Cardoso, M Jorge and Cheplygina, Veronika and Chen, Jianxu and Christodoulou, Evangelia and Cimini, Beth A and Farahani, Keyvan and Ferrer, Luciana and Galdran, Adrian and Ginneken, Bram van and Glocker, Ben and Godau, Patrick and Hashimoto, Daniel A and Hoffman, Michael M and Huisman, Merel and Isensee, Fabian and Jannin, Pierre and Kahn, Charles E and Kainmueller, Dagmar and Kainz, Bernhard and Karargyris, Alexandros and Kleesiek, Jens and Kofler, Florian and Kooi, Thijs and KoppandSchneider, Annette and Kozubek, Michal and Kreshuk, Anna and Kurc, Tahsin and Landman, Bennett A and Litjens, Geert and Madani, Amin and MaierandHein, Klaus and Martel, Anne L and Meijering, Erik and Menze, Bjoern and Moons, Karel GM and Müller, Henning and Nichyporuk, Brennan and Nickel, Felix and Petersen, Jens and Rafelski, Susanne M and Rajpoot, Nasir and Reyes, Mauricio and Riegler, Michael A and Rieke, Nicola and SaezandRodriguez, Julio and Sánchez, Clara I and Shetty, Shravya and Summers, Ronald M and Taha, Abdel A and Tiulpin, Aleksei and Tsaftaris, Sotirios A and Calster, Ben Van and Varoquaux, Gaël and Yaniv, Ziv R and Jäger, Paul F and MaierandHein, Lena}, article_location = {UNITED STATES}, article_number = {2}, doi = {http://dx.doi.org/10.1038/s41592-023-02150-0}, keywords = {SEGMENTATION}, language = {eng}, issn = {1548-7091}, journal = {NATURE METHODS}, title = {Understanding metric-related pitfalls in image analysis validation}, url = {https://www.nature.com/articles/s41592-023-02150-0}, volume = {21}, year = {2024} }
TY - JOUR ID - 2453698 AU - Reinke, Annika - Tizabi, Minu D - Baumgartner, Michael - Eisenmann, Matthias - Heckmann-Nötzel, Doreen - Kavur, A Emre - Rädsch, Tim - Sudre, Carole H - Acion, Laura - Antonelli, Michela - Arbel, Tal - Bakas, Spyridon - Benis, Arriel - Buettner, Florian - Cardoso, M Jorge - Cheplygina, Veronika - Chen, Jianxu - Christodoulou, Evangelia - Cimini, Beth A - Farahani, Keyvan - Ferrer, Luciana - Galdran, Adrian - Ginneken, Bram van - Glocker, Ben - Godau, Patrick - Hashimoto, Daniel A - Hoffman, Michael M - Huisman, Merel - Isensee, Fabian - Jannin, Pierre - Kahn, Charles E - Kainmueller, Dagmar - Kainz, Bernhard - Karargyris, Alexandros - Kleesiek, Jens - Kofler, Florian - Kooi, Thijs - Kopp-Schneider, Annette - Kozubek, Michal - Kreshuk, Anna - Kurc, Tahsin - Landman, Bennett A - Litjens, Geert - Madani, Amin - Maier-Hein, Klaus - Martel, Anne L - Meijering, Erik - Menze, Bjoern - Moons, Karel GM - Müller, Henning - Nichyporuk, Brennan - Nickel, Felix - Petersen, Jens - Rafelski, Susanne M - Rajpoot, Nasir - Reyes, Mauricio - Riegler, Michael A - Rieke, Nicola - Saez-Rodriguez, Julio - Sánchez, Clara I - Shetty, Shravya - Summers, Ronald M - Taha, Abdel A - Tiulpin, Aleksei - Tsaftaris, Sotirios A - Calster, Ben Van - Varoquaux, Gaël - Yaniv, Ziv R - Jäger, Paul F - Maier-Hein, Lena PY - 2024 TI - Understanding metric-related pitfalls in image analysis validation JF - NATURE METHODS VL - 21 IS - 2 SP - 182-194 EP - 182-194 PB - NATURE PORTFOLIO SN - 15487091 KW - SEGMENTATION UR - https://www.nature.com/articles/s41592-023-02150-0 N2 - Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation. ER -
REINKE, Annika, Minu D TIZABI, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NÖTZEL, A Emre KAVUR, Tim RÄDSCH, Carole H SUDRE, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Florian BUETTNER, M Jorge CARDOSO, Veronika CHEPLYGINA, Jianxu CHEN, Evangelia CHRISTODOULOU, Beth A CIMINI, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, Bram van GINNEKEN, Ben GLOCKER, Patrick GODAU, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Fabian ISENSEE, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Jens KLEESIEK, Florian KOFLER, Thijs KOOI, Annette KOPP-SCHNEIDER, Michal KOZUBEK, Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Erik MEIJERING, Bjoern MENZE, Karel GM MOONS, Henning MÜLLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Susanne M RAFELSKI, Nasir RAJPOOT, Mauricio REYES, Michael A RIEGLER, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SÁNCHEZ, Shravya SHETTY, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Ben Van CALSTER, Ga$\backslash$''el VAROQUAUX, Ziv R YANIV, Paul F JÄGER a Lena MAIER-HEIN. Understanding metric-related pitfalls in image analysis validation. \textit{NATURE METHODS}. UNITED STATES: NATURE PORTFOLIO, 2024, roč.~21, č.~2, s.~182-194, 20 s. ISSN~1548-7091. Dostupné z: https://dx.doi.org/10.1038/s41592-023-02150-0.