Metrics reloaded: recommendations for image analysis validation
Autoři
MAIER-HEIN, Lena, Annika REINKE, Patrick GODAU, Minu D TIZABI, Florian BUETTNER, Evangelia CHRISTODOULOU, Ben GLOCKER, Fabian ISENSEE, Jens KLEESIEK, Michal KOZUBEK (203 Česká republika, garant, domácí), Mauricio REYES, Michael A RIEGLER, Manuel WIESENFARTH, A Emre KAVUR, Carole H SUDRE, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NOETZEL, Tim RAEDSCH, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Matthew B BLASCHKO, M Jorge CARDOSO, Veronika CHEPLYGINA, Beth A CIMINI, Gary S COLLINS, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, van Ginneken BRAM, Robert HAASE, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Alan KARTHIKESALINGAM, Florian KOFLER, Annette KOPP-SCHNEIDER, Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Peter MATTSON, Erik MEIJERING, Bjoern MENZE, Karel G M MOONS, Henning MUELLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Nasir RAJPOOT, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SANCHEZ, Shravya SHETTY, van Smeden MAARTEN, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Van Calster BEN, Gael VAROQUAUX a Paul F JAEGER
Vydání
NATURE METHODS, UNITED STATES, NATURE PORTFOLIO, 2024, 1548-7091
Další údaje
Jazyk
angličtina
Typ výsledku
Článek v odborném periodiku
Obor
10201 Computer sciences, information science, bioinformatics
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint — a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Návaznosti
LM2018129, projekt VaV
Název: Národní infrastruktura pro biologické a medicínské zobrazování Czech-BioImaging
Investor: Ministerstvo školství, mládeže a tělovýchovy ČR, National research infrastructure for biological and medical imaging
MAIER-HEIN, Lena, Annika REINKE, Patrick GODAU, Minu D TIZABI, Florian BUETTNER, Evangelia CHRISTODOULOU, Ben GLOCKER, Fabian ISENSEE, Jens KLEESIEK, Michal KOZUBEK, Mauricio REYES, Michael A RIEGLER, Manuel WIESENFARTH, A Emre KAVUR, Carole H SUDRE, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NOETZEL, Tim RAEDSCH, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Matthew B BLASCHKO, M Jorge CARDOSO, Veronika CHEPLYGINA, Beth A CIMINI, Gary S COLLINS, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, van Ginneken BRAM, Robert HAASE, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Alan KARTHIKESALINGAM, Florian KOFLER, Annette KOPP-SCHNEIDER, Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Peter MATTSON, Erik MEIJERING, Bjoern MENZE, Karel G M MOONS, Henning MUELLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Nasir RAJPOOT, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SANCHEZ, Shravya SHETTY, van Smeden MAARTEN, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Van Calster BEN, Gael VAROQUAUX a Paul F JAEGER. Metrics reloaded: recommendations for image analysis validation. NATURE METHODS. UNITED STATES: NATURE PORTFOLIO, 2024, roč. 21, February, s. 195-212, 30 s. ISSN 1548-7091. Dostupné z: https://dx.doi.org/10.1038/s41592-023-02151-z.
@article{2453685, author = {MaierandHein, Lena and Reinke, Annika and Godau, Patrick and Tizabi, Minu D and Buettner, Florian and Christodoulou, Evangelia and Glocker, Ben and Isensee, Fabian and Kleesiek, Jens and Kozubek, Michal and Reyes, Mauricio and Riegler, Michael A and Wiesenfarth, Manuel and Kavur, A Emre and Sudre, Carole H and Baumgartner, Michael and Eisenmann, Matthias and HeckmannandNoetzel, Doreen and Raedsch, Tim and Acion, Laura and Antonelli, Michela and Arbel, Tal and Bakas, Spyridon and Benis, Arriel and Blaschko, Matthew B and Cardoso, M Jorge and Cheplygina, Veronika and Cimini, Beth A and Collins, Gary S and Farahani, Keyvan and Ferrer, Luciana and Galdran, Adrian and Bram, van Ginneken and Haase, Robert and Hashimoto, Daniel A and Hoffman, Michael M and Huisman, Merel and Jannin, Pierre and Kahn, Charles E and Kainmueller, Dagmar and Kainz, Bernhard and Karargyris, Alexandros and Karthikesalingam, Alan and Kofler, Florian and KoppandSchneider, Annette and Kreshuk, Anna and Kurc, Tahsin and Landman, Bennett A and Litjens, Geert and Madani, Amin and MaierandHein, Klaus and Martel, Anne L and Mattson, Peter and Meijering, Erik and Menze, Bjoern and Moons, Karel G M and Mueller, Henning and Nichyporuk, Brennan and Nickel, Felix and Petersen, Jens and Rajpoot, Nasir and Rieke, Nicola and SaezandRodriguez, Julio and Sanchez, Clara I and Shetty, Shravya and Maarten, van Smeden and Summers, Ronald M and Taha, Abdel A and Tiulpin, Aleksei and Tsaftaris, Sotirios A and Ben, Van Calster and Varoquaux, Gael and Jaeger, Paul F}, article_location = {UNITED STATES}, article_number = {February}, doi = {http://dx.doi.org/10.1038/s41592-023-02151-z}, keywords = {HEALTH; SEGMENTATION; CRITERIA}, language = {eng}, issn = {1548-7091}, journal = {NATURE METHODS}, title = {Metrics reloaded: recommendations for image analysis validation}, url = {https://www.nature.com/articles/s41592-023-02151-z}, volume = {21}, year = {2024} }
TY - JOUR ID - 2453685 AU - Maier-Hein, Lena - Reinke, Annika - Godau, Patrick - Tizabi, Minu D - Buettner, Florian - Christodoulou, Evangelia - Glocker, Ben - Isensee, Fabian - Kleesiek, Jens - Kozubek, Michal - Reyes, Mauricio - Riegler, Michael A - Wiesenfarth, Manuel - Kavur, A Emre - Sudre, Carole H - Baumgartner, Michael - Eisenmann, Matthias - Heckmann-Noetzel, Doreen - Raedsch, Tim - Acion, Laura - Antonelli, Michela - Arbel, Tal - Bakas, Spyridon - Benis, Arriel - Blaschko, Matthew B - Cardoso, M Jorge - Cheplygina, Veronika - Cimini, Beth A - Collins, Gary S - Farahani, Keyvan - Ferrer, Luciana - Galdran, Adrian - Bram, van Ginneken - Haase, Robert - Hashimoto, Daniel A - Hoffman, Michael M - Huisman, Merel - Jannin, Pierre - Kahn, Charles E - Kainmueller, Dagmar - Kainz, Bernhard - Karargyris, Alexandros - Karthikesalingam, Alan - Kofler, Florian - Kopp-Schneider, Annette - Kreshuk, Anna - Kurc, Tahsin - Landman, Bennett A - Litjens, Geert - Madani, Amin - Maier-Hein, Klaus - Martel, Anne L - Mattson, Peter - Meijering, Erik - Menze, Bjoern - Moons, Karel G M - Mueller, Henning - Nichyporuk, Brennan - Nickel, Felix - Petersen, Jens - Rajpoot, Nasir - Rieke, Nicola - Saez-Rodriguez, Julio - Sanchez, Clara I - Shetty, Shravya - Maarten, van Smeden - Summers, Ronald M - Taha, Abdel A - Tiulpin, Aleksei - Tsaftaris, Sotirios A - Ben, Van Calster - Varoquaux, Gael - Jaeger, Paul F PY - 2024 TI - Metrics reloaded: recommendations for image analysis validation JF - NATURE METHODS VL - 21 IS - February SP - 195-212 EP - 195-212 PB - NATURE PORTFOLIO SN - 15487091 KW - HEALTH KW - SEGMENTATION KW - CRITERIA UR - https://www.nature.com/articles/s41592-023-02151-z N2 - Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint — a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases. ER -
MAIER-HEIN, Lena, Annika REINKE, Patrick GODAU, Minu D TIZABI, Florian BUETTNER, Evangelia CHRISTODOULOU, Ben GLOCKER, Fabian ISENSEE, Jens KLEESIEK, Michal KOZUBEK, Mauricio REYES, Michael A RIEGLER, Manuel WIESENFARTH, A Emre KAVUR, Carole H SUDRE, Michael BAUMGARTNER, Matthias EISENMANN, Doreen HECKMANN-NOETZEL, Tim RAEDSCH, Laura ACION, Michela ANTONELLI, Tal ARBEL, Spyridon BAKAS, Arriel BENIS, Matthew B BLASCHKO, M Jorge CARDOSO, Veronika CHEPLYGINA, Beth A CIMINI, Gary S COLLINS, Keyvan FARAHANI, Luciana FERRER, Adrian GALDRAN, van Ginneken BRAM, Robert HAASE, Daniel A HASHIMOTO, Michael M HOFFMAN, Merel HUISMAN, Pierre JANNIN, Charles E KAHN, Dagmar KAINMUELLER, Bernhard KAINZ, Alexandros KARARGYRIS, Alan KARTHIKESALINGAM, Florian KOFLER, Annette KOPP-SCHNEIDER, Anna KRESHUK, Tahsin KURC, Bennett A LANDMAN, Geert LITJENS, Amin MADANI, Klaus MAIER-HEIN, Anne L MARTEL, Peter MATTSON, Erik MEIJERING, Bjoern MENZE, Karel G M MOONS, Henning MUELLER, Brennan NICHYPORUK, Felix NICKEL, Jens PETERSEN, Nasir RAJPOOT, Nicola RIEKE, Julio SAEZ-RODRIGUEZ, Clara I SANCHEZ, Shravya SHETTY, van Smeden MAARTEN, Ronald M SUMMERS, Abdel A TAHA, Aleksei TIULPIN, Sotirios A TSAFTARIS, Van Calster BEN, Gael VAROQUAUX a Paul F JAEGER. Metrics reloaded: recommendations for image analysis validation. \textit{NATURE METHODS}. UNITED STATES: NATURE PORTFOLIO, 2024, roč.~21, February, s.~195-212, 30 s. ISSN~1548-7091. Dostupné z: https://dx.doi.org/10.1038/s41592-023-02151-z.