| MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Tal ARBEL, Hrvoje BOGUNOVIC, Andrew BRADLEY, Aaron CARASS, Carolin FELDMANN, Alejandro FRANGI, Peter FULL, Bram VAN GINNEKEN, Allan HANBURY, Katrin HONAUER, Michal KOZUBEK, Bennett LANDMAN, Keno MÄRZ, Oskar MAIER, Klaus MAIER-HEIN, Bjoern MENZE, Henning MÜLLER, Peter NEHER, Wiro NIESSEN, Nasir RAJPOOT, Gregory SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Christian STOCK, Danail STOYANOV, Abdel Aziz TAHA, Fons VAN DER SOMMEN, Ching-Wei WANG, Marc-André WEBER, Guoyan ZHENG, Pierre JANNIN and Annette KOPP-SCHNEIDER. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications. Nature Publishing Group, 2018, vol. 9, No 5217, p. 1-13. ISSN 2041-1723. Available from: https://dx.doi.org/10.1038/s41467-018-07619-7. |
Other formats:
BibTeX
LaTeX
RIS
@article{1466456,
author = {MaierandHein, Lena and Eisenmann, Matthias and Reinke, Annika and Onogur, Sinan and Stankovic, Marko and Scholz, Patrick and Arbel, Tal and Bogunovic, Hrvoje and Bradley, Andrew and Carass, Aaron and Feldmann, Carolin and Frangi, Alejandro and Full, Peter and van Ginneken, Bram and Hanbury, Allan and Honauer, Katrin and Kozubek, Michal and Landman, Bennett and März, Keno and Maier, Oskar and MaierandHein, Klaus and Menze, Bjoern and Müller, Henning and Neher, Peter and Niessen, Wiro and Rajpoot, Nasir and Sharp, Gregory and Sirinukunwattana, Korsuk and Speidel, Stefanie and Stock, Christian and Stoyanov, Danail and Taha, Abdel Aziz and van der Sommen, Fons and Wang, ChingandWei and Weber, MarcandAndré and Zheng, Guoyan and Jannin, Pierre and KoppandSchneider, Annette},
article_number = {5217},
doi = {http://dx.doi.org/10.1038/s41467-018-07619-7},
keywords = {biomedical image analysis; benchmarking; challenge},
language = {eng},
issn = {2041-1723},
journal = {Nature Communications},
title = {Why rankings of biomedical image analysis competitions should be interpreted with care},
url = {http://doi.org/10.1038/s41467-018-07619-7},
volume = {9},
year = {2018}
}
TY - JOUR
ID - 1466456
AU - Maier-Hein, Lena - Eisenmann, Matthias - Reinke, Annika - Onogur, Sinan - Stankovic, Marko - Scholz, Patrick - Arbel, Tal - Bogunovic, Hrvoje - Bradley, Andrew - Carass, Aaron - Feldmann, Carolin - Frangi, Alejandro - Full, Peter - van Ginneken, Bram - Hanbury, Allan - Honauer, Katrin - Kozubek, Michal - Landman, Bennett - März, Keno - Maier, Oskar - Maier-Hein, Klaus - Menze, Bjoern - Müller, Henning - Neher, Peter - Niessen, Wiro - Rajpoot, Nasir - Sharp, Gregory - Sirinukunwattana, Korsuk - Speidel, Stefanie - Stock, Christian - Stoyanov, Danail - Taha, Abdel Aziz - van der Sommen, Fons - Wang, Ching-Wei - Weber, Marc-André - Zheng, Guoyan - Jannin, Pierre - Kopp-Schneider, Annette
PY - 2018
TI - Why rankings of biomedical image analysis competitions should be interpreted with care
JF - Nature Communications
VL - 9
IS - 5217
SP - 1-13
EP - 1-13
PB - Nature Publishing Group
SN - 20411723
KW - biomedical image analysis
KW - benchmarking
KW - challenge
UR - http://doi.org/10.1038/s41467-018-07619-7
L2 - http://doi.org/10.1038/s41467-018-07619-7
N2 - International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
ER -
MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Tal ARBEL, Hrvoje BOGUNOVIC, Andrew BRADLEY, Aaron CARASS, Carolin FELDMANN, Alejandro FRANGI, Peter FULL, Bram VAN GINNEKEN, Allan HANBURY, Katrin HONAUER, Michal KOZUBEK, Bennett LANDMAN, Keno MÄRZ, Oskar MAIER, Klaus MAIER-HEIN, Bjoern MENZE, Henning MÜLLER, Peter NEHER, Wiro NIESSEN, Nasir RAJPOOT, Gregory SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Christian STOCK, Danail STOYANOV, Abdel Aziz TAHA, Fons VAN DER SOMMEN, Ching-Wei WANG, Marc-André WEBER, Guoyan ZHENG, Pierre JANNIN and Annette KOPP-SCHNEIDER. Why rankings of biomedical image analysis competitions should be interpreted with care. \textit{Nature Communications}. Nature Publishing Group, 2018, vol.~9, No~5217, p.~1-13. ISSN~2041-1723. Available from: https://dx.doi.org/10.1038/s41467-018-07619-7.
|
