Why rankings of biomedical image analysis competitions should
be interpreted with care

J 2018

Why rankings of biomedical image analysis competitions should be interpreted with care

MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC et. al.

Základní údaje

Originální název

Why rankings of biomedical image analysis competitions should be interpreted with care

Autoři

MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Tal ARBEL, Hrvoje BOGUNOVIC, Andrew BRADLEY, Aaron CARASS, Carolin FELDMANN, Alejandro FRANGI, Peter FULL, Bram VAN GINNEKEN, Allan HANBURY, Katrin HONAUER, Michal KOZUBEK (203 Česká republika, garant, domácí), Bennett LANDMAN, Keno MÄRZ, Oskar MAIER, Klaus MAIER-HEIN, Bjoern MENZE, Henning MÜLLER, Peter NEHER, Wiro NIESSEN, Nasir RAJPOOT, Gregory SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Christian STOCK, Danail STOYANOV, Abdel Aziz TAHA, Fons VAN DER SOMMEN, Ching-Wei WANG, Marc-André WEBER, Guoyan ZHENG, Pierre JANNIN a Annette KOPP-SCHNEIDER

Vydání

Nature Communications, Nature Publishing Group, 2018, 2041-1723

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Švýcarsko

Utajení

není předmětem státního či obchodního tajemství

Odkazy

URL

Impakt faktor

Impact factor: 11.878

Kód RIV

RIV/00216224:14330/18:00101338

Organizační jednotka

Fakulta informatiky

DOI

http://dx.doi.org/10.1038/s41467-018-07619-7

UT WoS

000452282700012

Klíčová slova anglicky

biomedical image analysis; benchmarking; challenge

Štítky

cbia-web

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 31. 12. 2018 08:54, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Návaznosti

GBP302/12/G157, projekt VaV

Název: Dynamika a organizace chromosomů během buněčného cyklu a při diferenciaci v normě a patologii

Investor: Grantová agentura ČR, Dynamika a organizace chromosomů během buněčného cyklu a při diferenciaci v normě a patologii

LTC17016, projekt VaV

Název: Benchmarking algoritmů segmentace a sledování buněk

Investor: Ministerstvo školství, mládeže a tělovýchovy ČR, Benchmarking algoritmů segmentace a sledování buněk, INTER-COST

Citovat

MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Tal ARBEL, Hrvoje BOGUNOVIC, Andrew BRADLEY, Aaron CARASS, Carolin FELDMANN, Alejandro FRANGI, Peter FULL, Bram VAN GINNEKEN, Allan HANBURY, Katrin HONAUER, Michal KOZUBEK, Bennett LANDMAN, Keno MÄRZ, Oskar MAIER, Klaus MAIER-HEIN, Bjoern MENZE, Henning MÜLLER, Peter NEHER, Wiro NIESSEN, Nasir RAJPOOT, Gregory SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Christian STOCK, Danail STOYANOV, Abdel Aziz TAHA, Fons VAN DER SOMMEN, Ching-Wei WANG, Marc-André WEBER, Guoyan ZHENG, Pierre JANNIN a Annette KOPP-SCHNEIDER. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications. Nature Publishing Group, 2018, roč. 9, č. 5217, s. 1-13. ISSN 2041-1723. Dostupné z: https://dx.doi.org/10.1038/s41467-018-07619-7.

@article{1466456,
   author = {MaierandHein, Lena and Eisenmann, Matthias and Reinke, Annika and Onogur, Sinan and Stankovic, Marko and Scholz, Patrick and Arbel, Tal and Bogunovic, Hrvoje and Bradley, Andrew and Carass, Aaron and Feldmann, Carolin and Frangi, Alejandro and Full, Peter and van Ginneken, Bram and Hanbury, Allan and Honauer, Katrin and Kozubek, Michal and Landman, Bennett and März, Keno and Maier, Oskar and MaierandHein, Klaus and Menze, Bjoern and Müller, Henning and Neher, Peter and Niessen, Wiro and Rajpoot, Nasir and Sharp, Gregory and Sirinukunwattana, Korsuk and Speidel, Stefanie and Stock, Christian and Stoyanov, Danail and Taha, Abdel Aziz and van der Sommen, Fons and Wang, ChingandWei and Weber, MarcandAndré and Zheng, Guoyan and Jannin, Pierre and KoppandSchneider, Annette},
   article_number = {5217},
   doi = {http://dx.doi.org/10.1038/s41467-018-07619-7},
   keywords = {biomedical image analysis; benchmarking; challenge},
   language = {eng},
   issn = {2041-1723},
   journal = {Nature Communications},
   title = {Why rankings of biomedical image analysis competitions should be interpreted with care},
   url = {http://doi.org/10.1038/s41467-018-07619-7},
   volume = {9},
   year = {2018}
}

TY  - JOUR
ID  - 1466456
AU  - Maier-Hein, Lena - Eisenmann, Matthias - Reinke, Annika - Onogur, Sinan - Stankovic, Marko - Scholz, Patrick - Arbel, Tal - Bogunovic, Hrvoje - Bradley, Andrew - Carass, Aaron - Feldmann, Carolin - Frangi, Alejandro - Full, Peter - van Ginneken, Bram - Hanbury, Allan - Honauer, Katrin - Kozubek, Michal - Landman, Bennett - März, Keno - Maier, Oskar - Maier-Hein, Klaus - Menze, Bjoern - Müller, Henning - Neher, Peter - Niessen, Wiro - Rajpoot, Nasir - Sharp, Gregory - Sirinukunwattana, Korsuk - Speidel, Stefanie - Stock, Christian - Stoyanov, Danail - Taha, Abdel Aziz - van der Sommen, Fons - Wang, Ching-Wei - Weber, Marc-André - Zheng, Guoyan - Jannin, Pierre - Kopp-Schneider, Annette
PY  - 2018
TI  - Why rankings of biomedical image analysis competitions should be interpreted with care
JF  - Nature Communications
VL  - 9
IS  - 5217
SP  - 1-13
EP  - 1-13
PB  - Nature Publishing Group
SN  - 20411723
KW  - biomedical image analysis
KW  - benchmarking
KW  - challenge
UR  - http://doi.org/10.1038/s41467-018-07619-7
L2  - http://doi.org/10.1038/s41467-018-07619-7
N2  - International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
ER  -

MAIER-HEIN, Lena, Matthias EISENMANN, Annika REINKE, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Tal ARBEL, Hrvoje BOGUNOVIC, Andrew BRADLEY, Aaron CARASS, Carolin FELDMANN, Alejandro FRANGI, Peter FULL, Bram VAN GINNEKEN, Allan HANBURY, Katrin HONAUER, Michal KOZUBEK, Bennett LANDMAN, Keno MÄRZ, Oskar MAIER, Klaus MAIER-HEIN, Bjoern MENZE, Henning MÜLLER, Peter NEHER, Wiro NIESSEN, Nasir RAJPOOT, Gregory SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Christian STOCK, Danail STOYANOV, Abdel Aziz TAHA, Fons VAN DER SOMMEN, Ching-Wei WANG, Marc-André WEBER, Guoyan ZHENG, Pierre JANNIN a Annette KOPP-SCHNEIDER. Why rankings of biomedical image analysis competitions should be interpreted with care. \textit{Nature Communications}. Nature Publishing Group, 2018, roč.~9, č.~5217, s.~1-13. ISSN~2041-1723. Dostupné z: https://dx.doi.org/10.1038/s41467-018-07619-7.

Podrobný výpis o publikaci