How to Exploit Weaknesses in Biomedical Challenge Design and
Organization

D 2018

How to Exploit Weaknesses in Biomedical Challenge Design and Organization

REINKE, Annika, Matthias EISENMANN, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ et. al.

Basic information

Original name

How to Exploit Weaknesses in Biomedical Challenge Design and Organization

Authors

Edition

LNCS 11073. Cham (Switzerland), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, p. 388-395, 8 pp. 2018

Publisher

Springer

Other information

Language

English

Type of outcome

Stať ve sborníku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

Switzerland

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

printed version "print"

References:

URL

Impact factor

Impact factor: 0.402 in 2005

RIV identification code

RIV/00216224:14330/18:00104381

Organization unit

Faculty of Informatics

ISBN

978-3-030-00936-6

ISSN

DOI

http://dx.doi.org/10.1007/978-3-030-00937-3_45

UT WoS

000477769100045

Keywords in English

biomedical image analysis; benchmarking

Abstract

V originále

Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.

Citovat

REINKE, Annika, Matthias EISENMANN, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Peter M FULL, Hrvoje BOGUNOVIC, Bennett A LANDMAN, Oskar MAIER, Bjoern MENZE, Gregory C SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Fons VAN DER SOMMEN, Guoyan ZHENG, Henning MÜLLER, Michal KOZUBEK, Tal ARBEL, Andrew P BRADLEY, Pierre JANNIN, Annette KOPP-SCHNEIDER and Lena MAIER-HEIN. How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, Gabor Fichtinger. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. LNCS 11073. Cham (Switzerland): Springer, 2018, p. 388-395. ISBN 978-3-030-00936-6. Available from: https://dx.doi.org/10.1007/978-3-030-00937-3_45.

@inproceedings{1464318,
   author = {Reinke, Annika and Eisenmann, Matthias and Onogur, Sinan and Stankovic, Marko and Scholz, Patrick and Full, Peter M and Bogunovic, Hrvoje and Landman, Bennett A and Maier, Oskar and Menze, Bjoern and Sharp, Gregory C and Sirinukunwattana, Korsuk and Speidel, Stefanie and van der Sommen, Fons and Zheng, Guoyan and Müller, Henning and Kozubek, Michal and Arbel, Tal and Bradley, Andrew P and Jannin, Pierre and KoppandSchneider, Annette and MaierandHein, Lena},
   address = {Cham (Switzerland)},
   booktitle = {Medical Image Computing and Computer Assisted Intervention – MICCAI 2018},
   doi = {http://dx.doi.org/10.1007/978-3-030-00937-3_45},
   edition = {LNCS 11073},
   editor = {Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, Gabor Fichtinger},
   keywords = {biomedical image analysis; benchmarking},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {Cham (Switzerland)},
   isbn = {978-3-030-00936-6},
   pages = {388-395},
   publisher = {Springer},
   title = {How to Exploit Weaknesses in Biomedical Challenge Design and Organization},
   url = {https://doi.org/10.1007/978-3-030-00937-3_45},
   year = {2018}
}

TY  - JOUR
ID  - 1464318
AU  - Reinke, Annika - Eisenmann, Matthias - Onogur, Sinan - Stankovic, Marko - Scholz, Patrick - Full, Peter M - Bogunovic, Hrvoje - Landman, Bennett A - Maier, Oskar - Menze, Bjoern - Sharp, Gregory C - Sirinukunwattana, Korsuk - Speidel, Stefanie - van der Sommen, Fons - Zheng, Guoyan - Müller, Henning - Kozubek, Michal - Arbel, Tal - Bradley, Andrew P - Jannin, Pierre - Kopp-Schneider, Annette - Maier-Hein, Lena
PY  - 2018
TI  - How to Exploit Weaknesses in Biomedical Challenge Design and Organization
PB  - Springer
CY  - Cham (Switzerland)
SN  - 9783030009366
KW  - biomedical image analysis
KW  - benchmarking
UR  - https://doi.org/10.1007/978-3-030-00937-3_45
N2  - Since the first MICCAI grand challenge organized in 2007 in Brisbane, challenges have become an integral part of MICCAI conferences. In the meantime, challenge datasets have become widely recognized as international benchmarking datasets and thus have a great influence on the research community and individual careers. In this paper, we show several ways in which weaknesses related to current challenge design and organization can potentially be exploited. Our experimental analysis, based on MICCAI segmentation challenges organized in 2015, demonstrates that both challenge organizers and participants can potentially undertake measures to substantially tune rankings. To overcome these problems we present best practice recommendations for improving challenge design and organization.
ER  -

REINKE, Annika, Matthias EISENMANN, Sinan ONOGUR, Marko STANKOVIC, Patrick SCHOLZ, Peter M FULL, Hrvoje BOGUNOVIC, Bennett A LANDMAN, Oskar MAIER, Bjoern MENZE, Gregory C SHARP, Korsuk SIRINUKUNWATTANA, Stefanie SPEIDEL, Fons VAN DER SOMMEN, Guoyan ZHENG, Henning MÜLLER, Michal KOZUBEK, Tal ARBEL, Andrew P BRADLEY, Pierre JANNIN, Annette KOPP-SCHNEIDER and Lena MAIER-HEIN. How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, Gabor Fichtinger. \textit{Medical Image Computing and Computer Assisted Intervention – MICCAI 2018}. LNCS 11073. Cham (Switzerland): Springer, 2018, p.~388-395. ISBN~978-3-030-00936-6. Available from: https://dx.doi.org/10.1007/978-3-030-00937-3\_{}45.

Detailed Information on Publication Record