EISENMANN, M., A. REINKE, V. WERU, M. D. TIZABI, F. ISENSEE, T. J. ADLER, S. ALI, V. ANDREARCZYK, M. AUBREVILLE, U. BAID, S. BAKAS, N. BALU, S. BANO, J. BERNAL, S. BODENSTEDT, A. CASELLA, V. CHEPLYGINA, M. DAUM, M. DE BRUIJNE, A. DEPEURSINGE, R. DORENT, J. EGGER, D. G. ELLIS, S. ENGELHARDT, M. GANZ, N. GHATWARY, G. GIRARD, P. GODAU, A. GUPTA, L. HANSEN, K. HARADA, M. HEINRICH, N. HELLER, A. HERING, A. HUAULME, P. JANNIN, A. E. KAVUR, O. KODYM, Michal KOZUBEK (203 Česká republika, domácí), J. LI, H. LI, J. MA, C. MARTIN-ISLA, B. MENZE, A. NOBLE, V. OREILLER, N. PADOY, S. PATI, K. PAYETTE, T. RAEDSCH, J. RAFAEL-PATINO, V. Singh BAWA, S. SPEIDEL, C. H. SUDRE, K. VAN WIJNEN, M. WAGNER, D. WEI, A. YAMLAHI, M. H. YAP, C. YUAN, M. ZENK, A. ZIA, D. ZIMMERER, D. AYDOGAN, B. BHATTARAI, L. BLOCH, R. BRUENGEL, J. CHO, C. CHOI, Q. DOU, I. EZHOV, C. M. FRIEDRICH, C. FULLER, R. R. GAIRE, A. GALDRAN, A. Garcia FAURA, M. GRAMMATIKOPOULOU, S. HONG, M. JAHANIFAR, I. JANG, A. KADKHODAMOHAMMADI, I. KANG, F. KOFLER, S. KONDO, H. KUIJF, M. LI, M. LUU, T. MARTINCIC, P. MORAIS, M. A. NASER, B. OLIVEIRA, D. OWEN, S. PANG, J. PARK, S. PARK, S. PLOTKA, E. PUYBAREAU, N. RAJPOOT, K. RYU, N. SAEED, A. SHEPHARD, P. SHI, D. STEPEC, R. SUBEDI, G. TOCHON, H. R. TORRES, H. URIEN, J. L. VILACA, K. A. WAHID, H. WANG, J. WANG, L. WANG, X. WANG, B. WIESTLER, M. WODZINSKI, F. XIA, J. XIE, Z. XIONG, S. YANG, Y. YANG, Z. ZHAO, K. MAIER-HEIN, P. F. JAEGER, A. KOPP-SCHNEIDER a L. MAIER-HEIN
Vydání
Vancouver, 2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), od s. 19955-19966, 12 s. 2023
Nakladatel
IEEE COMPUTER SOC
Další údaje
Jazyk
angličtina
Typ výsledku
Stať ve sborníku
Obor
10201 Computer sciences, information science, bioinformatics
International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.
EISENMANN, M., A. REINKE, V. WERU, M. D. TIZABI, F. ISENSEE, T. J. ADLER, S. ALI, V. ANDREARCZYK, M. AUBREVILLE, U. BAID, S. BAKAS, N. BALU, S. BANO, J. BERNAL, S. BODENSTEDT, A. CASELLA, V. CHEPLYGINA, M. DAUM, M. DE BRUIJNE, A. DEPEURSINGE, R. DORENT, J. EGGER, D. G. ELLIS, S. ENGELHARDT, M. GANZ, N. GHATWARY, G. GIRARD, P. GODAU, A. GUPTA, L. HANSEN, K. HARADA, M. HEINRICH, N. HELLER, A. HERING, A. HUAULME, P. JANNIN, A. E. KAVUR, O. KODYM, Michal KOZUBEK, J. LI, H. LI, J. MA, C. MARTIN-ISLA, B. MENZE, A. NOBLE, V. OREILLER, N. PADOY, S. PATI, K. PAYETTE, T. RAEDSCH, J. RAFAEL-PATINO, V. Singh BAWA, S. SPEIDEL, C. H. SUDRE, K. VAN WIJNEN, M. WAGNER, D. WEI, A. YAMLAHI, M. H. YAP, C. YUAN, M. ZENK, A. ZIA, D. ZIMMERER, D. AYDOGAN, B. BHATTARAI, L. BLOCH, R. BRUENGEL, J. CHO, C. CHOI, Q. DOU, I. EZHOV, C. M. FRIEDRICH, C. FULLER, R. R. GAIRE, A. GALDRAN, A. Garcia FAURA, M. GRAMMATIKOPOULOU, S. HONG, M. JAHANIFAR, I. JANG, A. KADKHODAMOHAMMADI, I. KANG, F. KOFLER, S. KONDO, H. KUIJF, M. LI, M. LUU, T. MARTINCIC, P. MORAIS, M. A. NASER, B. OLIVEIRA, D. OWEN, S. PANG, J. PARK, S. PARK, S. PLOTKA, E. PUYBAREAU, N. RAJPOOT, K. RYU, N. SAEED, A. SHEPHARD, P. SHI, D. STEPEC, R. SUBEDI, G. TOCHON, H. R. TORRES, H. URIEN, J. L. VILACA, K. A. WAHID, H. WANG, J. WANG, L. WANG, X. WANG, B. WIESTLER, M. WODZINSKI, F. XIA, J. XIE, Z. XIONG, S. YANG, Y. YANG, Z. ZHAO, K. MAIER-HEIN, P. F. JAEGER, A. KOPP-SCHNEIDER a L. MAIER-HEIN. Why is the winner the best?. Online. In 979-8-3503-0129-8. 2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR). Vancouver: IEEE COMPUTER SOC, 2023, s. 19955-19966. ISBN 979-8-3503-0129-8. Dostupné z: https://dx.doi.org/10.1109/CVPR52729.2023.01911.
@inproceedings{2392236, author = {Eisenmann, M. and Reinke, A. and Weru, V. and Tizabi, M. D. and Isensee, F. and Adler, T. J. and Ali, S. and Andrearczyk, V. and Aubreville, M. and Baid, U. and Bakas, S. and Balu, N. and Bano, S. and Bernal, J. and Bodenstedt, S. and Casella, A. and Cheplygina, V. and Daum, M. and de Bruijne, M. and Depeursinge, A. and Dorent, R. and Egger, J. and Ellis, D. G. and Engelhardt, S. and Ganz, M. and Ghatwary, N. and Girard, G. and Godau, P. and Gupta, A. and Hansen, L. and Harada, K. and Heinrich, M. and Heller, N. and Hering, A. and Huaulme, A. and Jannin, P. and Kavur, A. E. and Kodym, O. and Kozubek, Michal and Li, J. and Li, H. and Ma, J. and MartinandIsla, C. and Menze, B. and Noble, A. and Oreiller, V. and Padoy, N. and Pati, S. and Payette, K. and Raedsch, T. and RafaelandPatino, J. and Bawa, V. Singh and Speidel, S. and Sudre, C. H. and van Wijnen, K. and Wagner, M. and Wei, D. and Yamlahi, A. and Yap, M. H. and Yuan, C. and Zenk, M. and Zia, A. and Zimmerer, D. and Aydogan, D. and Bhattarai, B. and Bloch, L. and Bruengel, R. and Cho, J. and Choi, C. and Dou, Q. and Ezhov, I. and Friedrich, C. M. and Fuller, C. and Gaire, R. R. and Galdran, A. and Faura, A. Garcia and Grammatikopoulou, M. and Hong, S. and Jahanifar, M. and Jang, I. and Kadkhodamohammadi, A. and Kang, I. and Kofler, F. and Kondo, S. and Kuijf, H. and Li, M. and Luu, M. and Martincic, T. and Morais, P. and Naser, M. A. and Oliveira, B. and Owen, D. and Pang, S. and Park, J. and Park, S. and Plotka, S. and Puybareau, E. and Rajpoot, N. and Ryu, K. and Saeed, N. and Shephard, A. and Shi, P. and Stepec, D. and Subedi, R. and Tochon, G. and Torres, H. R. and Urien, H. and Vilaca, J. L. and Wahid, K. A. and Wang, H. and Wang, J. and Wang, L. and Wang, X. and Wiestler, B. and Wodzinski, M. and Xia, F. and Xie, J. and Xiong, Z. and Yang, S. and Yang, Y. and Zhao, Z. and MaierandHein, K. and Jaeger, P. F. and KoppandSchneider, A. and MaierandHein, L.}, address = {Vancouver}, booktitle = {2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)}, doi = {http://dx.doi.org/10.1109/CVPR52729.2023.01911}, editor = {979-8-3503-0129-8}, keywords = {cell microscopy; Medical and biological vision}, howpublished = {elektronická verze "online"}, language = {eng}, location = {Vancouver}, isbn = {979-8-3503-0129-8}, pages = {19955-19966}, publisher = {IEEE COMPUTER SOC}, title = {Why is the winner the best?}, year = {2023} }
TY - JOUR ID - 2392236 AU - Eisenmann, M. - Reinke, A. - Weru, V. - Tizabi, M. D. - Isensee, F. - Adler, T. J. - Ali, S. - Andrearczyk, V. - Aubreville, M. - Baid, U. - Bakas, S. - Balu, N. - Bano, S. - Bernal, J. - Bodenstedt, S. - Casella, A. - Cheplygina, V. - Daum, M. - de Bruijne, M. - Depeursinge, A. - Dorent, R. - Egger, J. - Ellis, D. G. - Engelhardt, S. - Ganz, M. - Ghatwary, N. - Girard, G. - Godau, P. - Gupta, A. - Hansen, L. - Harada, K. - Heinrich, M. - Heller, N. - Hering, A. - Huaulme, A. - Jannin, P. - Kavur, A. E. - Kodym, O. - Kozubek, Michal - Li, J. - Li, H. - Ma, J. - Martin-Isla, C. - Menze, B. - Noble, A. - Oreiller, V. - Padoy, N. - Pati, S. - Payette, K. - Raedsch, T. - Rafael-Patino, J. - Bawa, V. Singh - Speidel, S. - Sudre, C. H. - van Wijnen, K. - Wagner, M. - Wei, D. - Yamlahi, A. - Yap, M. H. - Yuan, C. - Zenk, M. - Zia, A. - Zimmerer, D. - Aydogan, D. - Bhattarai, B. - Bloch, L. - Bruengel, R. - Cho, J. - Choi, C. - Dou, Q. - Ezhov, I. - Friedrich, C. M. - Fuller, C. - Gaire, R. R. - Galdran, A. - Faura, A. Garcia - Grammatikopoulou, M. - Hong, S. - Jahanifar, M. - Jang, I. - Kadkhodamohammadi, A. - Kang, I. - Kofler, F. - Kondo, S. - Kuijf, H. - Li, M. - Luu, M. - Martincic, T. - Morais, P. - Naser, M. A. - Oliveira, B. - Owen, D. - Pang, S. - Park, J. - Park, S. - Plotka, S. - Puybareau, E. - Rajpoot, N. - Ryu, K. - Saeed, N. - Shephard, A. - Shi, P. - Stepec, D. - Subedi, R. - Tochon, G. - Torres, H. R. - Urien, H. - Vilaca, J. L. - Wahid, K. A. - Wang, H. - Wang, J. - Wang, L. - Wang, X. - Wiestler, B. - Wodzinski, M. - Xia, F. - Xie, J. - Xiong, Z. - Yang, S. - Yang, Y. - Zhao, Z. - Maier-Hein, K. - Jaeger, P. F. - Kopp-Schneider, A. - Maier-Hein, L. PY - 2023 TI - Why is the winner the best? PB - IEEE COMPUTER SOC CY - Vancouver SN - 9798350301298 KW - cell microscopy KW - Medical and biological vision N2 - International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work. ER -
EISENMANN, M., A. REINKE, V. WERU, M. D. TIZABI, F. ISENSEE, T. J. ADLER, S. ALI, V. ANDREARCZYK, M. AUBREVILLE, U. BAID, S. BAKAS, N. BALU, S. BANO, J. BERNAL, S. BODENSTEDT, A. CASELLA, V. CHEPLYGINA, M. DAUM, M. DE BRUIJNE, A. DEPEURSINGE, R. DORENT, J. EGGER, D. G. ELLIS, S. ENGELHARDT, M. GANZ, N. GHATWARY, G. GIRARD, P. GODAU, A. GUPTA, L. HANSEN, K. HARADA, M. HEINRICH, N. HELLER, A. HERING, A. HUAULME, P. JANNIN, A. E. KAVUR, O. KODYM, Michal KOZUBEK, J. LI, H. LI, J. MA, C. MARTIN-ISLA, B. MENZE, A. NOBLE, V. OREILLER, N. PADOY, S. PATI, K. PAYETTE, T. RAEDSCH, J. RAFAEL-PATINO, V. Singh BAWA, S. SPEIDEL, C. H. SUDRE, K. VAN WIJNEN, M. WAGNER, D. WEI, A. YAMLAHI, M. H. YAP, C. YUAN, M. ZENK, A. ZIA, D. ZIMMERER, D. AYDOGAN, B. BHATTARAI, L. BLOCH, R. BRUENGEL, J. CHO, C. CHOI, Q. DOU, I. EZHOV, C. M. FRIEDRICH, C. FULLER, R. R. GAIRE, A. GALDRAN, A. Garcia FAURA, M. GRAMMATIKOPOULOU, S. HONG, M. JAHANIFAR, I. JANG, A. KADKHODAMOHAMMADI, I. KANG, F. KOFLER, S. KONDO, H. KUIJF, M. LI, M. LUU, T. MARTINCIC, P. MORAIS, M. A. NASER, B. OLIVEIRA, D. OWEN, S. PANG, J. PARK, S. PARK, S. PLOTKA, E. PUYBAREAU, N. RAJPOOT, K. RYU, N. SAEED, A. SHEPHARD, P. SHI, D. STEPEC, R. SUBEDI, G. TOCHON, H. R. TORRES, H. URIEN, J. L. VILACA, K. A. WAHID, H. WANG, J. WANG, L. WANG, X. WANG, B. WIESTLER, M. WODZINSKI, F. XIA, J. XIE, Z. XIONG, S. YANG, Y. YANG, Z. ZHAO, K. MAIER-HEIN, P. F. JAEGER, A. KOPP-SCHNEIDER a L. MAIER-HEIN. Why is the winner the best?. Online. In 979-8-3503-0129-8. \textit{2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)}. Vancouver: IEEE COMPUTER SOC, 2023, s.~19955-19966. ISBN~979-8-3503-0129-8. Dostupné z: https://dx.doi.org/10.1109/CVPR52729.2023.01911.