Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
Autoři
SHERRATT, Katharine (garant), Hugo GRUSON, Rok GRAH, Helen JOHNSON, Rene NIEHUS, Bastian PRASSE, Frank SANDMANN, Jannik DEUSCHEL, Daniel WOLFFRAM, Sam ABBOTT, Alexander ULLRICH, Graham GIBSON, Evan L RAY, Nicholas G REICH, Daniel SHELDON, Yijin WANG, Nutcha WATTANACHIT, Lijing WANG, Jan TRNKA, Guillaume OBOZINSKI, Tao SUN, Dorina THANOU, Loic POTTIER, Ekaterina KRYMOVA, Jan H MEINKE, Maria Vittoria BARBAROSSA, Neele LEITHÄUSER, Jan MOHRING, Johanna SCHNEIDER, Jaroslaw WŁAZŁO, Jan FUHRMANN, Berit LANGE, Isti RODIAH, Prasith BACCAM, Heidi GURUNG, Steven STAGE, Bradley SUCHOSKI, Jozef BUDZINSKI, Robert WALRAVEN, Inmaculada VILLANUEVA, Vit TUCEK, Martin SMID, Milan ZAJÍČEK, Cesar Pérez ÁLVAREZ, Borja REINA, Nikos I BOSSE, Sophie R MEAKIN, Lauren CASTRO, Geoffrey FAIRCHILD, Isaac MICHAUD, Dave OSTHUS, Pierfrancesco Alaimo Di LORO, Antonello MARUOTTI, Veronika ECLEROVÁ (203 Česká republika, domácí), Andrea KRAUS (703 Slovensko, domácí), David KRAUS (203 Česká republika, domácí), Lenka PŘIBYLOVÁ (203 Česká republika, domácí), Bertsimas DIMITRIS, Michael Lingzhi LI, Soni SAKSHAM, Jonas DEHNING, Sebastian MOHR, Viola PRIESEMANN, Grzegorz REDLARSKI, Benjamin BEJAR, Giovanni ARDENGHI, Nicola PAROLINI, Giovanni ZIARELLI, Wolfgang BOCK, Stefan HEYDER, Thomas HOTZ, David E SINGH, Miguel GUZMAN-MERINO, Jose L AZNARTE, David MORIÑA, Sergio ALONSO, Enric ÁLVAREZ, Daniel LÓPEZ, Clara PRATS, Jan Pablo BURGARD, Arne RODLOFF, Tom ZIMMERMANN, Alexander KUHLMANN, Janez ZIBERT, Fulvia PENNONI, Fabio DIVINO, Marti CATALÀ, Gianfranco LOVISON, Paolo GIUDICI, Barbara TARANTINO, Francesco BARTOLUCCI, Giovanna Jona LASINIO, Marco MINGIONE, Alessio FARCOMENI, Ajitesh SRIVASTAVA, Pablo MONTERO-MANSO, Aniruddha ADIGA, Benjamin HURT, Bryan LEWIS, Madhav MARATHE, Przemyslaw POREBSKI, Srinivasan VENKATRAMANAN, Rafal P BARTCZUK, Filip DREGER, Anna GAMBIN, Krzysztof GOGOLEWSKI, Magdalena GRUZIEL-SŁOMKA, Bartosz KRUPA, Antoni MOSZYŃSKI, Karol NIEDZIELEWSKI, Jedrzej NOWOSIELSKI, Maciej RADWAN, Franciszek RAKOWSKI, Marcin SEMENIUK, Ewa SZCZUREK, Jakub ZIELIŃSKI, Jan KISIELEWSKI, Barbara PABJAN, Holger KIRSTEN, Yuri KHEIFETZ, Markus SCHOLZ, Przemyslaw BIECEK, Marcin BODYCH, Maciej FILINSKI, Radoslaw IDZIKOWSKI, Tyll KRUEGER, Tomasz OZANSKI, Johannes BRACHER a Sebastian FUNK
Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts’ predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022. Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance. Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models.
Návaznosti
MUNI/A/1615/2020, interní kód MU
Název: Matematické a statistické modelování 5 (Akronym: MaStaMo5)
Investor: Masarykova univerzita, Matematické a statistické modelování 5
MUNI/11/02202001/2020, interní kód MU
Název: Online platforma pro monitoring, analýzu a management epidemických situací v reálném čase
Investor: Masarykova univerzita, Online platforma pro monitoring, analýzu a management epidemických situací v reálném čase
SHERRATT, Katharine, Hugo GRUSON, Rok GRAH, Helen JOHNSON, Rene NIEHUS, Bastian PRASSE, Frank SANDMANN, Jannik DEUSCHEL, Daniel WOLFFRAM, Sam ABBOTT, Alexander ULLRICH, Graham GIBSON, Evan L RAY, Nicholas G REICH, Daniel SHELDON, Yijin WANG, Nutcha WATTANACHIT, Lijing WANG, Jan TRNKA, Guillaume OBOZINSKI, Tao SUN, Dorina THANOU, Loic POTTIER, Ekaterina KRYMOVA, Jan H MEINKE, Maria Vittoria BARBAROSSA, Neele LEITHÄUSER, Jan MOHRING, Johanna SCHNEIDER, Jaroslaw WŁAZŁO, Jan FUHRMANN, Berit LANGE, Isti RODIAH, Prasith BACCAM, Heidi GURUNG, Steven STAGE, Bradley SUCHOSKI, Jozef BUDZINSKI, Robert WALRAVEN, Inmaculada VILLANUEVA, Vit TUCEK, Martin SMID, Milan ZAJÍČEK, Cesar Pérez ÁLVAREZ, Borja REINA, Nikos I BOSSE, Sophie R MEAKIN, Lauren CASTRO, Geoffrey FAIRCHILD, Isaac MICHAUD, Dave OSTHUS, Pierfrancesco Alaimo Di LORO, Antonello MARUOTTI, Veronika ECLEROVÁ, Andrea KRAUS, David KRAUS, Lenka PŘIBYLOVÁ, Bertsimas DIMITRIS, Michael Lingzhi LI, Soni SAKSHAM, Jonas DEHNING, Sebastian MOHR, Viola PRIESEMANN, Grzegorz REDLARSKI, Benjamin BEJAR, Giovanni ARDENGHI, Nicola PAROLINI, Giovanni ZIARELLI, Wolfgang BOCK, Stefan HEYDER, Thomas HOTZ, David E SINGH, Miguel GUZMAN-MERINO, Jose L AZNARTE, David MORIÑA, Sergio ALONSO, Enric ÁLVAREZ, Daniel LÓPEZ, Clara PRATS, Jan Pablo BURGARD, Arne RODLOFF, Tom ZIMMERMANN, Alexander KUHLMANN, Janez ZIBERT, Fulvia PENNONI, Fabio DIVINO, Marti CATALÀ, Gianfranco LOVISON, Paolo GIUDICI, Barbara TARANTINO, Francesco BARTOLUCCI, Giovanna Jona LASINIO, Marco MINGIONE, Alessio FARCOMENI, Ajitesh SRIVASTAVA, Pablo MONTERO-MANSO, Aniruddha ADIGA, Benjamin HURT, Bryan LEWIS, Madhav MARATHE, Przemyslaw POREBSKI, Srinivasan VENKATRAMANAN, Rafal P BARTCZUK, Filip DREGER, Anna GAMBIN, Krzysztof GOGOLEWSKI, Magdalena GRUZIEL-SŁOMKA, Bartosz KRUPA, Antoni MOSZYŃSKI, Karol NIEDZIELEWSKI, Jedrzej NOWOSIELSKI, Maciej RADWAN, Franciszek RAKOWSKI, Marcin SEMENIUK, Ewa SZCZUREK, Jakub ZIELIŃSKI, Jan KISIELEWSKI, Barbara PABJAN, Holger KIRSTEN, Yuri KHEIFETZ, Markus SCHOLZ, Przemyslaw BIECEK, Marcin BODYCH, Maciej FILINSKI, Radoslaw IDZIKOWSKI, Tyll KRUEGER, Tomasz OZANSKI, Johannes BRACHER a Sebastian FUNK. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations. eLife. eLife Sciences Publications Ltd, 2023, roč. 12, April, s. 1-19. ISSN 2050-084X. Dostupné z: https://dx.doi.org/10.7554/eLife.81916.
@article{2277717, author = {Sherratt, Katharine and Gruson, Hugo and Grah, Rok and Johnson, Helen and Niehus, Rene and Prasse, Bastian and Sandmann, Frank and Deuschel, Jannik and Wolffram, Daniel and Abbott, Sam and Ullrich, Alexander and Gibson, Graham and Ray, Evan L and Reich, Nicholas G and Sheldon, Daniel and Wang, Yijin and Wattanachit, Nutcha and Wang, Lijing and Trnka, Jan and Obozinski, Guillaume and Sun, Tao and Thanou, Dorina and Pottier, Loic and Krymova, Ekaterina and Meinke, Jan H and Barbarossa, Maria Vittoria and Leithäuser, Neele and Mohring, Jan and Schneider, Johanna and Włazło, Jaroslaw and Fuhrmann, Jan and Lange, Berit and Rodiah, Isti and Baccam, Prasith and Gurung, Heidi and Stage, Steven and Suchoski, Bradley and Budzinski, Jozef and Walraven, Robert and Villanueva, Inmaculada and Tucek, Vit and Smid, Martin and Zajíček, Milan and Álvarez, Cesar Pérez and Reina, Borja and Bosse, Nikos I and Meakin, Sophie R and Castro, Lauren and Fairchild, Geoffrey and Michaud, Isaac and Osthus, Dave and Loro, Pierfrancesco Alaimo Di and Maruotti, Antonello and Eclerová, Veronika and Kraus, Andrea and Kraus, David and Přibylová, Lenka and Dimitris, Bertsimas and Li, Michael Lingzhi and Saksham, Soni and Dehning, Jonas and Mohr, Sebastian and Priesemann, Viola and Redlarski, Grzegorz and Bejar, Benjamin and Ardenghi, Giovanni and Parolini, Nicola and Ziarelli, Giovanni and Bock, Wolfgang and Heyder, Stefan and Hotz, Thomas and Singh, David E and GuzmanandMerino, Miguel and Aznarte, Jose L and Moriña, David and Alonso, Sergio and Álvarez, Enric and López, Daniel and Prats, Clara and Burgard, Jan Pablo and Rodloff, Arne and Zimmermann, Tom and Kuhlmann, Alexander and Zibert, Janez and Pennoni, Fulvia and Divino, Fabio and Català, Marti and Lovison, Gianfranco and Giudici, Paolo and Tarantino, Barbara and Bartolucci, Francesco and Lasinio, Giovanna Jona and Mingione, Marco and Farcomeni, Alessio and Srivastava, Ajitesh and MonteroandManso, Pablo and Adiga, Aniruddha and Hurt, Benjamin and Lewis, Bryan and Marathe, Madhav and Porebski, Przemyslaw and Venkatramanan, Srinivasan and Bartczuk, Rafal P and Dreger, Filip and Gambin, Anna and Gogolewski, Krzysztof and GruzielandSłomka, Magdalena and Krupa, Bartosz and Moszyński, Antoni and Niedzielewski, Karol and Nowosielski, Jedrzej and Radwan, Maciej and Rakowski, Franciszek and Semeniuk, Marcin and Szczurek, Ewa and Zieliński, Jakub and Kisielewski, Jan and Pabjan, Barbara and Kirsten, Holger and Kheifetz, Yuri and Scholz, Markus and Biecek, Przemyslaw and Bodych, Marcin and Filinski, Maciej and Idzikowski, Radoslaw and Krueger, Tyll and Ozanski, Tomasz and Bracher, Johannes and Funk, Sebastian}, article_number = {April}, doi = {http://dx.doi.org/10.7554/eLife.81916}, keywords = {modelling; forecast; COVID-19; Europe; ensemble; prediction}, language = {eng}, issn = {2050-084X}, journal = {eLife}, title = {Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations}, url = {https://doi.org/10.7554/eLife.81916}, volume = {12}, year = {2023} }
TY - JOUR ID - 2277717 AU - Sherratt, Katharine - Gruson, Hugo - Grah, Rok - Johnson, Helen - Niehus, Rene - Prasse, Bastian - Sandmann, Frank - Deuschel, Jannik - Wolffram, Daniel - Abbott, Sam - Ullrich, Alexander - Gibson, Graham - Ray, Evan L - Reich, Nicholas G - Sheldon, Daniel - Wang, Yijin - Wattanachit, Nutcha - Wang, Lijing - Trnka, Jan - Obozinski, Guillaume - Sun, Tao - Thanou, Dorina - Pottier, Loic - Krymova, Ekaterina - Meinke, Jan H - Barbarossa, Maria Vittoria - Leithäuser, Neele - Mohring, Jan - Schneider, Johanna - Włazło, Jaroslaw - Fuhrmann, Jan - Lange, Berit - Rodiah, Isti - Baccam, Prasith - Gurung, Heidi - Stage, Steven - Suchoski, Bradley - Budzinski, Jozef - Walraven, Robert - Villanueva, Inmaculada - Tucek, Vit - Smid, Martin - Zajíček, Milan - Álvarez, Cesar Pérez - Reina, Borja - Bosse, Nikos I - Meakin, Sophie R - Castro, Lauren - Fairchild, Geoffrey - Michaud, Isaac - Osthus, Dave - Loro, Pierfrancesco Alaimo Di - Maruotti, Antonello - Eclerová, Veronika - Kraus, Andrea - Kraus, David - Přibylová, Lenka - Dimitris, Bertsimas - Li, Michael Lingzhi - Saksham, Soni - Dehning, Jonas - Mohr, Sebastian - Priesemann, Viola - Redlarski, Grzegorz - Bejar, Benjamin - Ardenghi, Giovanni - Parolini, Nicola - Ziarelli, Giovanni - Bock, Wolfgang - Heyder, Stefan - Hotz, Thomas - Singh, David E - Guzman-Merino, Miguel - Aznarte, Jose L - Moriña, David - Alonso, Sergio - Álvarez, Enric - López, Daniel - Prats, Clara - Burgard, Jan Pablo - Rodloff, Arne - Zimmermann, Tom - Kuhlmann, Alexander - Zibert, Janez - Pennoni, Fulvia - Divino, Fabio - Català, Marti - Lovison, Gianfranco - Giudici, Paolo - Tarantino, Barbara - Bartolucci, Francesco - Lasinio, Giovanna Jona - Mingione, Marco - Farcomeni, Alessio - Srivastava, Ajitesh - Montero-Manso, Pablo - Adiga, Aniruddha - Hurt, Benjamin - Lewis, Bryan - Marathe, Madhav - Porebski, Przemyslaw - Venkatramanan, Srinivasan - Bartczuk, Rafal P - Dreger, Filip - Gambin, Anna - Gogolewski, Krzysztof - Gruziel-Słomka, Magdalena - Krupa, Bartosz - Moszyński, Antoni - Niedzielewski, Karol - Nowosielski, Jedrzej - Radwan, Maciej - Rakowski, Franciszek - Semeniuk, Marcin - Szczurek, Ewa - Zieliński, Jakub - Kisielewski, Jan - Pabjan, Barbara - Kirsten, Holger - Kheifetz, Yuri - Scholz, Markus - Biecek, Przemyslaw - Bodych, Marcin - Filinski, Maciej - Idzikowski, Radoslaw - Krueger, Tyll - Ozanski, Tomasz - Bracher, Johannes - Funk, Sebastian PY - 2023 TI - Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations JF - eLife VL - 12 IS - April SP - 1-19 EP - 1-19 PB - eLife Sciences Publications Ltd SN - 2050084X KW - modelling KW - forecast KW - COVID-19 KW - Europe KW - ensemble KW - prediction UR - https://doi.org/10.7554/eLife.81916 N2 - Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts’ predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022. Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance. Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models. ER -
SHERRATT, Katharine, Hugo GRUSON, Rok GRAH, Helen JOHNSON, Rene NIEHUS, Bastian PRASSE, Frank SANDMANN, Jannik DEUSCHEL, Daniel WOLFFRAM, Sam ABBOTT, Alexander ULLRICH, Graham GIBSON, Evan L RAY, Nicholas G REICH, Daniel SHELDON, Yijin WANG, Nutcha WATTANACHIT, Lijing WANG, Jan TRNKA, Guillaume OBOZINSKI, Tao SUN, Dorina THANOU, Loic POTTIER, Ekaterina KRYMOVA, Jan H MEINKE, Maria Vittoria BARBAROSSA, Neele LEITHÄUSER, Jan MOHRING, Johanna SCHNEIDER, Jaroslaw WŁAZŁO, Jan FUHRMANN, Berit LANGE, Isti RODIAH, Prasith BACCAM, Heidi GURUNG, Steven STAGE, Bradley SUCHOSKI, Jozef BUDZINSKI, Robert WALRAVEN, Inmaculada VILLANUEVA, Vit TUCEK, Martin SMID, Milan ZAJÍČEK, Cesar Pérez ÁLVAREZ, Borja REINA, Nikos I BOSSE, Sophie R MEAKIN, Lauren CASTRO, Geoffrey FAIRCHILD, Isaac MICHAUD, Dave OSTHUS, Pierfrancesco Alaimo Di LORO, Antonello MARUOTTI, Veronika ECLEROVÁ, Andrea KRAUS, David KRAUS, Lenka PŘIBYLOVÁ, Bertsimas DIMITRIS, Michael Lingzhi LI, Soni SAKSHAM, Jonas DEHNING, Sebastian MOHR, Viola PRIESEMANN, Grzegorz REDLARSKI, Benjamin BEJAR, Giovanni ARDENGHI, Nicola PAROLINI, Giovanni ZIARELLI, Wolfgang BOCK, Stefan HEYDER, Thomas HOTZ, David E SINGH, Miguel GUZMAN-MERINO, Jose L AZNARTE, David MORIÑA, Sergio ALONSO, Enric ÁLVAREZ, Daniel LÓPEZ, Clara PRATS, Jan Pablo BURGARD, Arne RODLOFF, Tom ZIMMERMANN, Alexander KUHLMANN, Janez ZIBERT, Fulvia PENNONI, Fabio DIVINO, Marti CATALÀ, Gianfranco LOVISON, Paolo GIUDICI, Barbara TARANTINO, Francesco BARTOLUCCI, Giovanna Jona LASINIO, Marco MINGIONE, Alessio FARCOMENI, Ajitesh SRIVASTAVA, Pablo MONTERO-MANSO, Aniruddha ADIGA, Benjamin HURT, Bryan LEWIS, Madhav MARATHE, Przemyslaw POREBSKI, Srinivasan VENKATRAMANAN, Rafal P BARTCZUK, Filip DREGER, Anna GAMBIN, Krzysztof GOGOLEWSKI, Magdalena GRUZIEL-SŁOMKA, Bartosz KRUPA, Antoni MOSZY$\backslash$'NSKI, Karol NIEDZIELEWSKI, Jedrzej NOWOSIELSKI, Maciej RADWAN, Franciszek RAKOWSKI, Marcin SEMENIUK, Ewa SZCZUREK, Jakub ZIELI$\backslash$'NSKI, Jan KISIELEWSKI, Barbara PABJAN, Holger KIRSTEN, Yuri KHEIFETZ, Markus SCHOLZ, Przemyslaw BIECEK, Marcin BODYCH, Maciej FILINSKI, Radoslaw IDZIKOWSKI, Tyll KRUEGER, Tomasz OZANSKI, Johannes BRACHER a Sebastian FUNK. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations. \textit{eLife}. eLife Sciences Publications Ltd, 2023, roč.~12, April, s.~1-19. ISSN~2050-084X. Dostupné z: https://dx.doi.org/10.7554/eLife.81916.