On bias, variance, overfitting, gold standard and consensus in
single-particle analysis by cryo-electron microscopy

J 2022

On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy

SORZANO, Carlos, Amaya JIMÉNEZ-MORENO, David MALUENDA, Marta MARTÍNEZ, Erney RAMÍREZ-APORTELA et. al.

Základní údaje

Originální název

On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy

Autoři

SORZANO, Carlos, Amaya JIMÉNEZ-MORENO, David MALUENDA, Marta MARTÍNEZ, Erney RAMÍREZ-APORTELA, James KRIEGER, Roberto MELERO, Ana CUERVO, Javier CONESA, Jiří FILIPOVIČ (203 Česká republika, domácí), Pablo CONESA, Laura del CAÑO, Yunior FONSECA, Jorge Jiménez-de LA MORENA, Patricia LOSANA, Ruben SÁNCHEZ-GARCÍA, David STŘELÁK (203 Česká republika, garant, domácí), Estrella FERNÁNDEZ-GIMÉNEZ, Federico DE ISIDRO-GÓMEZ, David HERREROS, Jose Luis VILAS, Roberto MARABINI a Jose Maria CARAZO

Vydání

Acta Crystallographica Section D: Structural Biology, Chester, International Union of Crystallography, 2022, 2059-7983

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Velká Británie a Severní Irsko

Utajení

není předmětem státního či obchodního tajemství

Odkazy

URL

Impakt faktor

Impact factor: 2.200

Kód RIV

RIV/00216224:14330/22:00125553

Organizační jednotka

Fakulta informatiky

DOI

http://dx.doi.org/10.1107/S2059798322001978

UT WoS

000777860500002

Klíčová slova anglicky

single-particle analysis; cryo-electron microscopy; parameter estimation; image processing; bias; variance; overfitting; gold standard

Štítky

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 28. 3. 2023 10:10, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Cryo-electron microscopy (cryoEM) has become a well established technique to elucidate the 3D structures of biological macromolecules. Projection images from thousands of macromolecules that are assumed to be structurally identical are combined into a single 3D map representing the Coulomb potential of the macromolecule under study. This article discusses possible caveats along the image-processing path and how to avoid them to obtain a reliable 3D structure. Some of these problems are very well known in the community. These may be referred to as sample-related (such as specimen denaturation at interfaces or non-uniform projection geometry leading to underrepresented projection directions). The rest are related to the algorithms used. While some have been discussed in depth in the literature, such as the use of an incorrect initial volume, others have received much less attention. However, they are fundamental in any data-analysis approach. Chiefly among them, instabilities in estimating many of the key parameters that are required for a correct 3D reconstruction that occur all along the processing workflow are referred to, which may significantly affect the reliability of the whole process. In the field, the term overfitting has been coined to refer to some particular kinds of artifacts. It is argued that overfitting is a statistical bias in key parameter-estimation steps in the 3D reconstruction process, including intrinsic algorithmic bias. It is also shown that common tools (Fourier shell correlation) and strategies (gold standard) that are normally used to detect or prevent overfitting do not fully protect against it. Alternatively, it is proposed that detecting the bias that leads to overfitting is much easier when addressed at the level of parameter estimation, rather than detecting it once the particle images have been combined into a 3D map. Comparing the results from multiple algorithms (or at least, independent executions of the same algorithm) can detect parameter bias. These multiple executions could then be averaged to give a lower variance estimate of the underlying parameters.

Návaznosti

EF16_013/0001802, projekt VaV

Název: CERIT Scientific Cloud

MUNI/A/1145/2021, interní kód MU

Název: Rozsáhlé výpočetní systémy: modely, aplikace a verifikace XI. (Akronym: SV-FI MAV XI.)

Investor: Masarykova univerzita, Rozsáhlé výpočetní systémy: modely, aplikace a verifikace XI.

Citovat

SORZANO, Carlos, Amaya JIMÉNEZ-MORENO, David MALUENDA, Marta MARTÍNEZ, Erney RAMÍREZ-APORTELA, James KRIEGER, Roberto MELERO, Ana CUERVO, Javier CONESA, Jiří FILIPOVIČ, Pablo CONESA, Laura del CAÑO, Yunior FONSECA, Jorge Jiménez-de LA MORENA, Patricia LOSANA, Ruben SÁNCHEZ-GARCÍA, David STŘELÁK, Estrella FERNÁNDEZ-GIMÉNEZ, Federico DE ISIDRO-GÓMEZ, David HERREROS, Jose Luis VILAS, Roberto MARABINI a Jose Maria CARAZO. On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy. Acta Crystallographica Section D: Structural Biology. Chester: International Union of Crystallography, 2022, roč. 78, č. 4, s. 410-423. ISSN 2059-7983. Dostupné z: https://dx.doi.org/10.1107/S2059798322001978.

@article{1842337,
   author = {Sorzano, Carlos and JiménezandMoreno, Amaya and Maluenda, David and Martínez, Marta and RamírezandAportela, Erney and Krieger, James and Melero, Roberto and Cuervo, Ana and Conesa, Javier and Filipovič, Jiří and Conesa, Pablo and Caño, Laura del and Fonseca, Yunior and la Morena, Jorge Jiménezandde and Losana, Patricia and SánchezandGarcía, Ruben and Střelák, David and FernándezandGiménez, Estrella and de IsidroandGómez, Federico and Herreros, David and Vilas, Jose Luis and Marabini, Roberto and Carazo, Jose Maria},
   article_location = {Chester},
   article_number = {4},
   doi = {http://dx.doi.org/10.1107/S2059798322001978},
   keywords = {single-particle analysis; cryo-electron microscopy; parameter estimation; image processing; bias; variance; overfitting; gold standard},
   language = {eng},
   issn = {2059-7983},
   journal = {Acta Crystallographica Section D: Structural Biology},
   title = {On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy},
   url = {https://doi.org/10.1107/S2059798322001978},
   volume = {78},
   year = {2022}
}

TY  - JOUR
ID  - 1842337
AU  - Sorzano, Carlos - Jiménez-Moreno, Amaya - Maluenda, David - Martínez, Marta - Ramírez-Aportela, Erney - Krieger, James - Melero, Roberto - Cuervo, Ana - Conesa, Javier - Filipovič, Jiří - Conesa, Pablo - Caño, Laura del - Fonseca, Yunior - la Morena, Jorge Jiménez-de - Losana, Patricia - Sánchez-García, Ruben - Střelák, David - Fernández-Giménez, Estrella - de Isidro-Gómez, Federico - Herreros, David - Vilas, Jose Luis - Marabini, Roberto - Carazo, Jose Maria
PY  - 2022
TI  - On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy
JF  - Acta Crystallographica Section D: Structural Biology
VL  - 78
IS  - 4
SP  - 410-423
EP  - 410-423
PB  - International Union of Crystallography
SN  - 20597983
KW  - single-particle analysis
KW  - cryo-electron microscopy
KW  - parameter estimation
KW  - image processing
KW  - bias
KW  - variance
KW  - overfitting
KW  - gold standard
UR  - https://doi.org/10.1107/S2059798322001978
N2  - Cryo-electron microscopy (cryoEM) has become a well established technique to elucidate the 3D structures of biological macromolecules. Projection images from thousands of macromolecules that are assumed to be structurally identical are combined into a single 3D map representing the Coulomb potential of the macromolecule under study. This article discusses possible caveats along the image-processing path and how to avoid them to obtain a reliable 3D structure. Some of these problems are very well known in the community. These may be referred to as sample-related (such as specimen denaturation at interfaces or non-uniform projection geometry leading to underrepresented projection directions). The rest are related to the algorithms used. While some have been discussed in depth in the literature, such as the use of an incorrect initial volume, others have received much less attention. However, they are fundamental in any data-analysis approach. Chiefly among them, instabilities in estimating many of the key parameters that are required for a correct 3D reconstruction that occur all along the processing workflow are referred to, which may significantly affect the reliability of the whole process. In the field, the term overfitting has been coined to refer to some particular kinds of artifacts. It is argued that overfitting is a statistical bias in key parameter-estimation steps in the 3D reconstruction process, including intrinsic algorithmic bias. It is also shown that common tools (Fourier shell correlation) and strategies (gold standard) that are normally used to detect or prevent overfitting do not fully protect against it. Alternatively, it is proposed that detecting the bias that leads to overfitting is much easier when addressed at the level of parameter estimation, rather than detecting it once the particle images have been combined into a 3D map. Comparing the results from multiple algorithms (or at least, independent executions of the same algorithm) can detect parameter bias. These multiple executions could then be averaged to give a lower variance estimate of the underlying parameters.
ER  -

SORZANO, Carlos, Amaya JIMÉNEZ-MORENO, David MALUENDA, Marta MARTÍNEZ, Erney RAMÍREZ-APORTELA, James KRIEGER, Roberto MELERO, Ana CUERVO, Javier CONESA, Jiří FILIPOVIČ, Pablo CONESA, Laura del CAÑO, Yunior FONSECA, Jorge Jiménez-de LA MORENA, Patricia LOSANA, Ruben SÁNCHEZ-GARCÍA, David STŘELÁK, Estrella FERNÁNDEZ-GIMÉNEZ, Federico DE ISIDRO-GÓMEZ, David HERREROS, Jose Luis VILAS, Roberto MARABINI a Jose Maria CARAZO. On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy. \textit{Acta Crystallographica Section D: Structural Biology}. Chester: International Union of Crystallography, 2022, roč.~78, č.~4, s.~410-423. ISSN~2059-7983. Dostupné z: https://dx.doi.org/10.1107/S2059798322001978.

Podrobný výpis o publikaci