evaluation of the statistics-based ames mutagenicity model
TRANSCRIPT
![Page 1: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/1.jpg)
Evaluation Of The Statistics-Based Ames Mutagenicity Model Sarah Nexus And Interpretation Of The Results Obtained
Alex Cayley [email protected]
![Page 2: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/2.jpg)
Summary
• What is a “statistics-based Ames mutagenicity model” and
why are they important?
• How can we judge how useful these models will be?
(validation statistics, expert interpretation)
• How well does Sarah Nexus (SX) version 1.1 predict and
how can results from the program be interpreted to
increase performance?
• Worked examples of SX version 1.1 predictions
![Page 3: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/3.jpg)
What Are Statistical Models?
• In our sphere no real clear-cut definition but…
• Data for a given endpoint is fed in to the model builder (binary or continuous)
• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values
• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention
• Definition is important for ICH M7 guidance compliance
• May not be as important elsewhere and the distinction is
blurring…
![Page 4: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/4.jpg)
What Are Statistical Models?
• In our sphere no real clear-cut definition but…
• Data for a given endpoint is fed in to the model builder (binary or continuous)
• An algorithm based on one or multiple descriptors is used to distinguish compound categories or predict values
• Any patterns found in the data are the result of statistical relationships and are machine learnt and NOT human intervention
• Definition is important for ICH M7 guidance compliance
• May not be as important elsewhere and the distinction is
blurring…
![Page 5: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/5.jpg)
ICH M7 Guidance
![Page 6: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/6.jpg)
What Makes A Good Statistical Model?
![Page 7: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/7.jpg)
7
“right” = Every time Explanation = Full and Reasoned
“right” = Sometimes Explanation = None
Super Expert Scientist
What Makes A Good Statistical Model?
![Page 8: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/8.jpg)
8
“right” = Every time Explanation = Full and Reasoned
“right” = Sometimes Explanation = None
Super Expert Scientist
What Makes A Good Statistical Model?
“right” = Most Times Explanation = Full and Reasoned
“right” = Most Times Explanation = None
“right”= Sometimes Explanation = Some
Expert Scientist
![Page 9: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/9.jpg)
What Makes A Good Statistical Model
QSAR Model
Training Data Validation Expert
Interpretation
1
Pharma A2
Pharma B
3
Pharma C4
Pharma D
Test SetPerformance StatsBA72637275
xx
SEN68386465
xx
SPEC76898085
xx14 Pharma N
![Page 10: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/10.jpg)
Validation of (Q)SAR Models In The Literature
![Page 11: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/11.jpg)
Validation Of Sarah Nexus (v1.1)
Specificity = 69-91% (83% mean) Sensitivity = 38-68% (55% mean) Positive <50
Incorrectly assign 3-4 = ~10%
Balanced Accuracy = Sens + Spec
2 = 62-77% (69% mean)
![Page 12: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/12.jpg)
Prediction Scenarios in Sarah Nexus
Overall Prediction +
Confidence
Hypotheses +
Confidence Positive
Negative
Overruled
![Page 13: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/13.jpg)
Negative Predictions
![Page 14: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/14.jpg)
Negative Predictions
![Page 15: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/15.jpg)
Positive Predictions
![Page 16: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/16.jpg)
Positive Predictions
![Page 17: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/17.jpg)
Confidence Correlation With Predictivity
![Page 18: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/18.jpg)
Equivocal Predictions
![Page 19: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/19.jpg)
Out Of Domain Predictions
![Page 20: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/20.jpg)
An Update
Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM1 879 279 600 77 76 80 74 60 89 188 361 126 46 82 136 22 1212 513 97 416 72 76 64 80 44 90 51 253 65 29 78 84 31 03 4018 576 3442 69 78 56 82 33 92 254 2262 507 198 80 524 272 3134 2862 170 2692 67 83 48 85 17 96 56 1654 283 61 72 621 187 275 4040 725 3315 68 80 50 86 44 89 282 2205 361 278 77 590 343 389
Mean 71 60 81 78
Sarah Nexus V2.0.1
Sarah Nexus V1.2 Data Set ID SIZE POS NEG BAC ACC SEN SPEC PPV NPV TP TN FP FN COV EQ OOD EM
1 879 279 600 72 74 68 76 55 85 143 372 115 67 79 136 46 512 513 97 416 74 81 62 86 48 91 44 284 48 27 79 76 34 03 4018 576 3442 67 78 51 82 32 91 231 2261 488 224 80 498 316 1584 2862 170 2692 67 83 48 85 17 96 57 1661 283 61 72 646 192 285 4040 725 3315 62 81 33 90 42 87 186 2444 260 371 81 430 374 243
Mean 68 53 84 78
![Page 21: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/21.jpg)
Conclusions
• Proprietary data sets can give a good indication of the
performance of statistical prediction systems
• Sarah Nexus performs well when tested against a
number of different proprietary validation sets
• Additional information provided for each prediction is also
important in aiding the user to make a final decision
• Negative predictions based purely on negative hypotheses are more reliable
• Positive predictions with a higher confidence are more reliable
Barber et al.; Reg. Tox. Pharm.; 76; 7-20 (2016) http://www.sciencedirect.com/science/article/pii/S0273230015301410
![Page 22: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/22.jpg)
Acknowledgements
• Sandy Weiner
• Joerg Wichard
• Amanda Giddings
• Susanne Glowienke
• Alexis Parenty
• Alessandro Briggo
• Hans-Peter Spirkl
• Alexander Amberg
• Ray Kemper
• Nigel Greene
• Chris Barber
• Thierry Hanser
• Alex Harding
• Crina Heghes
• Jonathan Vessey
• Stephane Werner
![Page 23: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/23.jpg)
Questions?
![Page 24: Evaluation Of The Statistics-Based Ames Mutagenicity Model](https://reader030.vdocuments.net/reader030/viewer/2022012609/619cad648556c00d3a0b2540/html5/thumbnails/24.jpg)
Work in progress disclaimer
25
This document is intended to outline our general product direction and is for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon. The development, release, and timing of any features or functionality described for Lhasa Limited’s products remains at the sole discretion of Lhasa Limited.