caster chemical substructure representation - arxiv
TRANSCRIPT
CASTER: Predicting Drug Interactions withChemical Substructure Representation
Kexin Huang,1,2 Cao Xiao,2 Trong Nghia Hoang,3 Lucas M. Glass,2 Jimeng Sun4
1 Harvard T. H. Chan School of Public Health, Harvard University, Boston, MA, USA2 Analytic Center of Excellence, IQVIA, Cambridge, MA, USA
3 MIT-IBM Watson AI Lab, IBM Research, Cambridge, MA, USA4 Computational Science and Engineering, Georgia Institute of Technology, Atlanta, GA, USA
[email protected], [email protected], [email protected], [email protected], [email protected]
Abstract
Adverse drug-drug interactions (DDIs) remain a leadingcause of morbidity and mortality. Identifying potential DDIsduring the drug design process is critical for patients and so-ciety. Although several computational models have been pro-posed for DDI prediction, there are still limitations: (1) spe-cialized design of drug representation for DDI predictions islacking; (2) predictions are based on limited labelled dataand do not generalize well to unseen drugs or DDIs; and(3) models are characterized by a large number of param-eters, thus are hard to interpret. In this work, we developa ChemicAl SubstrucTurE Representation (CASTER) frame-work that predicts DDIs given chemical structures of drugs.CASTER aims to mitigate these limitations via (1) a sequen-tial pattern mining module rooted in the DDI mechanism toefficiently characterize functional sub-structures of drugs; (2)an auto-encoding module that leverages both labelled and un-labelled chemical structure data to improve predictive accu-racy and generalizability; and (3) a dictionary learning mod-ule that explains the prediction via a small set of coefficientswhich measure the relevance of each input sub-structures tothe DDI outcome. We evaluated CASTER on two real-worldDDI datasets and showed that it performed better than state-of-the-art baselines and provided interpretable predictions.
1 IntroductionAdverse drug-drug interactions (DDIs) are caused by phar-macological interactions of drugs. They result in a largenumber of morbidity and mortality, and incur huge med-ical costs (Giacomini et al. 2007; Onakpoya, Heneghan,and Aronson 2016). Thus gaining accurate and comprehen-sive knowledge of DDIs, especially during the drug de-sign process, is important to both patients and pharmaceu-tical industry. Traditional strategies of gaining DDI knowl-edge includes preclinical in vitro safety profiling and clin-ical safety trials, however they are restricted in terms ofsmall scales, long duration, and huge costs (Whitebreadet al. 2005). Recently, deep learning (DL) models thatleverage massive biomedical data for efficient DDI predic-tions emerged as a promising direction (Lo et al. 2018;Ryu, Kim, and Lee 2018; Xiao et al. 2017; Jin et al. 2017;Fu, Xiao, and Sun 2020). These methods are based on theassumption that drugs with similar representations (of chem-ical structure) will have similar properties (e.g., DDIs). De-spite the reported good performance, these works still have
the following limitations:1. Lack of specialized drug representation for DDI pre-
diction. One of the major mechanism of drug interac-tions results from the chemical reactions among only afew functional sub-structures of the entire drug’s molec-ular structure (Silverman and Holladay 2014), whilethe remaining substructures are less relevant. Many drugpairs that are not similar in terms of DDI interactioncan still have significant overlap on irrelevant substruc-tures. Previous works (Ryu, Kim, and Lee 2018; Gomez-Bombarelli et al. 2018; Jaeger, Fulle, and Turk 2018) of-ten generate drug representations using the entire chem-ical representation, which causes the learned represen-tations to be potentially biased toward irrelevant sub-structures. This undermines the learned drug similarityand DDI predictions.
2. Limited labels and generalizability. Some of the previ-ous methods need external biomedical knowledge for im-proved performance and cannot be generalized to drugsin early development phase (Ma et al. 2018; Ferdousi,Safdari, and Omidi 2017; Zhang et al. 2015). Others relyon a small set of labelled training data, which impairstheir generalizability to new drugs or DDIs (Ryu, Kim,and Lee 2018; Zhang et al. 2015).
3. Non-interpretable prediction. Although DL modelsshow good performance in DDI prediction, they oftenproduce predictions that are characterized by a largenumber of parameters, which is hard to interpret (Gomez-Bombarelli et al. 2018; Jaeger, Fulle, and Turk 2018).
In this paper, we present a new ChemicAl SubstrucTurERepresentation (CASTER) framework that performs DDIpredictions based on chemical structure data of drugs.CASTER mitigates the aforementioned limitations via thefollowing technical contributions:1. Specialized representation for DDI. We developed a
sequential pattern mining module (Alg. 1) to accountfor the interaction mechanism between drugs, whichboils down to the reaction between their functional sub-structures (i.e., functional groups) (Silverman and Hol-laday 2014). This allows us to explicitly model a pairof drugs as a set of their functional sub-structures (Sec-tion 2.2). Unlike previous works, this representation canbe learned using both labelled and unlabelled data (i.e.,drug pairs that have no reported interaction) and is there-
arX
iv:1
911.
0644
6v2
[cs
.LG
] 2
0 N
ov 2
019
Drug: MelatoninCC(=O)NCCC1=CNc2c1cc(OC)cc2 CN1CCC23C4OC5=C(O)C=CC(CC1C2C=CC4O)=C35
Drug: Morphine
O
CH3
N
HO
HO
OH
OH
HO
HO OH
O
In Kiwi: Quinic acid
NH
HNO
CH 3
OH 3C
OC1C[C@@](O)(C[C@@H](O)[C@H]1O)C(O)=O
Figure 1: Examples of SMILES strings representing the molecular graphs of drugs and food constituents.
fore more informative towards DDI prediction as demon-strated in our experiment (Section 3).
2. Improved generalizability. CASTER only uses chemicalstructure data that any drug has as model input, which ex-tends the learning beyond DDI to include any compoundpairs with associated chemical information, such as drug-food interaction (DFI). Food compounds are describedby the same set of chemical features, thus using unla-beled drug-food pair can help improve drug-drug repre-sentation’s generalizability for better DDI performance.To achieve this, CASTER has a deep auto-encoding mod-ule that extracts information from both labelled and un-labelled drug pairs to embed these patterns into a latentspace (Section 2.3). The embedded representation can begeneralized to new drug pairs with reduced bias towardsirrelevant or noisy artifacts in training data, thus improv-ing generalizability.
3. Interpretable prediction. CASTER has a dictionarylearning module that generates a small set of coefficientsto measure the relevance of each input sub-structures tothe DDI outcomes, thus making it more interpretable1 tohuman practitioners than the predictions produced by ex-isting DL methods. This is achieved by projecting drugpairs onto the subspace defined by the above general-izable embeddings of the frequent sub-structures (Sec-tion 2.4).
We compared CASTER with several state-of-the-art base-lines including DeepDDI (Ryu, Kim, and Lee 2018),mol2vec (Jaeger, Fulle, and Turk 2018) and molecularVAE (Gomez-Bombarelli et al. 2018) on two real-worldDDI datasets. Experimental results show CASTER outper-formed the best baselines consistently (Section 3). We alsoshow by a case study that CASTER produces interpretableresults that capture meaningful sub-structures with respectto the DDI outcomes.
2 MethodThis section presents our CASTER framework (Fig. 2). Ourproblem settings are summarized in Section 2.1. Then, Sec-tion 2.2 describes a functional representation of drugs whichis inspired by the mechanism of drug interactions. Sec-tion 2.3 develops a deep auto-encoding method that lever-ages both labelled and unlabelled data to further embed thefunctional representation of drugs into a parsimonious latentspace that can be generalized to new drugs. Section 2.4 then
1This is only to understand how CASTER makes predictions. Itis NOT our aim to interpret the prediction outcome in terms of thechemical process that regulates drug interactions.
explains how this representation can be leveraged to gener-ate accurate and interpretable predictions.
2.1 Problem SettingsThe DDI prediction task is about developing a computa-tional model that accepts two drugs (or their representa-tions) as inputs and generate an output prediction indicat-ing whether there exists an interaction between them. In thistask, each drug can be encoded by the simplified molecular-input line-entry system (SMILES) (System 2015). In par-ticular, the associated SMILES string S for each drug is asequence of symbols of the chemical atoms and bonds inits depth-first traversal order of its molecular structure graph(see Fig. 1) where nodes are atoms and edges are bonds be-tween them.
Our dataset consists of (a) a set of SMILES strings S ={S1, . . . ,Sn} where n is the total number of drugs, (b) a setI = {(St,S
′t)}mt=1 of m drug pairs reported to have interac-
tions; and (c) a set U = (S × S) \ I of length τ drug pairsthat have not been reported with interactions.
To predict drug interactions, we need to learn a mappingG : S × S → [0, 1] from a drug-drug pair (S,S′) ∈ S × Sto a probability that indicates the chance that S and S′ willhave interaction.
This will be achieved by learning an embedded functionalrepresentation z ∈ Rd for each drug via a deep embeddingmodule E : S → Rd (Sections 2.2 and 2.3), and a mappingG : E ×E → [0, 1] from the embedded representation (z, z′)of a drug pair to their interaction outcome (Section 2.4).
2.2 Generating Functional RepresentationsInstead of generating a wholistic representation for inputchemical, we generate a functional representation using sub-structures information. This section tackles the frequent se-quential pattern mining task to generate recurring chemicalsub-structures in our database S of molecular representa-tions of drugs. In particular, we define a frequent patternformally as:Definition 1 (Frequent Chemical Substructures) LetS ∈ S be a SMILES string representing the moleculargraph of a drug in depth-first traversal order (see Fig. 1)and let C be a consecutive sub-string of it. Then, C alsocorresponds to a depth-first traversal representation of amolecular sub-graph. C is a frequent substructure if itsoccurring frequency in S is above a threshold η.
This can be achieved by observing that by definitionsub-structures (or sub-graphs) that appear across differentdrugs will be represented by the same SMILES sub-strings
MelatoninCC(=O)NCCC1=CNc2c1cc(OC)cc2
ThiamineCc1ncc(C[n+]2csc(CCO)c2C)c(N)n1
SPMDecomposition
COc1ccc2[nH]cc
(CCNC(C)=O)c2c1
InputRepresentation
Cc1nccC[n+]2csc
(CCO)c2C
C(N)n1
EncoderModule
.
.
.
u1<latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="CQ83UILG6vyW1KqJe7LvAF5fpyc=">AAAB6nicbVBNSwMxFHzrZ61Vq1cvwVbwVHa96FHw4rGC/YB2Ldk024Ym2SV5q5Sl/8OLB0X8Qd78N2bbHrR1IDDMvMebTJRKYdH3v72Nza3tnd3SXnm/cnB4VD2utG2SGcZbLJGJ6UbUcik0b6FAybup4VRFkneiyW3hd564sSLRDzhNeajoSItYMIpOeqz3FcVxFOfZbBDUB9Wa3/DnIOskWJIaLNEcVL/6w4RlimtkklrbC/wUw5waFEzyWbmfWZ5SNqEj3nNUU8VtmM9Tz8i5U4YkTox7Gslc/b2RU2XtVEVusghpV71C/M/rZRhfh7nQaYZcs8WhOJMEE1JUQIbCcIZy6ghlRrishI2poQxdUWVXQrD65XXSvmwEfiO496EEp3AGFxDAFdzAHTShBQwMvMAbvHvP3qv3sahrw1v2dgJ/4H3+AJjxkLk=</latexit><latexit sha1_base64="CQ83UILG6vyW1KqJe7LvAF5fpyc=">AAAB6nicbVBNSwMxFHzrZ61Vq1cvwVbwVHa96FHw4rGC/YB2Ldk024Ym2SV5q5Sl/8OLB0X8Qd78N2bbHrR1IDDMvMebTJRKYdH3v72Nza3tnd3SXnm/cnB4VD2utG2SGcZbLJGJ6UbUcik0b6FAybup4VRFkneiyW3hd564sSLRDzhNeajoSItYMIpOeqz3FcVxFOfZbBDUB9Wa3/DnIOskWJIaLNEcVL/6w4RlimtkklrbC/wUw5waFEzyWbmfWZ5SNqEj3nNUU8VtmM9Tz8i5U4YkTox7Gslc/b2RU2XtVEVusghpV71C/M/rZRhfh7nQaYZcs8WhOJMEE1JUQIbCcIZy6ghlRrishI2poQxdUWVXQrD65XXSvmwEfiO496EEp3AGFxDAFdzAHTShBQwMvMAbvHvP3qv3sahrw1v2dgJ/4H3+AJjxkLk=</latexit><latexit sha1_base64="X31DDuAMnsBmBK4EuFEIvCPSJPk=">AAAB9XicbVDLSgMxFL1TX7W+qi7dBFvBVZkRQZcFNy4r2Ae0tWTSTBuayQzJHaUM/Q83LhRx67+482/MtLPQ1gOBwzn3ck+OH0th0HW/ncLa+sbmVnG7tLO7t39QPjxqmSjRjDdZJCPd8anhUijeRIGSd2LNaehL3vYnN5nffuTaiEjd4zTm/ZCOlAgEo2ilh2ovpDj2gzSZDbzqoFxxa+4cZJV4OalAjsag/NUbRiwJuUImqTFdz42xn1KNgkk+K/USw2PKJnTEu5YqGnLTT+epZ+TMKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RZVsCd7yl1dJ66LmuTXvzq3UL/M6inACp3AOHlxBHW6hAU1goOEZXuHNeXJenHfnYzFacPKdY/gD5/MH3gqSBw==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit><latexit sha1_base64="TWafxJPENshfpcWOwqqZPzQAhbM=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kCazgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MH3qqSCQ==</latexit>
. . .<latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit>
Dictionaryz
<latexit sha1_base64="1BKIREjyZSmsQJ7uC3OWkOGDaRc=">AAAB83icbVDLSgMxFL1TX7W+qi7dBFvBVZkpgi4LblxWsA/oDCWTZtrQTCYkGaEO/Q03LhRx68+482/MtLPQ1gOBwzn3ck9OKDnTxnW/ndLG5tb2Tnm3srd/cHhUPT7p6iRVhHZIwhPVD7GmnAnaMcxw2peK4jjktBdOb3O/90iVZol4MDNJgxiPBYsYwcZKft2PsZmEUfY0rw+rNbfhLoDWiVeQGhRoD6tf/ighaUyFIRxrPfBcaYIMK8MIp/OKn2oqMZniMR1YKnBMdZAtMs/RhVVGKEqUfcKghfp7I8Ox1rM4tJN5RL3q5eJ/3iA10U2QMSFTQwVZHopSjkyC8gLQiClKDJ9ZgoliNisiE6wwMbamii3BW/3yOuk2G57b8O6btdZVUUcZzuAcLsGDa2jBHbShAwQkPMMrvDmp8+K8Ox/L0ZJT7JzCHzifP7hWkWo=</latexit><latexit sha1_base64="1BKIREjyZSmsQJ7uC3OWkOGDaRc=">AAAB83icbVDLSgMxFL1TX7W+qi7dBFvBVZkpgi4LblxWsA/oDCWTZtrQTCYkGaEO/Q03LhRx68+482/MtLPQ1gOBwzn3ck9OKDnTxnW/ndLG5tb2Tnm3srd/cHhUPT7p6iRVhHZIwhPVD7GmnAnaMcxw2peK4jjktBdOb3O/90iVZol4MDNJgxiPBYsYwcZKft2PsZmEUfY0rw+rNbfhLoDWiVeQGhRoD6tf/ighaUyFIRxrPfBcaYIMK8MIp/OKn2oqMZniMR1YKnBMdZAtMs/RhVVGKEqUfcKghfp7I8Ox1rM4tJN5RL3q5eJ/3iA10U2QMSFTQwVZHopSjkyC8gLQiClKDJ9ZgoliNisiE6wwMbamii3BW/3yOuk2G57b8O6btdZVUUcZzuAcLsGDa2jBHbShAwQkPMMrvDmp8+K8Ox/L0ZJT7JzCHzifP7hWkWo=</latexit><latexit sha1_base64="1BKIREjyZSmsQJ7uC3OWkOGDaRc=">AAAB83icbVDLSgMxFL1TX7W+qi7dBFvBVZkpgi4LblxWsA/oDCWTZtrQTCYkGaEO/Q03LhRx68+482/MtLPQ1gOBwzn3ck9OKDnTxnW/ndLG5tb2Tnm3srd/cHhUPT7p6iRVhHZIwhPVD7GmnAnaMcxw2peK4jjktBdOb3O/90iVZol4MDNJgxiPBYsYwcZKft2PsZmEUfY0rw+rNbfhLoDWiVeQGhRoD6tf/ighaUyFIRxrPfBcaYIMK8MIp/OKn2oqMZniMR1YKnBMdZAtMs/RhVVGKEqUfcKghfp7I8Ox1rM4tJN5RL3q5eJ/3iA10U2QMSFTQwVZHopSjkyC8gLQiClKDJ9ZgoliNisiE6wwMbamii3BW/3yOuk2G57b8O6btdZVUUcZzuAcLsGDa2jBHbShAwQkPMMrvDmp8+K8Ox/L0ZJT7JzCHzifP7hWkWo=</latexit><latexit sha1_base64="1BKIREjyZSmsQJ7uC3OWkOGDaRc=">AAAB83icbVDLSgMxFL1TX7W+qi7dBFvBVZkpgi4LblxWsA/oDCWTZtrQTCYkGaEO/Q03LhRx68+482/MtLPQ1gOBwzn3ck9OKDnTxnW/ndLG5tb2Tnm3srd/cHhUPT7p6iRVhHZIwhPVD7GmnAnaMcxw2peK4jjktBdOb3O/90iVZol4MDNJgxiPBYsYwcZKft2PsZmEUfY0rw+rNbfhLoDWiVeQGhRoD6tf/ighaUyFIRxrPfBcaYIMK8MIp/OKn2oqMZniMR1YKnBMdZAtMs/RhVVGKEqUfcKghfp7I8Ox1rM4tJN5RL3q5eJ/3iA10U2QMSFTQwVZHopSjkyC8gLQiClKDJ9ZgoliNisiE6wwMbamii3BW/3yOuk2G57b8O6btdZVUUcZzuAcLsGDa2jBHbShAwQkPMMrvDmp8+K8Ox/L0ZJT7JzCHzifP7hWkWo=</latexit>
DictionaryLearningModule
uk<latexit sha1_base64="/6OzIFQ3FWANgVlQsWcaapBLcNY=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBpPKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPNtuSQw==</latexit><latexit sha1_base64="/6OzIFQ3FWANgVlQsWcaapBLcNY=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBpPKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPNtuSQw==</latexit><latexit sha1_base64="/6OzIFQ3FWANgVlQsWcaapBLcNY=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBpPKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPNtuSQw==</latexit><latexit sha1_base64="/6OzIFQ3FWANgVlQsWcaapBLcNY=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBpPKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPNtuSQw==</latexit>
u2<latexit sha1_base64="gODMpR8Tk3ddx9unsUNZyXdvztI=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucP4C+SCg==</latexit><latexit sha1_base64="gODMpR8Tk3ddx9unsUNZyXdvztI=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucP4C+SCg==</latexit><latexit sha1_base64="gODMpR8Tk3ddx9unsUNZyXdvztI=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucP4C+SCg==</latexit><latexit sha1_base64="gODMpR8Tk3ddx9unsUNZyXdvztI=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QZrMBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucP4C+SCg==</latexit>
. . .<latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit><latexit sha1_base64="lahmCktN/Pt2AvMJAxpKpUTqfl0=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LLaCp5IUQY8FLx4r2A9oQ9lstu3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSNnGqGW+xWMa6G1DDpVC8hQIl7yaa0yiQvBNMbud+54lrI2L1gNOE+xEdKTEUjKKVutW+DGM01UG54tbcBcg68XJSgRzNQfmrH8YsjbhCJqkxPc9N0M+oRsEkn5X6qeEJZRM64j1LFY248bPFvTNyYZWQDGNtSyFZqL8nMhoZM40C2xlRHJtVby7+5/VSHN74mVBJilyx5aJhKgnGZP48CYXmDOXUEsq0sLcSNqaaMrQRlWwI3urL66Rdr3luzbuvVxpXeRxFOINzuAQPrqEBd9CEFjCQ8Ayv8OY8Oi/Ou/OxbC04+cwp/IHz+QNx1I+E</latexit>b1
<latexit sha1_base64="eGsv5OhlzofBIrFu9TDoI2eTIXU=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHwZKR9g==</latexit><latexit sha1_base64="eGsv5OhlzofBIrFu9TDoI2eTIXU=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHwZKR9g==</latexit><latexit sha1_base64="eGsv5OhlzofBIrFu9TDoI2eTIXU=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHwZKR9g==</latexit><latexit sha1_base64="eGsv5OhlzofBIrFu9TDoI2eTIXU=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzgVcZlMpu1Z2DrBIvJ2XI0RiUvnrDiCUhSsME1brrubHpp1QZzgTOir1EY0zZhI6wa6mkIep+Ok89I+dWGZIgUvZJQ+bq742UhlpPQ99OZiH1speJ/3ndxATX/ZTLODEo2eJQkAhiIpJVQIZcITNiagllitushI2poszYooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYwUPAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHwZKR9g==</latexit>
b2<latexit sha1_base64="4HAP7dMlcGa98jrXueqgWG8ptAQ=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QerPBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPwxeR9w==</latexit><latexit sha1_base64="4HAP7dMlcGa98jrXueqgWG8ptAQ=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QerPBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPwxeR9w==</latexit><latexit sha1_base64="4HAP7dMlcGa98jrXueqgWG8ptAQ=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QerPBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPwxeR9w==</latexit><latexit sha1_base64="4HAP7dMlcGa98jrXueqgWG8ptAQ=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QerPBrXKoFR2q+4cZJV4OSlDjsag9NUbRiwJuUImqTFdz42xn1KNgkk+K/YSw2PKJnTEu5YqGnLTT+epZ+TcKkMSRNo+hWSu/t5IaWjMNPTtZBbSLHuZ+J/XTTC47qdCxQlyxRaHgkQSjEhWARkKzRnKqSWUaWGzEjammjK0RRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOEZXuHNeXJenHfnYzG65uQ7J/AHzucPwxeR9w==</latexit>
bk<latexit sha1_base64="7rFF+Nzk9gOjJn6+sjiSblPk3+A=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzwaQyKJXdqjsHWSVeTsqQozEoffWGEUtClIYJqnXXc2PTT6kynAmcFXuJxpiyCR1h11JJQ9T9dJ56Rs6tMiRBpOyThszV3xspDbWehr6dzELqZS8T//O6iQmu+ymXcWJQssWhIBHERCSrgAy5QmbE1BLKFLdZCRtTRZmxRRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOAZXuHNeXJenHfnYzG65uQ7J/AHzucPGcOSMA==</latexit><latexit sha1_base64="7rFF+Nzk9gOjJn6+sjiSblPk3+A=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzwaQyKJXdqjsHWSVeTsqQozEoffWGEUtClIYJqnXXc2PTT6kynAmcFXuJxpiyCR1h11JJQ9T9dJ56Rs6tMiRBpOyThszV3xspDbWehr6dzELqZS8T//O6iQmu+ymXcWJQssWhIBHERCSrgAy5QmbE1BLKFLdZCRtTRZmxRRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOAZXuHNeXJenHfnYzG65uQ7J/AHzucPGcOSMA==</latexit><latexit sha1_base64="7rFF+Nzk9gOjJn6+sjiSblPk3+A=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzwaQyKJXdqjsHWSVeTsqQozEoffWGEUtClIYJqnXXc2PTT6kynAmcFXuJxpiyCR1h11JJQ9T9dJ56Rs6tMiRBpOyThszV3xspDbWehr6dzELqZS8T//O6iQmu+ymXcWJQssWhIBHERCSrgAy5QmbE1BLKFLdZCRtTRZmxRRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOAZXuHNeXJenHfnYzG65uQ7J/AHzucPGcOSMA==</latexit><latexit sha1_base64="7rFF+Nzk9gOjJn6+sjiSblPk3+A=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZn0ThuayQxJRilD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx4Jr47rfztr6xubWdmGnuLu3f3BYOjpu6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yk/ntR1SaR/LeTGPsh3QkecAZNVZ6qPRCasZ+kPqzwaQyKJXdqjsHWSVeTsqQozEoffWGEUtClIYJqnXXc2PTT6kynAmcFXuJxpiyCR1h11JJQ9T9dJ56Rs6tMiRBpOyThszV3xspDbWehr6dzELqZS8T//O6iQmu+ymXcWJQssWhIBHERCSrgAy5QmbE1BLKFLdZCRtTRZmxRRVtCd7yl1dJq1b13Kp3VyvXL/M6CnAKZ3ABHlxBHW6hAU1goOAZXuHNeXJenHfnYzG65uQ7J/AHzucPGcOSMA==</latexit>
r1<latexit sha1_base64="XpEKMOMrzQFOVW2ZjsNzM88e5LA=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QapnA68yKJXdqjsHWSVeTsqQozEoffWGEUtCrpBJakzXc2Psp1SjYJLPir3E8JiyCR3xrqWKhtz003nqGTm3ypAEkbZPIZmrvzdSGhozDX07mYU0y14m/ud1Ewyu+6lQcYJcscWhIJEEI5JVQIZCc4ZyagllWtishI2ppgxtUUVbgrf85VXSqlU9t+rd1cr1y7yOApzCGVyAB1dQh1toQBMYaHiGV3hznpwX5935WIyuOfnOCfyB8/kD2hKSBg==</latexit><latexit sha1_base64="XpEKMOMrzQFOVW2ZjsNzM88e5LA=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QapnA68yKJXdqjsHWSVeTsqQozEoffWGEUtCrpBJakzXc2Psp1SjYJLPir3E8JiyCR3xrqWKhtz003nqGTm3ypAEkbZPIZmrvzdSGhozDX07mYU0y14m/ud1Ewyu+6lQcYJcscWhIJEEI5JVQIZCc4ZyagllWtishI2ppgxtUUVbgrf85VXSqlU9t+rd1cr1y7yOApzCGVyAB1dQh1toQBMYaHiGV3hznpwX5935WIyuOfnOCfyB8/kD2hKSBg==</latexit><latexit sha1_base64="XpEKMOMrzQFOVW2ZjsNzM88e5LA=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QapnA68yKJXdqjsHWSVeTsqQozEoffWGEUtCrpBJakzXc2Psp1SjYJLPir3E8JiyCR3xrqWKhtz003nqGTm3ypAEkbZPIZmrvzdSGhozDX07mYU0y14m/ud1Ewyu+6lQcYJcscWhIJEEI5JVQIZCc4ZyagllWtishI2ppgxtUUVbgrf85VXSqlU9t+rd1cr1y7yOApzCGVyAB1dQh1toQBMYaHiGV3hznpwX5935WIyuOfnOCfyB8/kD2hKSBg==</latexit><latexit sha1_base64="XpEKMOMrzQFOVW2ZjsNzM88e5LA=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7QapnA68yKJXdqjsHWSVeTsqQozEoffWGEUtCrpBJakzXc2Psp1SjYJLPir3E8JiyCR3xrqWKhtz003nqGTm3ypAEkbZPIZmrvzdSGhozDX07mYU0y14m/ud1Ewyu+6lQcYJcscWhIJEEI5JVQIZCc4ZyagllWtishI2ppgxtUUVbgrf85VXSqlU9t+rd1cr1y7yOApzCGVyAB1dQh1toQBMYaHiGV3hznpwX5935WIyuOfnOCfyB8/kD2hKSBg==</latexit>
LinearCoefficients
rk<latexit sha1_base64="/ugoANoimGth8cIoK5tHO+IcAPw=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7Qapng0llUCq7VXcOskq8nJQhR2NQ+uoNI5aEXCGT1Jiu58bYT6lGwSSfFXuJ4TFlEzriXUsVDbnpp/PUM3JulSEJIm2fQjJXf2+kNDRmGvp2Mgtplr1M/M/rJhhc91Oh4gS5YotDQSIJRiSrgAyF5gzl1BLKtLBZCRtTTRnaooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYw0PAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHMkOSQA==</latexit><latexit sha1_base64="/ugoANoimGth8cIoK5tHO+IcAPw=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7Qapng0llUCq7VXcOskq8nJQhR2NQ+uoNI5aEXCGT1Jiu58bYT6lGwSSfFXuJ4TFlEzriXUsVDbnpp/PUM3JulSEJIm2fQjJXf2+kNDRmGvp2Mgtplr1M/M/rJhhc91Oh4gS5YotDQSIJRiSrgAyF5gzl1BLKtLBZCRtTTRnaooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYw0PAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHMkOSQA==</latexit><latexit sha1_base64="/ugoANoimGth8cIoK5tHO+IcAPw=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7Qapng0llUCq7VXcOskq8nJQhR2NQ+uoNI5aEXCGT1Jiu58bYT6lGwSSfFXuJ4TFlEzriXUsVDbnpp/PUM3JulSEJIm2fQjJXf2+kNDRmGvp2Mgtplr1M/M/rJhhc91Oh4gS5YotDQSIJRiSrgAyF5gzl1BLKtLBZCRtTTRnaooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYw0PAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHMkOSQA==</latexit><latexit sha1_base64="/ugoANoimGth8cIoK5tHO+IcAPw=">AAAB9XicbVDLSgMxFL3js9ZX1aWbYCu4KjNF0GXBjcsK9gFtLZk004ZmMkNyRylD/8ONC0Xc+i/u/Bsz7Sy09UDgcM693JPjx1IYdN1vZ219Y3Nru7BT3N3bPzgsHR23TJRoxpsskpHu+NRwKRRvokDJO7HmNPQlb/uTm8xvP3JtRKTucRrzfkhHSgSCUbTSQ6UXUhz7Qapng0llUCq7VXcOskq8nJQhR2NQ+uoNI5aEXCGT1Jiu58bYT6lGwSSfFXuJ4TFlEzriXUsVDbnpp/PUM3JulSEJIm2fQjJXf2+kNDRmGvp2Mgtplr1M/M/rJhhc91Oh4gS5YotDQSIJRiSrgAyF5gzl1BLKtLBZCRtTTRnaooq2BG/5y6ukVat6btW7q5Xrl3kdBTiFM7gAD66gDrfQgCYw0PAMr/DmPDkvzrvzsRhdc/KdE/gD5/MHMkOSQA==</latexit>
CASTER<latexit sha1_base64="wBxt0+uNNcfqvwCvBcFfAquFX30=">AAAB+nicbVBNS8NAEN34WetXqkcvi63gqSRF0GOlCB6r9gvaUDbbbbt0swm7E7XE/hQvHhTx6i/x5r9x2+agrQ8GHu/NMDPPjwTX4Djf1srq2vrGZmYru72zu7dv5w4aOowVZXUailC1fKKZ4JLVgYNgrUgxEviCNf1RZeo375nSPJQ1GEfMC8hA8j6nBIzUtXOFDrBHAEgql3e1q9tJoWvnnaIzA14mbkryKEW1a391eiGNAyaBCqJ123Ui8BKigFPBJtlOrFlE6IgMWNtQSQKmvWR2+gSfGKWH+6EyJQHP1N8TCQm0Hge+6QwIDPWiNxX/89ox9C+8hMsoBibpfFE/FhhCPM0B97hiFMTYEEIVN7diOiSKUDBpZU0I7uLLy6RRKrpO0b0p5ctnaRwZdISO0Sly0Tkqo2tURXVE0QN6Rq/ozXqyXqx362PeumKlM4foD6zPH05tk1A=</latexit><latexit sha1_base64="wBxt0+uNNcfqvwCvBcFfAquFX30=">AAAB+nicbVBNS8NAEN34WetXqkcvi63gqSRF0GOlCB6r9gvaUDbbbbt0swm7E7XE/hQvHhTx6i/x5r9x2+agrQ8GHu/NMDPPjwTX4Djf1srq2vrGZmYru72zu7dv5w4aOowVZXUailC1fKKZ4JLVgYNgrUgxEviCNf1RZeo375nSPJQ1GEfMC8hA8j6nBIzUtXOFDrBHAEgql3e1q9tJoWvnnaIzA14mbkryKEW1a391eiGNAyaBCqJ123Ui8BKigFPBJtlOrFlE6IgMWNtQSQKmvWR2+gSfGKWH+6EyJQHP1N8TCQm0Hge+6QwIDPWiNxX/89ox9C+8hMsoBibpfFE/FhhCPM0B97hiFMTYEEIVN7diOiSKUDBpZU0I7uLLy6RRKrpO0b0p5ctnaRwZdISO0Sly0Tkqo2tURXVE0QN6Rq/ozXqyXqx362PeumKlM4foD6zPH05tk1A=</latexit><latexit sha1_base64="wBxt0+uNNcfqvwCvBcFfAquFX30=">AAAB+nicbVBNS8NAEN34WetXqkcvi63gqSRF0GOlCB6r9gvaUDbbbbt0swm7E7XE/hQvHhTx6i/x5r9x2+agrQ8GHu/NMDPPjwTX4Djf1srq2vrGZmYru72zu7dv5w4aOowVZXUailC1fKKZ4JLVgYNgrUgxEviCNf1RZeo375nSPJQ1GEfMC8hA8j6nBIzUtXOFDrBHAEgql3e1q9tJoWvnnaIzA14mbkryKEW1a391eiGNAyaBCqJ123Ui8BKigFPBJtlOrFlE6IgMWNtQSQKmvWR2+gSfGKWH+6EyJQHP1N8TCQm0Hge+6QwIDPWiNxX/89ox9C+8hMsoBibpfFE/FhhCPM0B97hiFMTYEEIVN7diOiSKUDBpZU0I7uLLy6RRKrpO0b0p5ctnaRwZdISO0Sly0Tkqo2tURXVE0QN6Rq/ozXqyXqx362PeumKlM4foD6zPH05tk1A=</latexit><latexit sha1_base64="wBxt0+uNNcfqvwCvBcFfAquFX30=">AAAB+nicbVBNS8NAEN34WetXqkcvi63gqSRF0GOlCB6r9gvaUDbbbbt0swm7E7XE/hQvHhTx6i/x5r9x2+agrQ8GHu/NMDPPjwTX4Djf1srq2vrGZmYru72zu7dv5w4aOowVZXUailC1fKKZ4JLVgYNgrUgxEviCNf1RZeo375nSPJQ1GEfMC8hA8j6nBIzUtXOFDrBHAEgql3e1q9tJoWvnnaIzA14mbkryKEW1a391eiGNAyaBCqJ123Ui8BKigFPBJtlOrFlE6IgMWNtQSQKmvWR2+gSfGKWH+6EyJQHP1N8TCQm0Hge+6QwIDPWiNxX/89ox9C+8hMsoBibpfFE/FhhCPM0B97hiFMTYEEIVN7diOiSKUDBpZU0I7uLLy6RRKrpO0b0p5ctnaRwZdISO0Sly0Tkqo2tURXVE0QN6Rq/ozXqyXqx362PeumKlM4foD6zPH05tk1A=</latexit>Latent
Decoder
Predictor
Reconstruction& Prediction
Module
C<latexit sha1_base64="ngbdIO6kAptJzo/egAuccse4qrI=">AAAB83icbVDLSsNAFL2pr1pfVZduBlvBVUm60WWxG5cV7AOaUCbTSTt0MgnzEErob7hxoYhbf8adf+OkzUJbDwwczrmXe+aEKWdKu+63U9ra3tndK+9XDg6Pjk+qp2c9lRhJaJckPJGDECvKmaBdzTSng1RSHIec9sNZO/f7T1QqlohHPU9pEOOJYBEjWFvJr/sx1tMwzNqL+qhacxvuEmiTeAWpQYHOqPrljxNiYio04VipoeemOsiw1Ixwuqj4RtEUkxme0KGlAsdUBdky8wJdWWWMokTaJzRaqr83MhwrNY9DO5lHVOteLv7nDY2OboOMidRoKsjqUGQ40gnKC0BjJinRfG4JJpLZrIhMscRE25oqtgRv/cubpNdseG7De2jWWndFHWW4gEu4Bg9uoAX30IEuEEjhGV7hzTHOi/PufKxGS06xcw5/4Hz+AGKikT0=</latexit><latexit sha1_base64="ngbdIO6kAptJzo/egAuccse4qrI=">AAAB83icbVDLSsNAFL2pr1pfVZduBlvBVUm60WWxG5cV7AOaUCbTSTt0MgnzEErob7hxoYhbf8adf+OkzUJbDwwczrmXe+aEKWdKu+63U9ra3tndK+9XDg6Pjk+qp2c9lRhJaJckPJGDECvKmaBdzTSng1RSHIec9sNZO/f7T1QqlohHPU9pEOOJYBEjWFvJr/sx1tMwzNqL+qhacxvuEmiTeAWpQYHOqPrljxNiYio04VipoeemOsiw1Ixwuqj4RtEUkxme0KGlAsdUBdky8wJdWWWMokTaJzRaqr83MhwrNY9DO5lHVOteLv7nDY2OboOMidRoKsjqUGQ40gnKC0BjJinRfG4JJpLZrIhMscRE25oqtgRv/cubpNdseG7De2jWWndFHWW4gEu4Bg9uoAX30IEuEEjhGV7hzTHOi/PufKxGS06xcw5/4Hz+AGKikT0=</latexit><latexit sha1_base64="ngbdIO6kAptJzo/egAuccse4qrI=">AAAB83icbVDLSsNAFL2pr1pfVZduBlvBVUm60WWxG5cV7AOaUCbTSTt0MgnzEErob7hxoYhbf8adf+OkzUJbDwwczrmXe+aEKWdKu+63U9ra3tndK+9XDg6Pjk+qp2c9lRhJaJckPJGDECvKmaBdzTSng1RSHIec9sNZO/f7T1QqlohHPU9pEOOJYBEjWFvJr/sx1tMwzNqL+qhacxvuEmiTeAWpQYHOqPrljxNiYio04VipoeemOsiw1Ixwuqj4RtEUkxme0KGlAsdUBdky8wJdWWWMokTaJzRaqr83MhwrNY9DO5lHVOteLv7nDY2OboOMidRoKsjqUGQ40gnKC0BjJinRfG4JJpLZrIhMscRE25oqtgRv/cubpNdseG7De2jWWndFHWW4gEu4Bg9uoAX30IEuEEjhGV7hzTHOi/PufKxGS06xcw5/4Hz+AGKikT0=</latexit><latexit sha1_base64="ngbdIO6kAptJzo/egAuccse4qrI=">AAAB83icbVDLSsNAFL2pr1pfVZduBlvBVUm60WWxG5cV7AOaUCbTSTt0MgnzEErob7hxoYhbf8adf+OkzUJbDwwczrmXe+aEKWdKu+63U9ra3tndK+9XDg6Pjk+qp2c9lRhJaJckPJGDECvKmaBdzTSng1RSHIec9sNZO/f7T1QqlohHPU9pEOOJYBEjWFvJr/sx1tMwzNqL+qhacxvuEmiTeAWpQYHOqPrljxNiYio04VipoeemOsiw1Ixwuqj4RtEUkxme0KGlAsdUBdky8wJdWWWMokTaJzRaqr83MhwrNY9DO5lHVOteLv7nDY2OboOMidRoKsjqUGQ40gnKC0BjJinRfG4JJpLZrIhMscRE25oqtgRv/cubpNdseG7De2jWWndFHWW4gEu4Bg9uoAX30IEuEEjhGV7hzTHOi/PufKxGS06xcw5/4Hz+AGKikT0=</latexit>
ChemicalStructures
.
.
.
SPM<latexit sha1_base64="G3Z1D9s2CyOnLP5TN4/Xf0IN3i4=">AAAB9XicbVC7TsMwFL3hWcqrwMhi0SIxVUkXGCtYWJCKoA+pDZXjOq1V24lsB1RF/Q8WBhBi5V/Y+BucNgO0HMnS0Tn36h6fIOZMG9f9dlZW19Y3Ngtbxe2d3b390sFhS0eJIrRJIh6pToA15UzSpmGG006sKBYBp+1gfJX57UeqNIvkvZnE1Bd4KFnICDZWeqj0BDYjJdK7xs200i+V3ao7A1omXk7KkKPRL331BhFJBJWGcKx113Nj46dYGUY4nRZ7iaYxJmM8pF1LJRZU++ks9RSdWmWAwkjZJw2aqb83Uiy0nojATmYh9aKXif953cSEF37KZJwYKsn8UJhwZCKUVYAGTFFi+MQSTBSzWREZYYWJsUUVbQne4peXSatW9dyqd1sr1y/zOgpwDCdwBh6cQx2uoQFNIKDgGV7hzXlyXpx352M+uuLkO0fwB87nD+Wbkhk=</latexit><latexit sha1_base64="G3Z1D9s2CyOnLP5TN4/Xf0IN3i4=">AAAB9XicbVC7TsMwFL3hWcqrwMhi0SIxVUkXGCtYWJCKoA+pDZXjOq1V24lsB1RF/Q8WBhBi5V/Y+BucNgO0HMnS0Tn36h6fIOZMG9f9dlZW19Y3Ngtbxe2d3b390sFhS0eJIrRJIh6pToA15UzSpmGG006sKBYBp+1gfJX57UeqNIvkvZnE1Bd4KFnICDZWeqj0BDYjJdK7xs200i+V3ao7A1omXk7KkKPRL331BhFJBJWGcKx113Nj46dYGUY4nRZ7iaYxJmM8pF1LJRZU++ks9RSdWmWAwkjZJw2aqb83Uiy0nojATmYh9aKXif953cSEF37KZJwYKsn8UJhwZCKUVYAGTFFi+MQSTBSzWREZYYWJsUUVbQne4peXSatW9dyqd1sr1y/zOgpwDCdwBh6cQx2uoQFNIKDgGV7hzXlyXpx352M+uuLkO0fwB87nD+Wbkhk=</latexit><latexit sha1_base64="G3Z1D9s2CyOnLP5TN4/Xf0IN3i4=">AAAB9XicbVC7TsMwFL3hWcqrwMhi0SIxVUkXGCtYWJCKoA+pDZXjOq1V24lsB1RF/Q8WBhBi5V/Y+BucNgO0HMnS0Tn36h6fIOZMG9f9dlZW19Y3Ngtbxe2d3b390sFhS0eJIrRJIh6pToA15UzSpmGG006sKBYBp+1gfJX57UeqNIvkvZnE1Bd4KFnICDZWeqj0BDYjJdK7xs200i+V3ao7A1omXk7KkKPRL331BhFJBJWGcKx113Nj46dYGUY4nRZ7iaYxJmM8pF1LJRZU++ks9RSdWmWAwkjZJw2aqb83Uiy0nojATmYh9aKXif953cSEF37KZJwYKsn8UJhwZCKUVYAGTFFi+MQSTBSzWREZYYWJsUUVbQne4peXSatW9dyqd1sr1y/zOgpwDCdwBh6cQx2uoQFNIKDgGV7hzXlyXpx352M+uuLkO0fwB87nD+Wbkhk=</latexit><latexit sha1_base64="G3Z1D9s2CyOnLP5TN4/Xf0IN3i4=">AAAB9XicbVC7TsMwFL3hWcqrwMhi0SIxVUkXGCtYWJCKoA+pDZXjOq1V24lsB1RF/Q8WBhBi5V/Y+BucNgO0HMnS0Tn36h6fIOZMG9f9dlZW19Y3Ngtbxe2d3b390sFhS0eJIrRJIh6pToA15UzSpmGG006sKBYBp+1gfJX57UeqNIvkvZnE1Bd4KFnICDZWeqj0BDYjJdK7xs200i+V3ao7A1omXk7KkKPRL331BhFJBJWGcKx113Nj46dYGUY4nRZ7iaYxJmM8pF1LJRZU++ks9RSdWmWAwkjZJw2aqb83Uiy0nojATmYh9aKXif953cSEF37KZJwYKsn8UJhwZCKUVYAGTFFi+MQSTBSzWREZYYWJsUUVbQne4peXSatW9dyqd1sr1y/zOgpwDCdwBh6cQx2uoQFNIKDgGV7hzXlyXpx352M+uuLkO0fwB87nD+Wbkhk=</latexit>
Figure 2: CASTER workflow: (a) CASTER extracts frequent substructures C from the molecular database S via SPM (seeAlg. 1); (b) x is generated for each input pair (S,S′); (c) the functional representation x is embedded into a latent spacevia minimizing a reconstruction loss, which results in a latent feature vector z; the function representations of each frequentsubstructures {ui}ki=1 are also embedded into the latent space to yield a dictionary entry {bi}ki=1 respectively; (d) the latentfeature z is projected onto the subspace of {bi}ki=1 and results in linear coefficients {ri}ki=1; and (e) {ri}ki=1 are used as featuresfor training DDI prediction module. All components are in one trainable pipeline which is optimized end-to-end by minimizingboth the reconstruction and prediction losses.
Algorithm 1: The Chemical Sequential Pattern MiningAlgorithm
Initialize V to the set of all atoms and bonds, W as theset of tokenized SMILES strings input
Input η as the practitioner-specified frequencythreshold, and ` as the maximum size of V
for t = 1 . . . ` do(A, B), FREQ ← scan W//(A,B), FREQ are the frequentest pair and its frequencyif FREQ < η then
break // frequency lower than thresholdendW← find(A, B) ∈W, replace with (AB)// update W with the new token (AB)V← V ∪ (AB)// add (AB) to the vocabulary set V
end
thanks to their depth-first traversal representations. As such,frequent chemical sub-graphs can be extracted efficientlyby finding frequent SMILES sub-strings across the entiredataset. This is similar to the concept of sub-word unitsin the natural language processing domain (Gage 1994;Sennrich, Haddow, and Birch 2016). More concretely, wepropose a data-driven Chemical Sequential Pattern Miningalgorithm (SPM) algorith (Huang et al. 2019) see Algo. 1.Given an arbitrary input chemical SMILES string S, SPMgenerates a set of discrete frequent sub-structures. This dis-creteness enables explainability that previous fingerprintsare not capable of. The generated substructures will be usedto induce functional representations for all drug pairs in both
labelled I and unlabelled U datasets, which is defined for-mally in Definition 2 below.
Definition 2 (Functional Representation) LetC = {C1, . . . ,Ck} denote the set of frequent sub-structures. Each drug pair (S,S′) is now represented viaa k-dimensional multi-hot vector x = [x(1) x(2) . . .x(k)]where x(i) = 1 if Ci ∈ S and Ci ∈ S′, and x(i) = 0otherwise. The resulting vector x is formally defined as thefunctional representation of (S,S′).
Note that this functional representation maps drug pairs tothe frequent substructures. Different substructures may in-teract with each other which lead to drug interactions. Wefind this representation alone has better predictive valuesthan classic fixed-sized drug fingerprints (Ryu, Kim, andLee 2018; Rogers and Hahn 2010; Riniker and Landrum2013; James 2004) in DDI prediction task.
2.3 Latent Feature EmbeddingThis section develops a latent feature embedding modulethat extracts information from both labelled and unlabelleddrug pairs to further embed the above functional represen-tations into a latent space, which can be generalized to newdrug pairs. This includes an encoder, a decoder and a recon-struction loss components which are detailed below.
Encoder. For each functional representation xt ∈ {0, 1}kof a drug-drug or drug-food pair (St,S
′t), a d-dimensional
(d � k) latent feature embedding is generated via a neuralnetwork (NN) parameterized with weights We and biasesbe:
zt = Wext + be , (1)
Decoder. To reconstruct the functional representation xt
given the latent embedding zt, we define another NN frame-work similarly parameterized by (Wd,bd) that maps ztback to
xt = sigmoid(Wdzt + bd
), (2)
where sigmoid(a) = [σ(a1) . . . σ(ak)] is a point-wise func-tion with σ(ai) = 1/(1 + exp(−ai)).
Reconstruction Loss. The encoding and decoding param-eters (Wd,bd) and (We,be) can be learned via minimizingthe following reconstruction loss:
Lr(xt, xt) =
k∑i=1
(x(i)t log
(x(i)t
)+(
1− x(i)t
)log(
1− x(i)t
)).
(3)Eq. (3) succinctly summarizes our latent feature embeddingprocedure, which is completely unsupervised since optimiz-ing Eq. (3) only requires access to unlabelled drug pairs astraining data. This allows CASTER to tap into the richersource of unlabelled drug data to extract more relevant fea-tures, which help improving the performance of a predictorlearned only from a much smaller set of labelled data. This isone of the key advantages of CASTER over previous effortson DDI prediction (Ryu, Kim, and Lee 2018).
2.4 Interpretable Prediction with DictionaryLearning
Instead of feeding the functional representation through asimple logistic regression model, we develop a dictionarylearning module to help human practitioner to understandhow CASTER makes prediction and identify which sub-structures can possibly lead to the interaction.
Deep Dictionary Representation. To provide a more par-simonious representation that also facilitates prediction in-terpretation, CASTER first generates a functional represen-tation ui for each frequent sub-structure Ci ∈ C as a single-hot vector ui = [u
(1)i . . .u
(k)i ] where u
(j)i = I(i = j).
CASTER then exploits the above encoder to generate a ma-trix B = [b1,b2, . . . ,bk] of latent feature vectors for U ={u1, . . . ,uk} such that,
bi = Weui + be . (4)
Likewise, each functional representation x of any drug-drugpair (S,S′) (see Definition 2) can also be first translated intoa latent feature vector z using the same encoder (see Eq. (1)).The resulting latent vector z can then be projected on a sub-space defined by span(B) for which z ' b1r1 + . . .bkrkwhere r = [r1 r2 . . . rk] is a column vector of projectioncoefficients. These coefficients can be computed via opti-mizing the following projection loss,
Lp(r) =1
2
∥∥z−B>r∥∥22
+ λ1‖r‖2 (5)
where the L2-norm ‖r‖2 controls the complexity of the pro-jection, and λ1 > 0 is a regularization parameter. In a morepractical implementation, we can also put both the encoder
and the above projection (which previously assumes a pre-computed B) in one pipeline to optimize both the basis sub-space and the projection. This is achieved via minimizingthe following augmented loss function,
Lp
(We,be, r
)=
(1
2
∥∥z−B>r∥∥22
+ λ1‖r‖2)
+λ2 ‖B‖2F ,
(6)where the expression for each column bi of B in Eq. (4) isplugged into the above to optimize the encoder parameters(We,be). Note that the first term in Eq. (6) can be min-imized more efficiently alone since it can be cast as solv-ing a ridge linear regression task where given a vector z tobe projected and a projection base B, we want to find non-trivial coefficients r that yields minimum projection loss. Bywarm-starting the optimization of r by initializing it withthe solution r∗ = arg minr Lp(r) of the above ridge re-gression task (see Eq. (5)), the optimizing process convergesfaster than randomly initializing r. The above problem canbe solved analytically with a closed-form solution:
r∗ =(B>B + λ1I
)−1B>z , (7)
where I is the identity matrix of size k. Empirically, we ob-served that with this warmed-up, the training pipeline canback-propagated through Eq. (6) more efficiently. Once opti-mized, the resulting projection coefficient r = [r1 r2 . . . rk]can be used as a dictionary representation for the corre-sponding drug-drug pair (S,S′).
End-to-End Training. Given each labelled drug-drug pair(S,S′), we generate its functional representation x ∈ [0, 1]k
(Section 2.2), latent feature embedding z ∈ Rd and thendeep dictionary representation r as detailed above. A neuralnetwork (NN) parameterized by weights Wp and biases bp
is then constructed to compute a probability score p(x) ∈[0, 1] based on r to assess the chance that S and S′ has aninteraction. This is defined below:
p(x) = σ(Wpr + bp
). (8)
We then optimize Wp(x) and bp via minimizing the fol-lowing binary cross-entropy loss with the true interactionoutcome y ∈ {0, 1}:
Lc(x, y) = y log p(x) + (1− y) log (1− p(x)) . (9)
More specifically, CASTER consists of two trainingstages. In the first stage, we pre-train the auto-encoder anddictionary learning module with our unlabelled drug-drugand drug-food pairs to let the encoder learns the most effi-cient representation for any chemical structures (e.g., drugsor food constituents) with the combined loss of αLr + βLp,where α and β are manually adjusted to balance betweenthe relative importance of these losses. Then, we fine-tunethe entire learning pipeline with the labelled dataset D forDDI prediction, using the aggregated loss αLr+βLp+γLc,where γ is also a loss adjustment parameter.
Interpretable Prediction. The learned predictor can thenbe used to make DDI prediction on new drugs and further-more, the projection coefficients r = [r1 r2 . . . rk] can also
Table 1: Dataset Statistics
Drugbank (DDI) BIOSNAP# Drugs 1,850 1,322# Positive DDI Labels 221,523 41,520# Negative Labels 221,523 41,520
Unlabelled# Drugs 9,675# Food Compounds 24,738# Drug-Drug Pairs 220,000# Drug-Food Pairs 220,000
be used to assess the relevance of the basis feature vectorsB = [b1,b2, . . . ,bk] to the prediction outcome. As eachbasis vector bi is associated with a frequent molecular sub-graph Ci, its corresponding coefficient ri reveals the statisti-cal importance of having the sub-graph Ci in the moleculargraphs of the drug pairs so that they would interact, thus ex-plaining the rationality behind CASTER’s prediction.
3 ExperimentWe design experiments to evaluate CASTER2 to answer thefollowing three questions:Q1: Does CASTER provide more accurate DDI prediction
than other strong baselines?Q2: Does the unlabelled data improve the DDI prediction in
situations with limited labelled data?Q3: Does CASTER dictionary module help interpret its pre-
dictions?
3.1 Experimental SetupDatasets. We evaluated CASTER using two datasets. (1)DrugBank (DDI) (Wishart et al. 2008) includes 1,850 ap-proved drugs. Each drug is associated with its chemicalstructure (SMILES), indication, protein binding, weight,mechanism and etc. The data includes 221,523 DDI positivelabels. (2) BIOSNAP (Marinka Zitnik and Leskovec 2018)that consists of 1,322 approved drugs with 41,520 labelledDDIs, obtained through drug labels and scientific publica-tions. For both dataset, we generate negative counterpartsby sampling complement set of positive drug pairs set as thenegative set following standard practice (Zitnik, Agrawal,and Leskovec 2018).
To demonstrate CASTER’s generalizability to unseendrugs and other chemical compounds, we generate a datasetconsisting of unlabelled drug-drug and drug-food com-pounds pairs. We consider all drugs from DrugBank in-cluding experimental, nutraceutical, investigational types ofdrugs and the food constituent SMILES strings come fromFooDB (FooDB 2018). In total, there are 9,675 drugs and24,738 food constituents. We randomly generate 220,000drug-drug pairs and 220,000 drug-food pairs as the unla-belled dataset to be pretrained. Data statistics are summa-rized in Table 1. For data processing, see (Appendix. A).
2Source code is at https://github.com/kexinhuang12345/CASTER
Metrics. We denote Y, Y as the ground truth, and pre-dicted values for drug-drug pairs dataset respectively. Wemeasure the prediction accuracy and efficiency using the fol-lowing metrics:1. ROC-AUC: Area under the receiver operating character-
istic curve: the area under the plot of the true positive rateagainst the false positive rate at various thresholds.
2. PR-AUC: Area under the precision-recall curve: the areaunder the plot of the precision rate against recall rate atvarious thresholds.
3. F1 Score: F1 Score is the harmonic mean of precisionand recall.
F1 = 2 · P · RP + R
, where P =|Y ∩ Y||Y|
,R =|Y ∩ Y||Y|
4. # Parameters: The number of parameters used in themodel.
CASTER Setup. We found the following hyperparametersa best fit to CASTER. For SPM, we set the frequency thresh-old η as 50 and it result in k = 1, 722 frequent substruc-tures. The dimension of latent representation from encoderd is set to be 50. The encoder and decoder both use threelayer perceptrons of hidden size 500 with ReLU activationfunction. The predictor uses six layer perceptrons with firstthree layers of hidden size 1,024 and last layer 64. The pre-dictor applies 1-D batch normalization with ReLU activa-tion units. For the trade-off between different loss, we setα = 1e − 1, β = 1e − 1, γ = 1. For regularization coef-ficients. we set λ1 = 1e − 5, λ2 = 1e − 1. We first pre-train the network for one epoch with the unlabelled datasetand then proceed to the labelled dataset training process. Weuse batch size 256 with Adam optimizer of learning rate1e − 3. All methods are implemented in PyTorch (Paszkeet al. 2017). For training, we use a server with 2 Intel XeonE5-2670v2 2.5GHz CPUs, 128 GB RAM and 3 NVIDIATesla K80 GPUs.
Evaluation Strategies. We randomly divided the datasetinto training, validation and testing sets in a 7:1:2 ratio. Foreach experiment, we conducted 5 independent runs with dif-ferent random splits of data and used early stopping basedon the ROC-AUC performance on the validation set. Theselected model via validation is then evaluated on the testset. The results are reported below.
3.2 Q1: CASTER achieves higher accuracy in DDIprediction
To answer Q1, we compared CASTER with the followingend-to-end algorithms:1. Logistic Regression (LR): LR with L2 regularization us-
ing representation generated from SPM.2. Nat.Prot: (Vilar et al. 2014) is the standard approach in
real-world DDI prediction task. It uses a similarity-basedmatrix heuristic method.
3. Mol2Vec: (Jaeger, Fulle, and Turk 2018) appliesWord2Vec on ECFP fingerprint of chemical compoundsto generate a dense representation of the chemical struc-tures. We adapt it by concatenating the latent variable of
Table 2: CASTER provides more accurate DDI prediction than other strong baselines. First / second row of each methodcorresponds to results reported on BIOSNAP / DrugBank (DDI) dataset respectively.
Model Dataset ROC-AUC PR-AUC F1 # Parameters
LR BIOSNAP 0.802± 0.001 0.779± 0.001 0.741± 0.002 1,723DrugBank 0.774± 0.003 0.745± 0.005 0.719± 0.006
Nat.Prot (Vilar et al. 2014) BIOSNAP 0.853± 0.001 0.848± 0.001 0.714± 0.001 N/ADrugBank 0.786± 0.003 0.753± 0.003 0.709± 0.004
Mol2Vec (Jaeger, Fulle, and Turk 2018) BIOSNAP 0.879± 0.006 0.861± 0.005 0.798± 0.007 8,061,953DrugBank 0.849± 0.004 0.828± 0.006 0.775± 0.004
MolVAE (Gomez-Bombarelli et al. 2018) BIOSNAP 0.892± 0.009 0.877± 0.009 0.788± 0.033 8,012,292DrugBank 0.852± 0.006 0.828± 0.009 0.769± 0.031
DeepDDI (Ryu, Kim, and Lee 2018) BIOSNAP 0.886± 0.007 0.871± 0.007 0.817± 0.007 8,517,633DrugBank 0.844± 0.003 0.828± 0.002 0.772± 0.006
CASTERBIOSNAP 0.910± 0.005 0.887± 0.008 0.843± 0.005 7,813,429DrugBank 0.861± 0.005 0.829± 0.003 0.796± 0.007
input pairs and use large multi-layer perceptron predic-tors to predict the drugs interaction.
4. MolVAE: (Gomez-Bombarelli et al. 2018) uses varia-tional autoencoders on SMILES input with the molec-ular property prediction auxiliary task to generate a com-pact representation. They use convolutional encoder andGRU-based decoder (Cho et al. 2014). We apply thisframework by following the same architecture but chang-ing the auxiliary task to drug interaction prediction.
5. DeepDDI: (Ryu, Kim, and Lee 2018) is a drug inter-action prediction model based on a task-specific chem-ical similarity profile. We modified the last layer of theoriginal DeepDDI implementation from multi-class to abinary class.
DDI prediction performance of all tested methods are re-ported in Table 2. The results show that CASTER achievesbetter predictive performance on DDI prediction across allevaluation metrics. This demonstrates that CASTER is ableto capture the necessary interaction mechanism and is there-fore more suitable for drug interaction prediction task thanprevious approaches.
3.3 Q2: CASTER leverages unlabelled datasuccessfully to improve predictionperformance
To answer Q2, we design an experiment to use only a smallset of labelled data and adjust the unlabelled data size tosee if the increasing amount of unlabelled data improves theperformance of DDI prediction on the test set. From Fig. 3,we observe that as the amount of unlabelled data increases,CASTER is able to leverage more information from the un-labelled data and consistently improves the accuracy of itsDDI prediction on both datasets.
3.4 Q3: CASTER generates interpretableprediction – A case study
Given an input pair, CASTER generates a set of coeffi-cients showing the relevance of substructures in terms ofCASTER’s predicted DDIs, a feature missing from otherbaselines. We select the interaction between Sildenafil and
other Nitrate-Based drugs as a case study. Sildenafil is aneffective treatment for erectile dysfunction and pulmonaryhypertension (Langtry and Markham 1999). Sildenafil is de-veloped as a phosphodiesterase-5 (PDE5) inhibitor. In thepresence of PDE5 inhibitor, nitrate (NO−3 )-based medica-tions such as Isosorbide Mononitrate (IM) can cause dra-matic increase in cyclic guanosine monophosphate (Mu-rad 1986), which leads to intense drops in blood pressurethat can cause heart attack (Langtry and Markham 1999;Ishikura et al. 2000; Chamsi-Pasha 2001). To demonstrateCASTER’s ability to identify potential causes of interaction,we want to test if CASTER assigns high coefficient to thenitrate group when it is asked to predict the interaction out-come between Sildenafil and Nitrate-Based drugs.
We first feed the SMILES strings of Sildenafil and IM intoCASTER, and it outputs 0.7928 confidence score of interac-tion, which is very high. We then examine the coefficientscorresponding to the functional substructures of the input,and observe that nitrate (O=[N+]([O-])) in IM has the high-est coefficient3 of 8.25 among 21 functional substructuresidentified by CASTER, which matches the chemical intuitiondiscussed above. Fig. 4 illustrates the coefficients generatedby CASTER for different sub-structures. To test the robust-ness of the interpretability result, we show that random ini-tialization does not affect our prediction by training fivemodels with different random seeds and calculated the co-efficients in Fig. 4. We consider coefficient from each run asa variable and then calculated their Correlation Matrix. Thehigh average correlation (0.7673) implies CASTER is robustbased on (Mukaka 2012). In order to avoid if the result is dueto chance, we search for all nitrate-based drugs in BIOS-NAP dataset, which results in the other four drugs: Nitro-glycerin, Isosorbide Dinitrate, Silver Nitrate, and ErythritylTetranitrate. We pair the SMILES string of Sildenafil witheach of these nitrate drugs and feed them to CASTER. Wethen observe that CASTER assigns on average 50% highercoefficient to nitrate than the mean of coefficients of othersub-structures existed in the input pair, which is not a coin-
3Note that this is the augmented coefficients multiplied by afactor, see Appendix A
0 4096 8192 20480 30720 61440Unlabelled Data Size
0.69
0.70
0.71
0.72
0.73
0.74
ROCA
UC
BIOSNAPCASTER95% CI
0 4096 8192 20480 30720 61440Unlabelled Data Size
0.60
0.62
0.64
0.66
0.68
ROCA
UC
DrugBank (DDI)CASTER95% CI
Figure 3: CASTER can leverage unlabelled data to improve DDI prediction in scarce labels scenario.
Isosorbide Mononitrate SildenafilO=[N+]([O-])O[C@@H]1CO[C@@H]2[C@@H](O)CO[C@H]12 CCCc1nn(C)c2c(=O)nc(-c3cc(S(=O)(=O)N4CCN(C)CC4)ccc3OCC)[nH]c12
������
�(�) = 0.793
Figure 4: CASTER produces interpretable coefficients for understanding drug interaction mechanism.
cidence. This case study thus demonstrated that CASTER iscapable of suggesting sparse and reasonable cues on findingsubstructures which are likely responsible to DDI.
4 ConclusionThis paper developed a new computational framework forDDI prediction called CASTER, which is an end-to-end dic-tionary learning framework that incorporates a specializedrepresentation for DDI prediction, inspired by the chemi-cal mechanism of drug interactions. We demonstrated em-pirically that CASTER can provide more accurate and inter-pretable DDI predictions than the previous approaches thatuse generic drug representations. For future works, we planto extend it to chemical sub-graph embedding and incorpo-rate metric learning for further improvement.
AcknowledgementThis work was in part supported by the National Sci-ence Foundation award IIS-1418511, CCF-1533768 andIIS-1838042, the National Institute of Health award NIHR01 1R01NS107291-01 and R56HL138415.
A Implementation DetailsFor preprocessing of the SMILES string, we use the RD-Kit software to convert all string into canonical form. As thesoftware first maps the string into a Mol format file, there arecases where RDKit cannot process some SMILES strings.We omit these data points. The unique number of drugs de-crease from 2,159 to 1,850 for DrugBank dataset and 1,514
to 1,322 for BIOSNAP. The overall number of DDI positivesamples drop from 222,127 to 221,523 for DrugBank and48,514 to 41,520 for BIOSNAP.
Due to the large size of DrugBank and limited time, ex-periment done on the DrugBank dataset uses another datasetsplit scheme to not only decrease the time but also fullyleverage the dataset. For the five independent run in eachexperiment, we divide the whole set into exclusive five foldsdata. We then report the average and standard deviationscores across these five runs. Hence, the evaluation considersthe full dataset but only uses a subsampled dataset in eachindependent run.Magnifying Factor. During training CASTER, we observethat since Eq. (7) applies ridge regression, the coefficientsr are relatively small. In order to allow faster training forthe predictor, we multiple each coefficient by a magnifyingfactor of 100, i.e. enlarging the learning rate for the inputweights, to obtain an augmented coefficient. This allow theloss decreases much faster.
References[Chamsi-Pasha 2001] Chamsi-Pasha, H. 2001. Sildenafil (vi-agra) and the heart. Journal of family & community medicine8(2):63.
[Cho et al. 2014] Cho, K.; van Merrienboer, B.; Gulcehre,C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Ben-gio, Y. 2014. Learning phrase representations using RNNencoder–decoder for statistical machine translation. In Pro-ceedings of the 2014 Conference on Empirical Methods inNatural Language Processing (EMNLP), 1724–1734.
[Ferdousi, Safdari, and Omidi 2017] Ferdousi, R.; Safdari,R.; and Omidi, Y. 2017. Computational prediction ofdrug-drug interactions based on drugs functional similari-ties. Journal of Biomedical Information.
[FooDB 2018] FooDB. 2018. Foodb version 1.0. http://foodb.ca/.
[Fu, Xiao, and Sun 2020] Fu, T.; Xiao, C.; and Sun, J. 2020.Core: Automatic molecule optimization using copy & refinestrategy. AAAI Conference on Artificial Intelligence.
[Gage 1994] Gage, P. 1994. A new algorithm for data com-pression. The C Users Journal 12(2):23–38.
[Giacomini et al. 2007] Giacomini, K.; Krauss, R.; Roden,D.; Eichelbaum, M.; and Hayden, M. 2007. When gooddrugs go bad. Nature 446:975–977.
[Gomez-Bombarelli et al. 2018] Gomez-Bombarelli, R.;Wei, J. N.; Duvenaud, D.; Hernandez-Lobato, J. M.;Sanchez-Lengeling, B.; Sheberla, D.; Aguilera-Iparraguirre,J.; Hirzel, T. D.; Adams, R. P.; and Aspuru-Guzik, A. 2018.Automatic chemical design using a data-driven contin-uous representation of molecules. ACS central science4(2):268–276.
[Huang et al. 2019] Huang, K.; Xiao, C.; Glass, L.; and Sun,J. 2019. Explainable substructure partition fingerprint forprotein, drug, and more. NeurIPS Learning Meaningful Rep-resentation of Life Workshop.
[Ishikura et al. 2000] Ishikura, F.; Beppu, S.; Hamada, T.;Khandheria, B. K.; Seward, J. B.; and Nehra, A. 2000. Ef-fects of sildenafil citrate (viagra) combined with nitrate onthe heart. Circulation 102(20):2516–2521.
[Jaeger, Fulle, and Turk 2018] Jaeger, S.; Fulle, S.; and Turk,S. 2018. Mol2vec: unsupervised machine learning approachwith chemical intuition. Journal of chemical informationand modeling 58(1):27–35.
[James 2004] James, C. A. 2004. Daylight theory manual.[Jin et al. 2017] Jin, B.; Yang, H.; Xiao, C.; Zhang, P.; Wei,X.; and Wang, F. 2017. Multitask dyadic prediction and itsapplication in prediction of adverse drug-drug interaction.
[Langtry and Markham 1999] Langtry, H. D., and Markham,A. 1999. Sildenafil. Drugs 57(6):967–989.
[Lo et al. 2018] Lo, Y.-C.; Rensi, S. E.; Torng, W.; and Alt-man, R. B. 2018. Machine learning in chemoinformaticsand drug discovery. Drug discovery today.
[Ma et al. 2018] Ma, T.; Xiao, C.; Zhou, J.; and Wang, F.2018. Drug similarity integration through attentive multi-view graph auto-encoders. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intel-ligence, IJCAI-18, 3477–3483. International Joint Confer-ences on Artificial Intelligence Organization.
[Marinka Zitnik and Leskovec 2018] Marinka Zitnik,Rok Sosic, S. M., and Leskovec, J. 2018. BioSNAPDatasets: Stanford biomedical network dataset collection.http://snap.stanford.edu/biodata.
[Mukaka 2012] Mukaka, M. M. 2012. A guide to appropriateuse of correlation coefficient in medical research. MalawiMedical Journal 24(3):69–71.
[Murad 1986] Murad, F. 1986. Cyclic guanosine monophos-phate as a mediator of vasodilation. The Journal of clinicalinvestigation 78(1):1–5.
[Onakpoya, Heneghan, and Aronson 2016] Onakpoya, I. J.;Heneghan, C. J.; and Aronson, J. K. 2016. Post-marketingwithdrawal of 462 medicinal products because of adversedrug reactions: a systematic review of the world literature.BMC medicine 14(1):10.
[Paszke et al. 2017] Paszke, A.; Gross, S.; Chintala, S.;Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.;Antiga, L.; and Lerer, A. 2017. Automatic differentiation inpytorch.
[Riniker and Landrum 2013] Riniker, S., and Landrum,G. A. 2013. Open-source platform to benchmark fin-gerprints for ligand-based virtual screening. Journal ofcheminformatics 5(1):26.
[Rogers and Hahn 2010] Rogers, D., and Hahn, M. 2010.Extended-connectivity fingerprints. Journal of chemical in-formation and modeling 50(5):742–754.
[Ryu, Kim, and Lee 2018] Ryu, J. Y.; Kim, H. U.; and Lee,S. Y. 2018. Deep learning improves prediction of drug–drug and drug–food interactions. Proceedings of the Na-tional Academy of Sciences 115(18):E4304–E4311.
[Sennrich, Haddow, and Birch 2016] Sennrich, R.; Haddow,B.; and Birch, A. 2016. Neural machine translation of rarewords with subword units. In Proceedings of the 54th An-nual Meeting of the Association for Computational Linguis-tics, 1715–1725. Berlin, Germany: Association for Compu-tational Linguistics.
[Silverman and Holladay 2014] Silverman, R. B., and Holla-day, M. W. 2014. The organic chemistry of drug design anddrug action. Academic press.
[System 2015] System, D. C. I. 2015. Smiles tutorial.[Vilar et al. 2014] Vilar, S.; Uriarte, E.; Santana, L.; Lorber-baum, T.; Hripcsak, G.; Friedman, C.; and Tatonetti, N. P.2014. Similarity-based modeling in large-scale predictionof drug-drug interactions. Nature protocols 9(9):2147.
[Whitebread et al. 2005] Whitebread, S.; Hamon, J.; Bo-janic, D.; and Urban, L. 2005. In vitro safety pharmacologyprofiling: an essential tool for successful drug development.Drug Discovery Today 10(21).
[Wishart et al. 2008] Wishart, D. S.; Knox, C.; Guo, A.;Cheng, D.; Shrivastava, S.; Tzur, D.; Gautam, B.; and Has-sanali, M. 2008. Drugbank: a knowledgebase for drugs, drugactions and drug targets. Nucleic Acids Research 36:901–906.
[Xiao et al. 2017] Xiao, C.; Zhang, P.; Chaowalitwongse,W. A.; Hu, J.; and Wang, F. 2017. Adverse drug reactionprediction with symbolic latent dirichlet allocation. In Pro-ceedings of the Thirty-First AAAI Conference on ArtificialIntelligence, AAAI’17, 1590–1596. AAAI Press.
[Zhang et al. 2015] Zhang, P.; Wang, F.; Hu, J.; and Sor-rentino, R. 2015. Label propagation prediction of drug-druginteractions based on clinical side effects. Scientific reports5.
[Zitnik, Agrawal, and Leskovec 2018] Zitnik, M.; Agrawal,M.; and Leskovec, J. 2018. Modeling polypharmacy sideeffects with graph convolutional networks. Bioinformatics34(13):i457–i466.