GPU-Enabled Spatiotemporal Model of

Stochastic Cardiac Calcium Dynamics and Arrhythmias

M. Saleet Jafri

Hoang Trong Minh Tuan

Department of Bioinformatics and Computational Biology George Mason University

Institute of Computational Medicine, The Johns Hopkins University

Department of Biomedical Engineering and Technology The University of Maryland Baltimore

Understanding Ca2+-Dependent

Cardiac Arrhythmias

• Heart disease is the leading cause of death in developed nations and an increasing problem in the developing world.

• The contraction of heart muscle pumps blood throughout the body to supply oxygen and nutrients to and remove carbon dioxide and waste products from the body tissues.

• Death often occurs by cardiac arrhythmias that prevent this normal function of the heart.

Membrane

Currents

Calcium

Handling

Force

Generation

Basic Components of

Cardiac E-C Coupling

Action

Potential

Calcium

Transient

Force

Transient

Understanding Ca2+-Dependent

Cardiac Arrhythmias • As cardiac myocyte calcium dynamics are essential for the

contraction of the heart, the dysfunction of normal calcium dynamics is often a major factor in cardiac arrhythmias.

• Computational models are an essential tool to understand the complex dynamcs of cardiac myocyte calcium signaling as the explain mechanisms by integrating the known information about this system.

Ca2+ Transient and Ca2+ Sparks in

Ventricular Myocytes

[Ca2+] "sparks" are the elementary release events. They are synchronized by the electrical signal of the cell to produce the elevation of [Ca2+].

What is a Ca2+ spark? 0.0 0.5 1.0 1.5 2.0 2.5 seconds

cell images at 0.5 sec per image

(from Cheng, Lederer & Cannell (1993), Science 262:740)

sparks

line-scan image at 2 ms per line

spark time

location

(From L. Fernando Santana, unpublished)

Heart Cell

• Many diads

• Diads separated

• NSR connections

• All sarcomeres shorten

uniformly

Sarcomere Geometry

M Z-line

Modified from J. Frank (1990)

T-tubule

TT-SR

junction

SR

RyRs

T-Tubules and SR Apposition

39

25

36

61

1 micron

from Baddeley, et al., 2009

Ryanodine Receptor Organization

RyR Channel Properties

[Ca2+]lumen and RyR gating

from Gyorke & Gyorke (1998) Biophys J. 75:280

trans [Ca2+]=20 mM trans [Ca2+]=5 mM

Three properties of RyR gating need to be included in the model.

2. SR lumenal [Ca2+]

3. Coupled gating of RyRs

Coupled gating of RyRs

Skeletal Muscle RyRs: Marx et al., (1998) Science 281:818.

Heart RyRs: Gaburjakova et al. (2001) Biophys. J. 80:380A.

control

+ FK506

0

2 1

RY

R'S

0

2 1

RY

R'S

1. Large number of RyRs (Franzini-Armstrong et al., 1998)

• Contraction of the heart is caused by the summation of calcium sparks.

• Build a model of contraction in the heart starting with these stochastic events.

• Use this model to answer fundamental questions about the mechanisms of arrhythmia that could not be answered with previous modeling efforts.

Approach

• What is the molecular basis of the calcium leak from the SR?

• How does the leak play a role in arrhythmogenisis?

Questions

Computational Challenges

• Reduction methods

– Reduction methods make assumptions about the system to make the reduction possible. These

might not be valid under all relevant conditions.

– Sometimes the reduction require simplification of the system by reducing model complexity.

This limits the details that can be included.

• Monte Carlo Simulation of simpler system

– Other attempts have simplified the dynamics of ryanodine receptor gating or the system.

– Using slower kinetics, fewer channels, release units, omitting physiological/biophysical detail

reduced veracity of the model.

We have developed the Ultrafast Monte Carlo Method that makes the

computation possible …

Stochastic Simulation is very computationally expensive. Various approaches

have been suggested to address these.

New Compartment Model Benchmarks

• 20,000 release units

• 49 RyRs

• 6 DHPR

• 1 second physiological simulation time

NOTE: Colleagues running MPI spatial cardiac code use at most 50 CPUs before losing speedup.

Method

Ultra-fast

Monte Carlo

On CPU

Ultra-fast

Monte Carlo

On GPU

Speedup

No I/O 69 min 3:40 min 19 x

I/O 70:52 min 4:26 min 16 x

Ultrafast Monte Carlo Method

CONDITION: Row sums at each matrix are zero

Ultrafast Monte Carlo Method (cont.)

• Single channel state: [x] with x=1..N, N = 5

• Cluster state: [y1,y2,…,yN] with yi = number of

channels in state i-th

• Heterogeneous cluster state:

– [y1z1,y1z2,…,y1zN,…,yNz1,yNz2,…,yNzN] with yi

= number of channels in state i-th, zj = number of

channels in state j-th

Ultrafast Monte Carlo Method (cont.)

• The method works

– Non-stationary state transition, i.e. functional transition rate

– Heterogeneous clusters where channels are modeled as

Markov-chain

– Memory efficient using compact form representation of

state transition matrix

– Lazy approach: only calculate probability of next state

transition if needed

SR Ca2+ Leak

• The leak of calcium out of the SR helps maintain calcium

homeostasis. It balances the SR Ca2+-ATPase flux.

• It increases when Ca2+ in the SR increases limiting SR loading.

Increases in SR Ca2+ can lead to larger Ca2+ release events.

• In some conditions, such as heart failure, disease, and Ca2+

overload, the leak has been suggested to increase the generation

of cardiac arrhythmias.

Ca2+ Leak Mechanisms

• Calcium release through the RyRs in the form of calcium sparks has been suggested to account for part of the SR Ca2+ leak.

– Calcium spark rate increases with increasing SR load.

• However, there remains a certain amount of leak, called ‘invisible leak’ that is not yet measured. Various sources have been suggested:

– Backflux through the SR Ca2+ ATP ase

– Other ion channels – IP3 receptors

– Non-junctional RyRs

Ca2+ Leak Mechanisms

• Backflux through the SR Ca2+ ATPase – Backflux through the SR Ca2+ ATPase has not been observed under physiological

conditions. Only extreme experimental manipulation can do so.

• Other ion channels – IP3 receptors – Other ion channels have not been found.

– Calcium flux through IP3 receptors has not been observed in adult myocytes where they are low in number <5% the number of RyRs.

• Non-junctional RyRs – Non-junctional channels are very small in number less than 5% of the total

number of RyRs. They see bulk myoplasmic calcium which does not reach the high levels needed to trigger calcium release in the diad. Flux through these is likely small.

We propose and alternative mechanism to account for invisible leak …

Model: “Revised Sticky Cluster”

Model Equations

SERCA Models

Model Solution

• RyR open state calculated using our Ultrafast Monte Carlo Method

• Fluxes calculated to determine derivatives

• Differential equations solved using a Euler Method

• Programmed in Fortran 90/CUDA Fortran (Portland Group Compiler) on a HP z800 Linux Workstation with NVIDIA Fermi 2050 GPUs

Ca2+ Dependence of Open Probability

Calcium Spark Mechanism

Calcium Transient

Resting Ca2+ spark behavior

20,000 CRU - 1% plotted

Individual Ca2+ Spark

Spark and Quark Visualization

RyR Opening to Spark Transition

SR Ca2+ Leak - Experiment

(From Zima et al., 2010)

SR Ca2+ Leak - Simulation

Leak Analysis

Effects of Phosphorylation

New Spatial Model Benchmarks

• 20,000 release units

• 49 RyRs

• 6 DHPR

• > 4,000,000 grid elements

• 1 second physiological simulation time

Method

Ultra-fast

Monte Carlo

On GPU

No I/O 3:09 hr

I/O 3:50 hr

Whole-cell Modeling • The cell of size 100x20x18

mm3 is modeled with a

rectangular grid with a mesh

of 0.2 mm

• At each grid point, it contains

calcium in the myoplasm,

calcium in the network

SR.The T-tubule is assumed

to be everywhere.

Whole-cell modelling (cont.)

• Euler method with forward difference in time and

central difference in space

• Neumann boundary condition

Resting Myocyte Activity Experimental Spontaneous Calcium Sparks

Resting Myocyte Activity Simulated Spontaneous Calcium Sparks

Calcium Entrained Arrhythmias Experimental Calcium overload

Calcium Entrained Arrhythmias Simulated Calcium Overload

Conclusions Our Ultrafast Markov chain Monte Carlo method make stochastic

simulation of calcium dynamics possible.

Calcium release initiation occurs stochastically with the opening of one RyR that can trigger additional RyRs to open. One a critical number (~6 RyRs) opens, the remaining channels open causing a spark.

Calcium release termination occurs through a combination of reduced SR Ca2+ that results in reduced RyR opening, stochastic closure, and coupled gating.

Calcium leak is comprised of spontaneous calcium sparks, and the opening of one or a few RyR channels in the release sites (invisible leak).

Propagation between release sites depends upon calcium load, and release site placement.

Co-workers: Modeling Ca2+ sparks and leak

W. Jonathan Lederer

BioMET / Johns Hopkins University

Hoang Trong Minh Tuan

George Mason University

Aristide Chikando

Univ. Maryland Baltimore

George (Blair) Williams

George Mason U. / Univ. Maryland Baltimore

Eric A. Sobie

Mt. Sinai School of Medicine Greg Smith

William & Mary College

• This work was supported by the National Science Foundation, the National Institutes of Health, and

the European Union 7th Framework Program.

• Thanks to Steve Worley for use of his Pseudo-Random Number Generator GPU code.

Traditional Monte Carlo Algorithm

(A) State Diagram with states X, Y, Z and transition probabilities p and q

(B) A Uniform random number [0, 1] determines state transition.

Submitted for Publication and Patent Pending

Properties:

Fast

Exact stochastic method

Low memory usage

Can be used for any Monte Carlo simulation.

Ultra-Fast Markov Chain Monte Carlo

Algorithm

Q-Matrix of Transition Probabilities

Ryanodine Receptor

M=2 states minimal model

x = f(Ca) : Ca-dependent: C → O

y = g(*) : Ca-independent: O → C

Single-channel rate-transition matrix

Chapman-Kolmogorow equation

State Matrix A cluster of RyR

State: (c1, c2) with

c1 = number of Closed RyRs

c2 = number of Open RyRs

E.g: N=5 RyRs

Cluster rate-transition matrix

AR(:,:), BR(:,:) of size 6x6

This reduces the state space from NM states increasing computational efficiency.

)!1(!

)!1(ˆ

MN

NMS

Transition Matrix for Cluster Exact simulation:

In a small time-step, only a SINGLE

channel can change state

NOTE: rate out + rate in = 0

This allows use of vector-matrix algebra to perform Monte Carlo leveraging CPU/GPU design

Adaptive Time Step

• The time step is chosen so that transitions only occur 10% of the time.

• Pmin is the most negative row sum in the transition matrix.

Using the adaptive time step decreases simulation time by about 100x

min1.0min Pt

Heterogenous Cluster Transition Matrix

A heterogeneous cluster:

e.g. release site with DHPR + RyR

• m = # of k-state RyR cluster states

• n = # of j-state DHPR cluster states

Motivation Complexity

50 2-state RyR: cluster of 51 states

7 6-state DHPR: cluster of 924 states

Release site: 47,124 states

Memory demand (double-precision): 16GB

K matrix:

Highly sparse

Question - How to handle the computation with such sparse matrices in the GPU?

Answer - We have developed a novel compact form representation and compact form Kronecker product.

Compact Form Matrix Representation

Compact form for K:

Use two separate matrices:

Kcomp(:,:) = keep ONLY non-zero rate transition

Kidx(:,:) = keep true column index of Kcomp(i,j)

0 12 0 1

0 0 0 2

... ...

... ...

12 1

2 x

2 2 4

1 4 x

Acomp

Aidx A

Compact Form Matrix Representation

• A homogeneous cluster:

– Full matrix

– Compact form

Less number of conditional comparisons

Less memory demand

Compact Form Matrix Representation

• A heterogeneous cluster, e.g. RyR + DHPR:

– Full matrix

– Compact form

• Pros:

– Based upon matrix representation of the release units and their state space derived in part from the theory of solution of stochastic automata networks.

– Uses an adaptive time step.

– Only considers possible transitions from current state of each release units (scales as the number of possible transitions rather than the size of the state space)

– Reduces number of conditional comparisons.

– Amenable to parallelization.

Ultra-fast Monte Carlo Method

• Caveats:

– Reduces number of conditional comparisons by treating

channels as identical so the number of channels in a particular

configuration is counted.

– The transition from one state to another should be single agent

dependent, e.g. either Vm or [Ca], but not both.

– This does not allow for more than one event in a time step. In

our simulations this only occurs ~6% of the time and this

fraction can be reduced if needed.

Ultra-fast Monte Carlo Method

Presentation Outline

1. Introduction to Excitation-Contraction Coupling

2. Calcium Sparks – Experiment and Model

3. Ultrafast Monte Carlo Algorithm

4. GPU Implementation

5. Calcium-Entrained Arrhythmias

6. Conclusions

Computational Model of Resting Ca2+ Dynamics

RyR

Minimal model with 2 states

Release site:

Each site has 50 RyR

Whole-cell model

10,000 release sites

GPU Issues How GPU fit to our problem & algorithm

Highly independent of release site computation Large amount of computation can be done in parallel

State space is reused at every computational step Low memory demand makes it fit to the limited device memory (4GB in Tesla 1060, 3GB in Fermi)

Need to minimize transfers between CPU and GPU due to Memory access latency

Large amounts of I/O