GPU-AcceleratedBig Data Pipelines for

Desktop, HPC and CloudDr. Melissa Smith1, Ben Shealy1, Josh Burns2, Dr. Alex Feltus3, Dr. Stephen Ficklin2

1 Department of Electrical and Computer Engineering, Clemson University2 Department of Horticulture, Washington State University

3 Department of Genetics and Biochemistry, Clemson University

2

Overview

- KINC

- KINC Nextflow Pipeline

- Running KINC Pipeline on Kubernetes

- Demo

- Challenges / Opportunities

3

The Gene Co-Expression Network (GCN)

4

Knowledge Independent Network Construction (KINC)

→

Gene expression matrix

Gene co-expression network

Similarity matrix

Pairwise scatter plot

↓

5

Human Brain Tissue-Specific Network

6

Human Kidney Tumor-Specific Network

7

KINC GPU Implementation

Shealy, Burns, et. al., “GPU Implementation of Pairwise Gaussian Mixture Models for Multi-Modal Gene Co-Expression Networks”, IEEE Access

8

KINC Pipeline

9

Pipeline Portability with Nextflow

# Local

nextflow run systemsgenetics/KINC-nf -with-docker

# HPC

nextflow run systemsgenetics/KINC-nf -profile pbs -with-singularity

# Kubernetes

nextflow kuberun systemsgenetics/KINC-nf -v <pvc-name>

10

Running Genomics Pipelines on Kubernetes

11

Integrating Nextflow / Kubernetes / GPUs

nextflow.config

process {

cpus = 4

memory = 8.GB --->

accelerator = 1

}

pod.yaml

resources:

limits:

cpu: 4

memory: "8Gi"

nvidia.com/gpu: 1

12

DemoKINC / Nextflow / Kubernetes / NVIDIA GPUs

13

Challenges and Opportunities

- Using Kubernetes as an HPC platform- Transient failures (network timeouts, node-specific issues)

- Scheduling policies (walltime)

- Access controls (users)

- “SLURM-enetes” / “KubLURM”

- Scale up

- Increase usability for domain scientists- Friendly user interface

- Assist user with machine learning 14

Thank you! Questions?Research funded by NSF Grant #1659300

15

Nextflow API / Workflows

16

Nextflow API / Workflow Instance

17

Grafana / Cluster Usage

18

Grafana / GPU Usage

19