Pipe[r] Dreams: Making NoSQL Great

Again

Kathryn Dahlgren DisorderlyLabs@UCSC

HPTS, 10Oct2017

TL;DR• NoSQL is great, but could be better.

• Problem: The k-implementation of database management functionality across the total set of instances of

NoSQL products around the world is sub-optimal.

• Goals: 1. Motivation? 2. Articulate solution qualities. 3. Present Piper as a model.

2

BACKGROUND+

MOTIVATION

4

Ancestral Database Systems

5

Ancestral Database Systems

6

Ancestral Database Systems

7

Ancestral Database Systems

8

Database Management Systems

91980s-1990s

101980s-1990s

1990s11

Early 2000s12

• Revolution!• Coincides w/ rise of Internet + code repos (SourceForge, Github, …)• 200+ NoSQL tools in existence (http://nosql-database.org/index.html)

2000s13

• Flexibility

• Customizability

What makes NoSQL Great?

2010s14

15

16

17

Problem

• Raw NoSQL systems are Database Systems (in the common case)

• Transforming a NoSQL product into a Database Management System necessitates engaging in a k-implementation nightmare with all the other NoSQL users in the world.

Valid Solution Qualities

Thou Shalt Not :1. impede flexibility

2. jeopardize customizability

3. hamper dissemination

18

19

Piper

• Developers working with NoSQL systems can: 1. install management packages from the index. 2. publish management packages to the index.

• Usage standards impose strict regulations on the ease of package installation + deinstallation.

• Inspired by • Package indexes (PyPI, NPM, …) • Source code repositories (Github, BitBucket, …)

20

Piper in action!

aggsPack

SuperApp

piper_pickledb

Devyn

MongoDB

21

Piper in action!piper_pickledb

Piper SuperApp

piper_mongodb

MongoDB

Devyn

aggsPack

22

Piper in action!piper_pickledb

Piper SuperApp

piper_mongodb

simpleJoinMongoDB

Devyn

aggsPack

piper_mongodb

MongoDBsimpleJoin

23

Piper in action!piper_pickledb

aggsPack

Piper SuperApp

Devyn

piper_mongodb

MongoDBsimpleJoin

24

Piper in action!piper_pickledb

aggsPack

Piper SuperApp

Devyn

25

Piper in action!piper_mongodb piper_pickledb

simpleJoin

Piper SuperApp

piper_mongodb

MongoDB

Devyn

simpleJoin

aggsPack

26

Piper in action!piper_mongodb piper_pickledb

simpleJoin

SuperDuperApp Piper SuperApp

piper_mongodb

Eunice

MongoDBPickleDB

Devyn

simpleJoin

aggsPack

27

Piper in action!piper_mongodb piper_pickledb

simpleJoin

Piper SuperDuperApp

simpleJoin

Piper SuperApp

piper_mongodb

Eunice

MongoDBPickleDB

piper_pickledb

Devyn

simpleJoin

aggsPack

28

Piper in action!piper_mongodb piper_pickledb

simpleJoin

PickleDB

Piper SuperDuperApp

MongoDB

Piper SuperApp

piper_mongodb

simpleJoin

piper_pickledb

Eunice Devyn

simpleJoin

aggsPack

Devyn

29

Piper in action!piper_mongodb piper_pickledb

simpleJoin

PyMongo

Piper SuperApp

piper_mongodb

MongoDB

Piper SuperDuperApp

simpleJoin

Eunice

PickleDBsimpleJoin

piper_pickledb

aggsPack

Valid Solution!

30

WISH LIST :1. Promotes flexibility

2. Champions customizability

3. Eases solution dissemination

Valid Solution!

31

WISH LIST :1. Promotes flexibility

2. Champions customizability

3. Eases solution dissemination

Valid Solution!

32

WISH LIST :1. Promotes flexibility

2. Champions customizability

3. Eases solution dissemination

Valid Solution!

33

WISH LIST :1. Promotes flexibility

2. Champions customizability

3. Eases solution dissemination

Project Status• Bare-bones proof-of-concept.

• Supports two NoSQL DBSs : • MongoDB (document store) • PickleDB (key-value store)

• Contains two packages : • aggsPack • simpleJoin

https://github.com/PiperProject

34

Research Directions???

Research Directions?

36

Research Directions?

37

Automatic adapter

generation?

Research Directions?

38

Automatic package optimization per target environment?

Automatic adapter

generation?

Research Directions?

39

Automatic package optimization per target environment?

Automatic adapter

generation?

Generalizability standards?

Conclusion• NoSQL is here to stay • No rules != totally awesome • k-implementation is sub-optimal • Piper is a proof-of-concept tool modeling a valid

solution

40

Conclusion• NoSQL is here to stay • No rules != totally awesome • k-implementation is sub-optimal • Piper is a proof-of-concept tool modeling a valid

solution

Making NoSQL great :-

Being stronger together!

41

Thanks!

42