Rapid Data Modeling and TestingAARON BENTON

bit.ly/cbcommunity

Access to Dev Team and other Experts and Champions Brant Burnett (CenterEdge) / Dave Starling (SeenIt)Early Access to BuildsLaura Czajkowski

About MeWho we areWhat we do

+

We use Couchbase and Sync Gateway for a variety of solutionsIncluding: Several mobile applications, media, short urls, session management, looking for more and more use casesWhile Couchbase is a fantastic database, choosing NoSQL presents a new set of challenges

We choose Couchbase for many reasons: Performance, Scalability, all of the reasons that many of you chose CouchbaseIn particular Mobile was a big use caseBut this introduced a whole other set of issues, how do we represent data?

OUR CAPABILITIES

Make fun of different workflows, regardless of flow we still have to deal with data

When you get a new project, what is the first thing your business owners want you to do?

Given ProjectStart DevelopmentWrite Code to Fit DataGather RequirementsRestart Development

EnthusiasmDisillusionmentPanicSearch for the GuiltyPunishment of the Innocent Praise for Non-Participants

Lighten mood with The Six Phases of a ProjectAnyway, moving on data modeling is a critical piece of any project

First, solve the problem. Then, write the code.

John Johnson

Data modeling is a problem that needs to be solved

Data Modeling

Who is data modeling now? What phases? What tools? How do you go about it? Where is it in your process?Checkout the session: Agile document models & data structures with Clarence and Tyler from Couchbase

PhasesConceptual Logical Physical

Conceptual Data ModelingEntity NamesEntity Relationships

Users

Products

Line Items

Orders

Entity NamesEntity RelationshipsDatabase Agnostic

N1PersonBirthplaceLocation

PersonLocation

Born InBirthplace of

PersonLocation

(1, 1) Born InBirthplace of (0, N)

>Person

0. .N1>Location>Born in >< Birthplace ofChen

PersonLocation

IDEF1x

PersonLocation

Born InBirthplace ofBachmanCrow's FootISOUML

Modeling Notations

Logical Data ModelingEntity NamesEntity AttributesEntity RelationshipsUsersUser ID (PK)UsernamePasswordFirst NameLast NameAddressCityStateZip CodeCountryEmailPhone

OrdersOrder ID (PK)Order NumberUser ID (FK)Order DateAddressCityStateZip CodeCountryPhoneEmailCredit CardExpiration Date

Order Line ItemsOrder ID (FK)Product ID (FK)QuantityTotal

ProductsProduct ID (PK)Product NameDescriptionPrice

AttributesPrimary / Foreign Keys: Database Agnostic

Physical Data ModelingEntity Names -> Table NamesEntity Attributes -> Field NamesEntity Relationships -> KeysPrimary / Foreign KeysData Types

NoSql ModelingHow do you represent a model?What tools do you use?How do you show relationships?Does code define the model?What key patterns should be used?

JSON

Our physical models initially were JSONThis worked but was lackingNo ContextNo DescriptionNo Rules

JSON Schema

We also explored JSON Schema

YAML

We moved on to YAMLThis provided:clearer structureless verbosedocumentationdata typingWe stored these models with the code in the repository

Now What?

You've defined your physical NoSQL models, now what?

Starting Development! KiddingWe need data to work withHow many of you have spent time building throw away code, just to have data to work with your applications?

To begin development often times we need a data set to work withWe need pretend or fake data to work with

Manually creating documents is very time consuming and not practical

Fake Data GeneratorsFakerJSChanceJSjson-generator.comobjgen.com/jsonmockaroo.comjsonschema.net

There are a lot of Fake data generators available, some are actually very flexible, but they all only deal with a single model.

Often times we need to relate models

Relationships are Hard

All of these tools only deal with a single a model Or than can only generate data and you still have to write code to use themRelationships are hard

FakeIT

Just FakeIt, now all your problems are solved.

Any fancy logo makers?

FakeitCLI Tool inspired by SwaggerModels defined in YAML Generates JSON, CSON, YAML, CSVExports to Couchbase, Sync Gateway, Zip, Console

Like many of you I've spent countless hours writing code to generate fake dataI'm lazyI wanted to learn N1QL

Fakeit DataExposes FakerJS and ChanceJSCustom Build Functions w/ Pre and PostMultiple ModelsDependenciesDefinitions*fakeit is for structure, cbworkloadgen is for sizing

npm install fakeit -g

If you have NodeJS install it now

Original YAML Model

Our original YAML model

Simple Model

Data Attribute

A "data" attribute is added to each property"description" has been removed just to fit code on the screen

Console Output

- From the directory of the YAML file we run the fakeit command- Which by default will generate 1 document from the model- We can use the -n parameter to specify the number of documents to generate from the model

File Output

We can output each document to an individual file, where the file name is the keyIf the directory does not exist it is created

Output Formats

Using the -o parameter we can specify the file format to use, the default is JSONYAML and CSON are also supported

CSV Output

Another output format that can be used is csvEven if you were using a SQL backend, you can still generate random data and import into that system

Zip Archive

We can take any of these output formats and flush them into a Zip archive using the -a parameterThis Zip file could be imported into Couchbase through cbdocloaderThis is using the default output format of JSON

. . . ?

What about Couchbase?Of course there is Couchbase support, would I be up hear talking about this if there wasn't? :)

Couchbase

We could take the exported Zip archive and import it into Couchbase using cbdocloaderBut wouldn't it be better to stream the generated documents directly into Couchbase?

What about Mobile?

Sync Gateway

We can stream generate documents through the Sync GatewayThis is using Sync Gateway without any authentication

Sync Gateway with Auth

In this example we are using a Sync Gateway that requires authWe specify the username using the -u parameter and the password using the -p parameterA sync session is created prior to the documents being loaded

How does it Work?

Property Data:

valueA static value to be usedExecuted at build time

fakeA FakerJS template stringExecuted at build time

pre_buildFunction body used to initialize a propertyProperty is set to the returned value

buildFunction body used to set a propertyProperty is set to the returned value

post_buildFunction body used to manipulate a property after it has been setProperty is set to the returned value

pre_runFunction body that is executed before any models are generated

post_runFunction body that is executed before after all models have been generated

Data can be generated using any of these methods

You are defining a function bodyDescribe what each of the available variables are

Flow

Model Pre_RUN

Model PRE_BUILD

Initialize Property Value(EACH)

Start Document Run

Property PRE_BUILD + BUILD(EACH)

PROPERTY POST_BUILD (EACH)

MODEL POST_BUILD

MODEL POST_RUN

Define pre / post run

Flow

Build order example using console.logs to show the flow

Flow

Build order example using console.logs to show the flow

Multiple Models

Multiple models in the same directory can be executed together

Multiple Models

Dependencies

UsersProductsReviewsOrders

More often than not our models have dependencies on other models

Dependencies

We specify dependencies on Models

FakeIt resolves the dependencies prior to generation, making the data from previous models available to the current one being generated

Dependencies

By default fakeit looks at the CWD, however you can specify the model directory by using --models

Definitions

Models can get very large, Model definitionsDefinitions can have definitions

Definitions

Often times our models cannot be 100% random data, we have to rely on existing data structures from various systems

continents.json

countries.csvInput

We can provide input to fakeit models using the -i parameter

continents.json

countries.csv

Input

How we use FakeItYAML models store in the repositoryMake commands to generate dataDocker Compose: NginxCouchbaseSync GatewayTesting / QA

Small footprint that can generate a large amount of datacompletely scriptedDevelopers can use and try out various models quickly

Demos

If time permits

Fakeit ExamplesContactsEcommerceFlatFlight DataMusicSimpleSocialgithub.com/bentonam/fakeit-examples

Several Examples also come with multiple N1QL queries

Questions?

HOW TO CONTACT MEDont hesitate to contact with me. I will be happy to answer your questions.

TWITTER: @bentonam

github.com/bentonam

EMAIL: aaron@shop.com

THANK YOU