PREFACE

Commandlineclientsareeverywhere.Almosteveryone,atleastintech,isusingthem.

Therearealotofsuccessfulcommandlineclientsoutthere:theLinuxprojecthasgitandthe Node.js project has npm. We use some of them multiple times per day. ApacheCouchDBrecentlygotnmo(speak:nemo),atooltomanagethedatabasecluster.Wecanlearnalotfromsuccessfulcommandlineinterfacesinordertowritebettercommandlineclients.

WhenIstartedtoget interestedincommandlineclientsIrealisedthat therearealotofdiscussionsandinformationsonthewebaboutwritingAPIs.Thewebisfulloftutorialstoteach you how to build APIs, especiallyREST-APIs, but almost nothing can be foundaboutwritinggoodCLIs.ThisbooktriestoexplainwhatmakesagoodCLI.InthesecondpartofthebookwewillbuildasmallcommandlineclienttolearnhowtouseNode.jstocreategreatcommandlineclientsthatpeoplelove.

Thegoalofthebookistoshowtheprinciplestobuildasuccessfulcommandlineclient.The provided code should give you a good understanding what is important to buildsuccessfulcommandlineclientsandhowyoucouldimplementthem.

Everysectionhasitsowncodeexamples.Beforeyourunthecode,youhavetorunnpm

installinthefolderthatbelongstothesection.

Youcandownloadthecodesamplesathttp://theclibook.com/material/sourcecode.zip.

I trust you and published this book without DRM. Please buy a copy athttp://theclibook.comincaseyoudidnotbuythebook.

I am very happy about feedback. Please send and feedback or corrections totheclibook@kowalski.gd.Youcanalsocontactmeontwitter:@robinson_k

Ihopeyouenjoythebook–pleaserecommenditincaseyoulikeit. �

▪

▪

▪

WHATMAKESAGOODCLI?

Inthischapterwewilltakealookatsuccessfulcommandlineclientsandwhattheyaredoing pretty well, which will help us to understand the problems users face using theTerminal.UnderstandingtheproblemsofouruserswillhelpustobuildbettercommandlineclientswithNodelaterinthebook.

Let’stakealookathowpeopleusuallyuseaCLI:mostofthetimeahumansitsinfrontof a keyboard and interacts with a terminal. We want to use simple and recognisablecommandsforourCLI.Sadlyjusteasyrecognisablecommandsdon’tgetusveryfarrightnow.

Maybetheproblemiseasier tounderstandifwetakea lookatsomethingwhatIwouldcallabadCLI:

$mycli-A-a16rfoo.html

error:undefinedisnotafunction

Inmy example I have to enter cryptic commandswhich is answered by a very crypticerrormessage.Whatdoes -A -a16 and rmean?Why I amgetting an error back, am Iusingitwrong?WhatdoestheerrormeanandhowcanIgetmytaskdone?

SowhatmakesagoodCLI?Let’stryitwiththefollowingthreeprinciples:

younevergetstuck

itissimpleandsupportspowerusers

youcanuseitforallthethings!

Inshort:AsuccessfulCLIissuccessfulbecauseitsusersaresuccessfulandhappy.

YounevergetstuckNobodylikestobeinatrafficjam,stuck,justmakingafewmetersperminute.Wewantto reachour targetdestination, that’sallwewant!Thesameapplies forourusers.Bothdevelopersandusersareextremelyunhappywhenthetoolstheyusearestandingintheirway.Theyjustwanttogettheirtaskdone.

Sowhatdoes,„Younevergetstuck“mean,exactly?Itmeansthatweshouldalwaysofferourusersawaytosolvetheirtask,acommandshouldneverbeadeadend.Additionally

thedevelopersoftheCLIshouldavoideverysourceoffrictionintheirtool.

Let’stakealookatme,tryingtousegit:

$gitpoll

git:'poll'isnotagitcommand.See'git--help'.

Didyoumeanthis?

pull

InthisexampleIenteredawrongcommand.gitanswersfriendly:„HeyRobert, it lookslikeyouenteredawrongcommand,butifyoutypeingit--help,youcanlistallthe

existingcommands.Andhey, it just looks likeyoumistypedgitpull, didyoumeangitpull?“

gitoffersusawaytocontinueourworkandfinishthetask.

Andifwetakealookatnpm,anothersuccessfulCLIclient,we’llseethesameconcept:

$npmragrragr

Usage:npm<command>

where<command>isoneof:

access,add-user,adduser,apihelp,author,bin,bugs,c,

cache,completion,config,ddp,dedupe,deprecate,dist-tag,

dist-tags,docs,edit,explore,faq,find,find-dupes,get,

help,help-search,home,i,info,init,install,issues,la,

link,list,ll,ln,login,logout,ls,outdated,owner,

pack,prefix,prune,publish,r,rb,rebuild,remove,repo,

restart,rm,root,run-script,s,se,search,set,show,

shrinkwrap,star,stars,start,stop,t,tag,test,tst,un,

uninstall,unlink,unpublish,unstar,up,update,upgrade,

v,verison,version,view,whoami

npm<cmd>-hquickhelpon<cmd>

npm-ldisplayfullusageinfo

npmfaqcommonlyaskedquestions

npmhelp<term>searchforhelpon<term>

npmhelpnpminvolvedoverview

Specifyconfigsintheini-formattedfile:

/Users/robert/.npmrc

oronthecommandlinevia:npm<command>—keyvalue

Configinfocanbeviewedvia:npmhelpconfig

npm@2.7.4/Users/robert/.nvm/versions/node/v0.12.2/lib/node_modules/npm

In thisexample I try toputgarbage intonpm,sonpmanswers friendly:„HeyRobert, Idon’tknowthatcommand,buthereareallthecommandsthatwouldbepossible.Youcanusethemlikethisandgethelpaboutthembytypinginnpmhelp<command>.“

Likegit,npmimmediatelyoffershelp toenablemetofinishmytask,evenif Ihavenoideahowtousenpmatall.

Stilllost?WhatifIstillneedhelp?MaybeIwanttogetsomehelpbeforeIjusttryoutcommands.Turns out there is a quite reliableway to ship documentation onUnix or Linux,man-pages!

Figure1.Theman-pageforgitpull

Man-pagesarequitenice,asyoudon’tneedtheinternettoopenthem.Youcanalsostayin thesameterminalwindowtoreadthemanddon’thavetoswitchtoanotherwindow,e.g.abrowser.

Butsomeusersdon’tknowaboutman-pagesortheydon’tliketousethem.AdditionallymanyofthemwillbeonWindowswhichcan’thandleman-pagesnatively,sogitandnpmoffertheirdocumentationaswebpages,too:

Figure2.Thedocumentationwebsiteofthegitproject

Bothgitandnpmaremakinguseofatrick:theywritetheirdocumentationonce(e.g.inMarkdownorAsciidoc)andusetheinititalsourceasthebaseforthedifferentformatsoftheirdocs.Latertheyconvertthemtodifferentformats,e.g.tohtml.

Ifyoutakealookattheman-pagesofgitandnpm,youwillnoticethattheirwebsitesarebasicallyframingthecontentfromtheman-pagewithaheaderandasidebar.

Figure3.Themanpagefornpmpublish

Figure4.Thedocumentationwebsiteofnpm

ErrorhandlingSometimesthingsgostillhorriblywrong…Let’stakealookatmyexampleforabadCLIagain:

$mycli-A-a16rfoo.py

events.js:85

thrower;//Unhandled'error'event

^

Error:ENOENT,open'cli.js'

atError(native)

Inthiscasewearegettingbackastacktracewithoutmuchcontext.Formostpeoplethesestacktraces look quite cryptic, especially for people that don’twriteNode.js on a dailybasis.

Anditisevenworse:Ireallycan’ttellifIjusthitabuginthecommandlineclientorifIamjustusingtheCLIinawrongway.Lookingatthatsmallterminal,withnoideawhattodo,Igetextremelyunhappyandsoouruserswillgetunhappy.

Onethingnmosupportsis„usageerrors“—hereiswhattheylooklike:

$nmoclusterdsf

ERR!Usage:

nmoclusterget[<clustername>],[<nodename>]

nmoclusteradd<nodename>,<url>,<clustername>

nmoclusterjoin<clustername>

Ifausertriestouseacommandinawrongway,nmowilltellthemimmediatelyhowtheycanusethecommandtogettheirjobdone.Noneedtoopenthedocumentation.

nmoalsoshowsstacktracestoauser,ifnmocrashesforseriousreasons:

$nmoclusterjoinanemone

ERR!dfisnotdefined

ERR!ReferenceError:dfisnotdefined

ERR!at/Users/robert/apache/nmo/lib/cluster.js:84:5

ERR!atcli(/Users/robert/apache/nmo/lib/cluster.js:68:27)

ERR!at/Users/robert/apache/nmo/bin/nmo-cli.js:32:6

ERR!

ERR!nmo:1.0.1node:v0.12.2

ERR!pleaseopenanissueincludingthislogon

https://github.com/robertkowalski/nmo/issues

nmoaddsthecurrentnmoandnodeversiontothestacktrace,likenpmdoes.Wealsoasktheusertocopythestacktraceandtoopenanissuecontainingthestacktrace.

The reports make it easy for the team to identify the bug, solve it, and release a newversionofnmobyseeingthestacktrace.

Andagaintheuserisnotstuck.Theusergetshelptosolvetheirtask,intheworstcasewehelptheminourissuetracker.

ItsupportspowerusersPowerusersareimportantforyourCLI.TheyareuserswhowilltalkaboutyourCLIandraisetheoveralladoptionbyspreadingtheword.

ShortcutsMostpowerusersareusingyourCLImultipletimeseveryday.Aneasywaytosupportthemisbyprovidingshortcuts.

npmhaslotsofshortcuts.Forinstance,npmiistheshortformfornpminstall.git

letsyoudefineyourownshortcutsinthe.gitconfigfile.Iusegitcoasashortcut

forgitcheckout,forexample.

Scripting

Atsomepointyourcommandlineclientwillgetquitesuccessful,peoplearelovingitandstart usingyourCLI in creativeways.The command line clientwill suddenly runon aJenkins,aspartof theirdeployment inacheforPuppet run,oryouruserswilluse it inwaysyounevercouldhaveimagined!

SoonerorlaternotonlyhumanswilluseyourCLI,butalsoautomatedprocesses.TomakeyourCLIevenmoresuccessfulit’sagoodideatosupportscripting.

exitcodesOperatingsystemshavedifferentexitcodestosignalifacommandwassuccessful.Youwillgetbacka0ifyourrecentcommandwassuccessful.1wouldbeageneralerror.

Exitcodesareveryusefulforusersthatwanttowrapyourcommandlineclientinabashscript.

Hereisanexample:

$gitpoll

git:'poll'isnotagitcommand.See'git--help'.

Didyoumeanthis?

pull

$echo$?

1

git notifiesme that something went wrong - I am getting back a1 as exit code.With

properexitcodeseverywriterofabashscriptcanhandlethecaseswhereacommandisnotsuccessful.

JSONoutputInnmoeverycommandthatgivesbackinformationsupportsjson-formattedoutput:

$nmoclusterget--json

{anemone:

{node1:'http://node1.local',

node2:'http://node2.local',

node3:'http://node3.local'}}

JSONsupport enablesuser toprocessdata easily in theprogramming languageof theirchoice,asmostlanguagessupportJSON.Theyspawnachildprocessinlanguagexand

listentostdoutfortheoutput.Theycanalsodirectlypipetheoutputintoaconsumeron

theshell:

$nmoclusterget--json|consumer.py

JSONoutputgivesalotofflexibilitytotheusers.

TheAPIintheCommandLineClientThereisanotherconcepttomakescriptingeasier:Icallitthe„TheAPIintheCommandLineClient“:

constnmo=require('nmo');

nmo.load({}).then(()=>{

nmo.commands.cluster

.get('testcluster','node1@127.0.01')

.then((res)=>{

console.log(res);

});

});

Innmoeverycommand is exposedonnmo.commands. If auserwants tousenmoas

partof theirnode scripts, theyare able to require it.The JavaScriptAPI isdocumentedliketheCLI.

TheJavaScriptAPIenablestheuserstoembednmointheirNode.jsscriptsforcomplexprocesses.Theycouldevenforknmoandembeditintotheirowncommandlineclient.

ConfigurationPowerusers love configuration. Given they use a command line client a lot, maybemultipletimesaday,itisnosurprisethattheywouldliketohavesomefeaturesenabledperdefault.Butinrarecases,thereisanexceptionandtheydon’tneedthedefaultsetting.

npmsupportsoptionargumentsonthecommandline:

$npmihapi--registry=https://reg.kowalski.gd

/Users/robert

└──hapi@9.0.4

This command tries to download the packagehapi from a private registry at

https://reg.kowalski.gd.

ButIcanalsosetthisprivateregistryasthenewdefaultregistry:

$npmconfigsetregistryhttps://reg.kowalski.gd

npmwritesthenewregistryintotheconfig:

$cat~/.npmrc

loglevel=http

registry=https://reg.kowalski.gd

The next time I try to install a package, npm will use my new default registry,https://reg.kowalski.gd:

$npmihapi

IfIdon’twanttousethisnewdefaultregistryIcanpassanargumenttotheCLIanditwillusethealternateregistryjustforthiscall:

$npmihapi--registry=https://registry.npmjs.org

/Users/robert

└──hapi@9.0.4

Thatmeanswehavedifferentprioritiesbetweendefaultconfigurationsandcommandlineargumentsinnpmandthiscombinationisextremelypowerful.

Youcanuseitforallthethings!Let’s take a look at the last principle, and the solution to it sounds very easy at first.WheneverIhavetodoataskmultipletimesanditfitsintothedomainofmycommandlineclient, I’ll justadd it asanewcommand.Thishabit turns intoawin-winsituation:Youhavetodolessboringtasksandyourusersgethappybecausetheygetanewfeatureandtheyalsohavetodolessmonkeytasks-makingyourcommandlineclientevenmoresuccessful.Sadlyitcanbequitehardtospotcommonpainpoints,especiallyifyouworkwithmultiple teams and/or a lot different people.Additionallymost of us are sufferingorganisationalblindnessworkingonthesametopicafteracertaintime.Butifyouidentifyatasktoautomateforyouandyourusers,youwillbehugelyrewarded! �

▪

▪

▪

WRITINGADATABASEADMINISTRATIONTOOLWITHNODE.JS

In thispartof thebookwewillwriteadatabaseadministration toolnamed loungerandfollowtheprinciplesthatmakeagoodCLI.Thecodeforeverysectioncanbefoundinthezipfile thatcomeswiththebook.Ifyouwanttoplaywiththecode,don’tforget torunnpminstallinthefolderofthesectionyouwanttotest.Thecodealsohasatestsuite,

youcanrunitwithnpmtest.

Afterwewrote the code that bootstraps the client, you can run the client directlywithnodebin/lounger-cli.Ifyouprefertoruntheclientlikeaninstallationfromthe

registry,typenpmlinkinthedirectoryofthesectionyouwanttotest.Afterwardsyou

canruntheversionthatiscurrentlylinkedwithlounger.

WhyuseNode.js?IsometimesgetaskedwhyIwritecommandlineclientsusingNode.js.Formethemainreasonsare:

ahugeecosystemwithmodulesineveryflavour

veryfastdevelopmentspeed

writingJavaScriptisfun!

For me these three reasons make Node.js the perfect platform to write command lineclients.

SetupWewillwriteourcommandlineclientinES2015(alsoknownasES6),whichisthemostrecent version of JavaScript. In order to use it, we have to install Node v4 fromhttps://nodejs.org. If youwant to support olderNode.js versions, I can recommend theBabel transpiler to transpile ES6 code to ES5 compatible code. You can get Babel athttps://babeljs.io/.

The tool that we’ll write will be a small database administration tool for CouchDB /PouchDB.Therearemultiplewaystogetadevelopmentdatabaseserverupandrunning.

OnewayistoinstallErlangandCouchDBforyourOperatingSystem.Youcandownloadofficial packages athttp://couchdb.apache.org and many Linux distributions haveCouchDBintheirpackagerepository,too.

I think the easiest way is to use the PouchDB Server that is available in the attachedsourcecode for thebookor togetaCouchDBinstanceathttps://cloudant.comwhich isfreeuntilyouhitalimit.

UsingthePouchDBdatabaseserverThedatabaseserverislocatedinsourcecode/database,inordertouseitwehaveto

installtheneededdependencies:

$cdsourcecode/database

$npminstall

Tobootthedatabasewejustrun:

$npmrunstart

WecannowinteractwiththedatabaseserverviaHTTP,asCouchDBandPouchDBaredatabaseswithanHTTPAPI:

$curl-XGEThttp://127.0.0.1:5984/

{"express-pouchdb":"Welcome!","version":"1.0.1","vendor":{"name":"PouchDB

authors","version":"1.0.1"},"uuid":"4fad2c01-ba32-4249-8278-8786e877c397"}

Let’screateadatabasecalledpeople:

$curl-XPUThttp://127.0.0.1:5984/people

{"ok":true}

Wecannowinsertdocumentsintoourdatabasepeople:

$curl-XPOSThttp://127.0.0.1:5984/people-d'{"name":"RockoArtischocko",

\

"likes":["Burritos","Node.js","Music"]}'-H'Content-Type:

application/json'

{"ok":true,"id":"21b5ad83-0ad6-47c7-86f8d9636113160a","rev":"1-

411894affa038a6fd7a164e1bfd84146"}

Usingtheidwecanretrievethedocumentsfromthedatabase:

$curl-XGEThttp://127.0.0.1:5984/people/21b5ad83-0ad6-47c7-86f8-

d9636113160a

{"name":"RockoArtischocko","likes":

["Burritos","Node.js","Music"],"_id":"21b5ad83-0ad6-47c7-86f8-

d9636113160a","_rev":"1-411894affa038a6fd7a164e1bfd84146"}

Great!Wehaveadatabaseupandrunning!

TroubleshootingGettingcurlcurl is a command lineclient forHTTP requests. It is available forallmajorOperatingSystems.OSXuserscaninstallitusingbrewandforWindowsthereareWindowsbuilds

availableathttp://curl.haxx.se/download.html.

FilewatchersOnLinuxIgotanerrorbecausemyuseralreadywatchedtoomuchfiles:

$npmrunstart

>theclibook-database@1.0.0start/home/rocko/clibook/sourcecode/database

>pouchdb-server--in-memory

fs.js:1236

throwerror;

^

Error:watch./log.txtENOSPC

atexports._errnoException(util.js:874:11)

atFSWatcher.start(fs.js:1234:19)

atObject.fs.watch(fs.js:1262:11)

atTail.watch

(/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-

server/node_modules/tail/tail.js:83:32)

atnewTail

(/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-

server/node_modules/tail/tail.js:72:10)

at/home/rocko/clibook/sourcecode/database/node_modules/pouchdb-

server/lib/logging.js:69:20

atFSReqWrap.cb[asoncomplete](fs.js:212:19)

npmERR!Linux3.13.0-71-generic

npmERR!argv"/home/rocko/.nvm/versions/node/v4.2.3/bin/node"

"/home/rocko/.nvm/versions/node/v4.2.3/bin/npm""run""start"

npmERR!nodev4.2.3

npmERR!npmv3.5.1

npmERR!codeELIFECYCLE

npmERR!theclibook-database@1.0.0start:`pouchdb-server--in-memory`

npmERR!Exitstatus1

npmERR!

npmERR!Failedatthetheclibook-database@1.0.0startscript'pouchdb-server

--in-memory'.

npmERR!Makesureyouhavethelatestversionofnode.jsandnpminstalled.

npmERR!Ifyoudo,thisismostlikelyaproblemwiththetheclibook-

databasepackage,

npmERR!notwithnpmitself.

npmERR!Telltheauthorthatthisfailsonyoursystem:

npmERR!pouchdb-server--in-memory

npmERR!Youcangetinformationonhowtoopenanissueforthisproject

with:

npmERR!npmbugstheclibook-database

npmERR!Orifthatisn'tavailable,youcangettheirinfovia:

npmERR!npmownerlstheclibook-database

npmERR!Thereislikelyadditionalloggingoutputabove.

npmERR!Pleaseincludethefollowingfilewithanysupportrequest:

npmERR!/home/rocko/clibook/sourcecode/database/npm-debug.log

Ifixeditwithbyraisingthelimitusingthiscommand:

$echofs.inotify.max_user_watches=524288|sudotee-a/etc/sysctl.conf&&

sudosysctl-p

AsimplestatuscheckOurfirstcommandwillcheckifthedatabaseisupandrunning.Ouruserscantakealookifthedatabaseserverisrunningandwecanusethecommandinternallyforthecommandswhichrequirearunningdatabase.

Thecommandtocheckifadatabaseserverisonlinewilllooklikethis:

$loungerisonlinehttp://192.168.0.1:5984

http://192.168.0.1:5984isupandrunning

TheAPIwouldlooklikethis:

$lounger.commands.isonline('http://example.com')

GettingstartedfromscratchTo get startedwe have to create apackage.json file. Luckily npm provides a nice

assistanttocreatethose:

$npminit

Wethenjustanswerthequestionsnpmasksus.

Figure1.Theassistantfromnpminittocreateapackage.json

Additionallywehavetocreatethreefolders:test,libandbin.testwillcontainour

unitandintegrationtests,libwillcontainthecoreofourcommandlineclient.Thebin

folderwillcontainasmallwrapperthatwillbootupthecoreofourclient.

CouchDBandPouchDBbothreturnawelcomemessagewhenweaccess therooturlathttp://localhost:5984

$curllocalhost:5984

CouchDBreturns:

{"couchdb":"Welcome","uuid":"17ed4b2d8923975cf658e20e219faf95","version":"1.5

.0","vendor":{"version":"14.04","name":"Ubuntu"}}

PouchDBreturns:

{"express-pouchdb":"Welcome!","version":"1.0.1","vendor":{"name":"PouchDB

authors","version":"1.0.1"},"uuid":"4fad2c01-ba32-4249-8278-8786e877c397"}

Chooseyourownflavours

Wewillmakeuseofthisbehaviourtocheckifthedatabaseisonline.

As already mentioned inWhy use Node.js? Node.js has a great ecosystem. There aremanybattleprovenmodulesthathelpustosolveourtasks.

For our status check we will use the modulerequest to handle our HTTP requests.

mocha will run our testsuite andnock helps us tomockHTTP responseswithout the

needtobootadatabaseinstanceforthetestsuite.

The arguments--save and--save-dev will add the packages to the

dependencies anddevDependencies section of ourpackage.json.

devDependencies are needed just for development, not for running the package in

production:

$npmi--saverequest

$npmi--save-devmochanock

Afterrunningthecommandsweshouldhaveeverythingwewillneedfornow.

There aremany good test runners for Node.js, some alternatives tomocha are the npm modulestap,

tapeorlab

Mypackage.jsonlookslikethisnow:

{

"name":"lounger",

"version":"1.0.0",

"description":"atoolforcouchdb/pouchdbadministration",

"main":"lib/lounger.js",

"directories":{

"test":"test"

},

"dependencies":{

"request":"^2.67.0"

},

"devDependencies":{

"mocha":"^2.3.4",

"nock":"^5.2.1"

},

"scripts":{

"test":"mocha-Rspec"

},

"keywords":[

"couchdb",

"pouchdb"

],

"author":"RobertKowalski<rok@kowalski.gd>"

}

Thisbookisnotfocussedondifferentdevelopment techniques likeTDD,but ifyouarereally intoTest-Driven-Development, you canwrite failing testswithmocha beforeweimplementtheactualcode.Afewsuggestions:

1. itdetectsifthedatabaseisonline

2. itdetectsofflinedatabases

3. itdetectsifsomethingisonline,butnotaCouchDB/PouchDB

4. itonlyacceptsvalidurls

WritteninmochaandES6wegetafewfailingtestsintest/isonline.js:

'usestrict';

constassert=require('assert');

constnock=require('nock');

describe('isonline',()=>{

it('detectsifthedatabaseisonline',()=>{

assert.equal('foo','toimplement');

});

it('detectsofflinedatabases',()=>{

assert.equal('foo','toimplement');

});

it('detectsifsomethingisonline,butnotaCouchDB/PouchDB',()=>{

assert.equal('foo','toimplement');

});

it('justacceptsvalidurls',()=>{

assert.equal('foo','toimplement');

});

});

Torun the testsuite,wehave to typeeithernpmtest ornpmton the terminal.The

codeofthissectioncanbefoundinsourcecode/client-boilerplate.

TheinternalsofthecommandLet’screateandeditthefilelib/isonline.js.Thefilenameisimportant,aswewill

usethenameofthefilelaterduringthebootstrapoftheclient.Asafirststep,wehavetorequireourdependencyrequest:

'usestrict';

constrequest=require('request');

TomakearequestwecreatethefunctionisOnlinewhichwilltakeanurlandsendthe

request:

functionisOnline(url){

returnnewPromise((resolve,reject)=>{

request({

uri:url,

json:true

},(err,res,body)=>{

IfthereisnoHTTPserviceatalllisteningonthespecifiedurlweresolvethePromisewithanobjectwithcontainstheurlasakey,andfalseasavalue:

if(err&&(err.code==='ECONNREFUSED'||err.code==='ENOTFOUND')){

returnresolve({[url]:false});

}

ForallothererrorswerejectthePromise:

//anyothererror

if(err){

returnreject(err);

}

If we get aWelcome from CouchDB or PouchDB, we can safely assume that the

databaseserverisonline:

//maybewegotawelcomefromCouchDB/PouchDB

constisDatabase=(body.couchdb==='Welcome'||

body['express-pouchdb']==='Welcome!');

returnresolve({[url]:isDatabase});

Asalaststepwehavetoexportthefunction:

exports.api=isOnline;

Hereisthewholefunction:

functionisOnline(url){

returnnewPromise((resolve,reject)=>{

request({

uri:url,

json:true

},(err,res,body)=>{

//dbisdown

if(err&&(err.code==='ECONNREFUSED'||err.code==='ENOTFOUND')){

returnresolve({[url]:false});

}

//anyothererror

if(err){

returnreject(err);

}

//maybewegotawelcomefromCouchDB/PouchDB

constisDatabase=(body.couchdb==='Welcome'||

body['express-pouchdb']==='Welcome!');

returnresolve({[url]:isDatabase});

});

});

}

exports.api=isOnline;

WecantryourfunctionontheNode.jsREPL:

$node

>constisonline=require('./lib/isonline.js');

undefined

>isonline('http://example.com').then(console.log);

Promise{<pending>}

>{'http://example.com':false}

>isonline('http://doesnotexist.example.com').then(console.log);

Promise{<pending>}

>{'http://doesnotexist.example.com':false}

>isonline('http://localhost:5984').then(console.log);

Promise{<pending>}

>{'http://localhost:5984':true}

Congratulations!Wejustfinishedthefirstpart,theAPIpartofournewcommand!

TheCLIpartItwouldbefrustratingfortheenduserifthatwasthecommandlineinterfacetheyhavetouse.Theoutput is not easily readable and the functionalitynot easy tounderstand.TheAPIfunctiondoesn’tprinttotheconsole,whichisperfectforanAPI,butnotdesirableforaCLI.ItevenrequiressomeNode.jsknowledgetorunit.Solet’saddaniceCLIfunctiontoourfileisonline.jsandexportitascli:

functioncli(url){

returnnewPromise((resolve,reject)=>{

});

}

exports.cli=cli;

Thefunctionreturnsapromiselikeallourotherfunctions.WethencallisOnlineand

printtheresultonstdoutfortheusersthatusethecommandlineclientontheterminal:

isOnline(url).then((res)=>{

Object.keys(results).forEach((entry)=>{

letmsg='seemstobeoffline';

if(results[entry]){

msg='seemstobeonline';

}

//printonstdoutforterminalusers

console.log(entry,msg);

resolve(results);

});

});

functioncli(url){

returnnewPromise((resolve,reject)=>{

isOnline(url).then((res)=>{

Object.keys(results).forEach((entry)=>{

letmsg='seemstobeoffline';

if(results[entry]){

msg='seemstobeonline';

}

//printonstdoutforterminalusers

console.log(entry,msg);

resolve(results);

});

});

});

}

exports.cli=cli;

Itisimportanttonote:weexportthecommandfortheAPIasexports.api,whilewe

exporttheCLIcommandunderthecliproperty.

Thecodeforthissectioncanbefoundatsourcecode/the-status-check.

BootingthetoolLounger still needs the code thatmakes it usable on the command line. It also lacks acomfortablewaytorunourAPIcommands.Rightnowwejusthaveonecommand,butwe’lladdmoresoon(see[_you_can_use_it_for_all_the_things]).Tomakeallcommandseasy to use, we have to load all available commands into a namespace. The filelib/lounger.jswilltakecareofit.

Thefilelounger.jsistheheartofourcommandlineclient.Werequirethefsmodule

aswehavetolistthefilesthatcouldcontaincommmands:

'usestrict';

constfs=require('fs');

We require thepackage.json and expose the current version of the module as a

property on the lounger object, which will come in handy later. We also setlounger.loadedtofalse:

constpkg=require('../package.json');

constlounger={loaded:false};

lounger.version=pkg.version;

Wewill need a place to store theAPI and theCLI commands.As the bootstrapping isasync,wewillthrowanerrorifsomeonetriestoaccesstheexposedfunctionsbeforethe

bootstrapisfinished:

constapi={},cli={};

Object.defineProperty(lounger,'commands',{

get:()=>{

if(lounger.loaded===false){

thrownewError('runlounger.loadbefore');

}

returnapi;

}

});

Object.defineProperty(lounger,'cli',{

get:()=>{

if(lounger.loaded===false){

thrownewError('runlounger.loadbefore');

}

returncli;

}

});

ThecustomgetterforObject.definePropertywillthrowiflounger.loadedis

falseandwetrytoaccessapropertyonlounger.cliandlounger.api.

It also works for the autocomplete in the Node.js REPL when you try to hit tab forcompletion.Justtryit!

The last part of the file is the actual bootstrapping, where we get all files inlib and

requirethemiftheyareJavaScriptfilesandnotlounger.js, thefilewearecurrently

workingwith.

Thefunctiontobootstraploungeriscalledlounger.load.Inthiscaseweareusinga

named function.A named function can be very helpful in a stacktrace in case the appcrashes:

lounger.load=functionload(){

returnnewPromise((resolve,reject)=>{

});

};

Thefunctionfs.readdirwilllistallfilesinadirectory.Weiterateoverthelistoffiles

fs.readdirreturns:

lounger.load=functionload(){

returnnewPromise((resolve,reject)=>{

fs.readdir(__dirname,(err,files)=>{

files.forEach((file)=>{

});

});

});

};

IfthefileisnotaJSfileorislounger.jsitself,weignoreitbyreturningearly:

if(!/\.js$/.test(file)||file==='lounger.js'){

return;

}

Inallothercasesweassumethatwefoundacommandforlounger.Wetakeeverythingfromthefilenamebeforethe.jsandsaveitascmd:

constcmd=file.match(/(.*)\.js$/)[1];

Werequirethefile:

constmod=require('./'+file);

If a file exports an API command as theapi property, we expose it on

lounger.commands.AllCLIcommandsareavailableonlounger.cli:

if(mod.cli){

cli[cmd]=mod.cli;

}

if(mod.api){

api[cmd]=mod.api;

}

After theforEach loop is finished and all commands are loaded we can set

lounger.loaded totrue. This will prevent the checks we added previously from

throwing:

lounger.loaded=true;

AslaststepweresolvethePromise:

resolve(lounger);

Thewholefunctionlounger.load:

lounger.load=functionload(){

returnnewPromise((resolve,reject)=>{

fs.readdir(__dirname,(err,files)=>{

files.forEach((file)=>{

if(!/\.js$/.test(file)||file==='lounger.js'){

return;

}

constcmd=file.match(/(.*)\.js$/)[1];

constmod=require('./'+file);

if(mod.cli){

cli[cmd]=mod.cli;

}

if(mod.api){

api[cmd]=mod.api;

}

});

lounger.loaded=true;

resolve(lounger);

});

});

};

Wealmostforgottoexportlounger,soweaddamodule.exportsattheendofthe

file:

module.exports=lounger;

Wecanalreadyusethecode:

$node

>constlounger=require('./lib/lounger.js');

lounger.load().then(console.log);

Promise{<pending>}

>{loaded:true,version:'1.0.0',load:[Function:load]}

>lounger.commands

{isonline:[Function:isOnline]}

>lounger.cli

{isonline:[Function:cli]}

>lounger.commands.isonline('http://localhost:5984').then(console.log);

Promise{<pending>}

>{'http://localhost:5984':true}

Thelaststepinthissectionistomakeloungerusableontheterminalitself.Forthistaskwe’vecreated thebinfolderalready.Timetoputafilecalledlounger-cli.jsinto

bin!

npm has a nice feature: if we add a JavaScript file to a property calledbin in the

package.jsonofamodule,npmwilladd it toourPATH ifweare installing thepackage

globally(e.g.withnpminstall-glounger).

Ifwehavethebin-propertydefinedandtheloungerisgloballyinstalled,itgetsavailableaslounger(whichisourpackagename)ontheterminal,quitecomfortable!Toinform

the user that a package is intended to get installed globally, we can addpreferGlobal: true to thepackage.json (see also:

https://docs.npmjs.com/files/package.json#preferglobal).

Toenablethetwofeaturesweaddthesetwolinestoourpackage.json:

"bin":"./bin/lounger-cli",

"preferGlobal":true,

Additionally we have tomakebin/lounger-cli executable, ifwe are onLinuxor

OSX:

$chmod+xbin/lounger-cli

The first line of ourlounger-cli filewill be a “shebang line”, it tellsLinxux/Unix

shellusersthatofLinux/UnixusersthatitmustrunourfilewithNode.js:

#!/usr/bin/envnode

Afterwardsweloadlib/lounger.js,thecoreofourcommandlinetool:

constlounger=require('../lib/lounger.js');

The next step is to parse our command line arguments. In this case we are using themodulenopttoparsethecommandlinearguments(installitwith$npmi--save

nopt).Itwouldbepossibletotrytoparsetheargumentsonourown,butabattleproven

module likenopt offers a lot more features and is easier to use. We get the passed

commandbyaccessingparsed.argv.remain:

constnopt=require('nopt');

constparsed=nopt({},{},process.argv,2);

constcmd=parsed.argv.remain.shift();

Thenextstep is toboot theclientbycallinglounger.load,whichwillbootstrapthe

client and populatelounger.commands andlounger.cli. After the promise got

resolved,wecallthecommandthatwaspassedonthecommandline:

lounger.load().then(()=>{

lounger.cli[cmd]

.apply(null,parsed.argv.remain)

.catch((err)=>{

console.error(err);

});

}).catch((err)=>{

console.error(err);

});

AseverycommandreturnsaPromise,wecatcherrorswithcatchandprintthemtothe

console.

Wecannowtestourminimalisticcommandlineclientonthecommandline:

$npminstall-g.

$loungerisonlinehttp://couchdb.example.com

http://couchdb.example.comseemstobeofflineornodatabase

AndgivenourPouchDB/CouchDBserverisrunning:

$loungerisonlinehttp://localhost:5984

http://localhost:5984seemstobeonline

Insteadofrunningnpminstall-g.aftereachcodechange,youcanalsorunnpm

linkinyourmoduledirectory.Itwilllinktheglobalinstallationtothecurrentdirectory,

whichmeansthateverychangeisimmediatelyavailable,aslongasthedirectorydoesnotchange.

ChooseyourownflavoursTherearecountlessgoodargumentparsinglibrariesonnpm,somealternativestonoptare:commander,

optimistandyargs.

Thatwas the firstbasicbuildingblock forourcommand lineclient,butweare still faraway from a product that our users would really love and promote.Wewill fix thoseissues(errorhandling,help,documentation)inthenextsections.Thecodeforthecurrentsectionisavailableatsourcecode/client-bootstrap.

ErrorhandlingWealreadylearnedin[_you_never_get_stuck]thatnouserenjoyscrypticerrormessagesandstacktraces.Sadlythatisstillthecaseforourloungerapplicationandmakingthe

applicationmoreaccessibleshouldbeourfirstpriority.

Rightnowloungerdoesnotdoanythingaboutwronginputfortheisonlinecommand:

$loungerisonlineragrragr

$

Another caveat to consider is, are we handling errors correctly so people can use thecommandlineclientintheirbashscripts?

$loungerisonlineragrragr

$echo$?

0

Wouldn’titbeniceriftheuserswouldgetahintaboutthecorrectusageoftheprogramrightaway,withoutopeninganydocumentation?Nobodyenjoyssittinginfrontofablackterminalhavingno ideawhat todo(see[_you_never_get_stuck]).Whileweareatitwecan also fix the wrong exit code which is currently signalling a successful executedprogram(seealso[_it_supports_powerusers_exit_codes]).

Whyisn’tourconsole.errorcall inbin/lounger-cliprintinganything?Turns

outweintroducedasubtlebug:weforgotthe.catchforourPromisereturningcallin

lib/isonline.js.GiventheAPIfunctionisOnlinerejectsthePromise,wehave

nohandlerinthefunctionclitotakecareofit.Noproblem,we’lladdthe.catchright

now:

functioncli(url){

returnnewPromise((resolve,reject)=>{

isOnline(url).then((results)=>{

//printonstdoutforterminalusers

Object.keys(results).forEach((entry)=>{

letmsg='seemstobeofflineornodatabaseserver';

if(results[entry]){

msg='seemstobeonline';

}

console.log(entry,msg);

resolve(results);

});

}).catch(reject);//addthemissingcatch

});

}

exports.cli=cli;

Ournexttryisabitmoresuccessful:

$loungerisonlineragrragr

[Error:InvalidURI"ragrragr"]

Not sure ifyouarehappywith it… I’mnot! Just imagine someone thathasneverusedNode.js or the Terminal. Maybe even someone that is completely new to computers.InvalidURI won’t help themmuch to get their task done. Twenty years ago they

wouldhavehad togetabookfromthe library inorder to findoutwhatanURI is,andtodaytheywouldhavetogoogleforit.UsefultimetheycouldhavefunwithourCLIandgetthingsdone!

Gladlywecanfixitbyaddingvalidationsfortheargumentsinisonline.js.

IftheuserdoesnotprovideaurltotheCLIwearecreatinganewerrorwiththemessageUsage: lounger isonline <url> which describes how they have to use the

command.WearesettingthetypeoftheerrortoEUSAGE,whichwillbeimportantlater.

In lounger all errors that are thrownbecause theusermadewrong input aregetting thetypeEUSAGE.Allothercaseswhereweintroducedbugsdon’tgetthetypeEUSAGE:

functioncli(url){

returnnewPromise((resolve,reject)=>{

if(!url){

consterr=newError('Usage:loungerisonline<url>');

err.type='EUSAGE';

returnreject(err);

}

The less-than sign and greater-than sign around the url indicate thaturl is a required

argumentandnotoptional.The lastcommand in theblock rejects thePromisewithourerrorandreturns,inordertopreventtheexecutionoffollowingcode.

Earlyreturnsareusefultoreducecyclomaticcomplexity.Cyclomaticcomplexityappearswhereifandelseblocks are nested,whichmakes it harder for the human brain to reason about the flow of the programexecution.

Additionallywehavetocheckiftheurlisavalidurl:

if(!/^(http:|https:)/.test(url)){

consterr=newError([

'invalidprotocol,mustbehttpsorhttp',

'Usage:loungerisonline<url>'

].join('\n'));

err.type='EUSAGE';

returnreject(err);

}

In this case we are setting the error type toEUSAGE again and reject the Promise.

Additionallywearetellingtheuserthatweexpectavalidurlwithaprotocolthatisusableforus.

Onournexttrywewillgetaslightlybetterresult:

$./bin/lounger-cliisonlinedsf

{[Error:invalidprotocol,mustbehttpsorhttp

Usage:loungerisonline<url>]type:'EUSAGE'}

Aswe reject the promise, theconsole.error thatwe added in inbin/lounger-

cliprintstheerrorobject.Byaddingafewlinesofcodewecanformatit,sohumanscan

read it better.Wewill install thenpmlog logger for it (hint:npmi is short fornpm

install):

$npmi--savenpmlog

We require it at the top ofbin/lounger-cli, the filewherewe catch the rejected

Promise:

constlog=require('npmlog');

As a next step we add the functionerrorHandler tobin/lounger-cli. If the

errorisausageerror(oftypeEUSAGE),welogthemessageandexitwitherrorcode1.

Allothererrorsareloggedusinglog.error(err)fornow:

functionerrorHandler(err){

if(!err){

process.exit(1);

}

if(err.type==='EUSAGE'){

err.message&&log.error(err.message);

process.exit(1);

}

log.error(err);

process.exit(1);

}

Nowwehave to switch from theoldconsole.error call toournewerrorhandling

function:

lounger.load().then(()=>{

lounger.cli[cmd]

.apply(null,parsed.argv.remain)

.catch(errorHandler);

}).catch(errorHandler);

Cool,let’sseeifitworks:

$loungerisonline

ERR!Usage:loungerisonline<url>

$echo$?

1

Thatlooksalotbetter!

Figure2.AusageerrorresultingfromprovidingwronginputtotheCLI

We still did not touch other errors: errors from dependencieswe use, or evil bugs thatsneak in, like reference errors. To simulate such an error we can add a call to a non-existingfunctionintheclifunction:

functioncli(url){

returnnewPromise((resolve,reject)=>{

doesNotExist();

Ifwenowrunthecommandlineclientweget:

$loungerisonlinehttp://example.com

ERR!ReferenceError:doesNotExistisnotdefined

IfIjustdownloadedthecommandlineclientandtriedtouseit,Iwouldbequitepuzzled.MaybeIgotanewjobandtriedtousethesametoolmycoworkersuse,butdownloadedanewerreleasewithbugs.Iwouldbestuck,withnofurthernoticehowtocontinue.Tobehonest,ifIhadn’tprogrammedinJavaScriptforyearsthisstacktracewouldreallypuzzleme!Formostpeoplethesearerocksintheirwaywheretheyjuststopusingourprogramandswitchtoanalternative.VeryfewgoonthejourneytofindoutwheretheycansubmitanissueorevenwriteaPR.Usuallycomputersaretoofrustratingandpeopledon’treallywanttospendmultiplehourstryingtofindsomeonetohelpthemwithacrypticmessage.Sowhat aboutmaking this process as easy as possible, reducing the frictionwherewecan?

npmitselfsupportsabugspropertyinthepackage.json.Ifweadd:

"bugs":{

"url":"http://example.com/lounger/issues"

},

to thepackage.json of lounger, a call to$ npm bugs will open

http://example.com/lounger/issues in a browser for us. Cool, we got a

centralplacewherewearestoringtheurltoourissuetracker.Wecanalsoaddtheurltoourstacktraces,inordertomakesubmittingbugsforouruserseasier.Weneedtorequirethepackage.jsoninbin/lounger-cli,thefilewhereweprintourerrorsanyway:

constpkg=require('./package.json');

ByalteringourerrorHandlerwemakeitprintfullstacktraces.Additionallyweadd

asktheusertoopenanissueasitisprettyclearrightnowthattheerrorwasnotausageerrorthatwascaughtbyourvalidations:

functionerrorHandler(err){

if(!err){

process.exit(1);

}

if(err.type==='EUSAGE'){

err.message&&log.error(err.message);

process.exit(1);

}

err.message&&log.error(err.message);

if(err.stack){

log.error('',err.stack);

log.error('','');

log.error('','');

log.error('','lounger:',pkg.version,'node:',process.version);

log.error('','pleaseopenanissueincludingthislogon'+

pkg.bugs.url);

}

process.exit(1);

}

Ok,nexttry:

$loungerisonlinehttp://example.com

ERR!doesNotExistisnotdefined

ERR!ReferenceError:doesNotExistisnotdefined

ERR!at/home/rocko/clibook/sourcecode/error-

handling/lib/isonline.js:35:7

ERR!atcli(/home/rocko/clibook/sourcecode/error-

handling/lib/isonline.js:34:10)

ERR!at/home/rocko/clibook/sourcecode/error-handling/bin/lounger-

cli:15:6

ERR!

ERR!

ERR!lounger:1.0.0node:v4.2.3

ERR!pleaseopenanissueincludingthislogon

http://example.com/lounger/issues

Awesome!Thestacktracewith linenumbers isuseful forus.Thecurrentversionof theprogramandtheNode.jsenvironmenthelpus,too.Incasethecommandlineclientreallyhits awall,we receive a lot of informations in order to debug the process. Evenmoreimportant:theuserget’salltheinformationneededtocreateanissue.Weremovealotoffriction from the process by directly pointing to the issue tracker and providing allinformationthatisneededtodescribethebug-nolongbackandforthaboutthecurrentNodeversionorthemissinglogfile!

Thecodeforthissectioncanbefoundatsourcecode/error-handling.

JSONsupportandShorthandsJSONsupportisusefulforallusersthatwanttotaketheoutputfromtheCLIandprocessit programmatically with their own tools (we talked about that in[_it_supports_powerusers_json]inthefirstpartofthebook).Byaddinga--jsonflagto

our commandisonlinewe can add this useful featurewith a few lines of code.We

havetotellourargumentparseraboutit,inthiscase,wearetellingnoptthatwewantto

have--jsonhandledasabooleaninbin/lounger-cli:

constparsed=nopt({

'json':[Boolean]

},{'j':'--json'},process.argv,2);

Based on the typeBoolean nopt will automatically also add--no-json for us,

whichwillcomehandywhenweaddadditionalconfigurationbyfile later.Additionallyweregisterashorthandforourpowerusers,theycanalsouse-jinsteadof—json.

Wearethenpassingtheresultparsedintolounger.load:

constparsed=nopt({

'json':[Boolean]

},{'j':'--json'},process.argv,2);

constcmd=parsed.argv.remain.shift();

lounger.load(parsed).then(()=>{

lounger.cli[cmd]

.apply(null,parsed.argv.remain)

.catch(errorHandler);

}).catch(errorHandler);

longer.loadaddsalounger.config.getcommandandmakesitavailableforus

aspartofthebootstrap:

lounger.load=functionload(opts){

returnnewPromise((resolve,reject)=>{

lounger.config={

get:(key)=>{

returnopts[key];

}

};

fs.readdir(__dirname,(err,files)=>{

Werequirelounger.jsinourfileisonline.js:

constlounger=require('./lounger.js');

As last stepwe the check for the json-flag in our functioncli afterwegot the results

back:

isOnline(url).then((results)=>{

if(lounger.config.get('json')){

console.log(results);

resolve(results);

return;

}

That’sit!Wecantestthecommand:

$loungerisonlinehttp://example.com

http://example.comseemstobeofflineornodatabaseserver

$loungerisonlinehttp://example.com--json

{'http://example.com':false}

$loungerisonlinehttp://example.com-j

{'http://example.com':false}

Ouruserscannowpipetheoutputontheirterminalsintootherconsumersandprocesstheresults.Wealsoaddedourfirstcommandlineflagtoloungertomodifytheexecutionofacommand - great! The code and tests for this section are insourcecode/json-

flags.

DocumentationThelaststeptofinishourcommandisonlineistoaddproperdocumentation.Wewill

write the documentation in Markdown and need documentation for the API and CLIcommands. The API docs will live indoc/api and the CLI commands will live in

doc/cli.Iknowthatmostprogrammershatewritingdocumentation,butitwillhelpus

alot:newuserswillbeabletogetupandrunningeasierandwewon’tlosethembeforethey had the chance to enjoy our product. Additionally, we make our lives easier bydocumenting the functionality once, so people don’t have to open issues or ask in chathow they can use a command. Basically a win-win situation.We start withdoc/api

/lounger-isonline.md, which describes the API that is available at

lounger.commands:

lounger-isonline(3)—checkifadatabaseisonline

====================================================

Theheadingdescribesourcommandaslounger-isonline(3)andthenaddsashort

explanationwhatthecommandisabout.Thenumberinbracketsdescribesthetypeofthesection.Foraman-pageaLibraryFunctionisnotedbya3andaUserCommandwould

bea1(spoiler:ourCLIcommandisaUserCommand).

Thenextsectiondescribeshowouruserscanusethecommand:

##SYNOPSIS

lounger.commands.isonline(url)

Thelastpartisadetaileddescriptionofhowthecommandworks:

##DESCRIPTION

CheckifaCouchDB/PouchDBdatabaseisavailableonthecurrent

network.

url:

Theurlmustbea`String`andmustbeaurlusingthehttporhttps

protocol.

Thecommandreturnsapromise.ThepromisereturnsanObject.Thekey

oftheObjectistheprovidedurlandthevaluesareoftype`Boolean`.

`true`indicatesanonlineCouchDB/PouchDBnode.

That’s it for the API part, we can now add the text fordoc/cli/lounger-

isonline.md:

lounger-isonline(1)—checkifadatabaseisonline

====================================================

loungerisonline<url>[--json]

##DESCRIPTION

<url>:

Checkifadatabasenodeiscurrentlyonlineoravailable.

`isonline`printstheresultashumanreadableoutput.JSONoutputis

alsosupportedbypassingthe`--json`flag.

With the small1 inlounger-isonline(1)we are signalling that this help section

explainsaUserCommand.Theless-thanandgreater-thansymbolsfor<url> showthe

userthaturlisamandatoryargument-withoutitthecommandwon’twork.Thesquare

bracketsof[--json]meanthatthejson-flagisanoptionalcommand.

Aswe got the sources for our documentation,we can start to build our documentationfromoursourceswithmarkedandmarked-man:

$npmi--save-devmarkedmarked-man

AMakefilewould be a great fit for generating the documentation from the source, butsadlyitishardtogetMakefilestoworkonWindows,sowewillwriteourbuildstepsinJavaScript. In therootdirectoryof lounger,wecreate thefilebuild.js.Additionally,

we have to installmkdirp andrimraf,mkdirp provides the functionalitywe know

from the Linux commandmkdir-p in a cross-platform compatibleway: it creates a

directories and subdirectories in a recursive way. The modulerimraf brings us the

equivalent ofrm-rf to theNode.js platform: deleting directories in a recursiveway.

Additionally we will use the moduleglob to match all needed files for our

documentationbuild.

$npmi--save-devmkdirprimrafglob

Ourfirstfunctionwillbeafunctiontocleanafreshfolderstructurewherewecansaveourman-pages:

'usestrict';

constmkdirp=require('mkdirp');

constrimraf=require('rimraf');

constglob=require('glob');

constpath=require('path');

functioncleanUpMan(){

rimraf.sync(__dirname+'/man/');

//recreatethetargetdirectory

mkdirp.sync(__dirname+'/man/');

}

Wehavetofindoutwhichmarkdownfilesareavailableforthecompile.Oursourcesfordocumentationareatdoc/apioratdoc/cli.Additionallywewillhavesomecontent

indoc/websitewhichisspecifictothewebsite,i.e.thecontentfortheindex.html.

The functiongetSourceshelpsus toget thefullpathto themarkdownfilesforeach

typeofour sources (api,doc,website). It returns the relativepathof thematching

globandthenusesthefunctionpath.resolvetogetthefullpathinacross-platform

compatibleway.

functiongetSources(type){

constfiles=glob.sync('doc/'+type+'/*.md');

returnfiles.map(file=>path.resolve(file));

}

Theobjectsourcesstoresanarrayofthefoundfilesforeachtype:

constsources={

api:getSources('api'),

cli:getSources('cli'),

websiteIndex:getSources('website'),

};

Weareabletocleanupourtargetdirectorynowandtogetalistoffilenamesthatwewanttoconvert.Westillneedtofindoutthetargetpathandfilenamefortheconvertedfiles.Man-pages have different file endings depending of the kind of functionality theydescribe.Ourman-pagesfortheCLIwouldgettheending.1(UserCommands)andour

API function would get the ending.3 (Library Functions). Additionally we have to

change the/doc/cli/ and/doc/api/ in thepathof the file toour targetdirectory

/man/.Wehavetotakespecialcareofthepath-separators.OnWindowstheseparators

area\\,insteadof/.Thatmeansthepathdoc/apigetsdoc\\apionWindows.The

goodnewsisthatwecanaccessthecurrentpath-separatorusingpath.sepinNode.js

(theseparatorisprovidedbythecoremodulepath):

functiongetTargetForManpages(currentFile,type){

lettarget;

//settherightsectionforthemanpageonunixsystems

if(type==='cli'){

target=currentFile.replace(/\.md$/,'.1');

}

if(type==='api'){

target=currentFile.replace(/\.md$/,'.3');

}

//replacethesourcedirwiththetargetdir

//doitforthewindowspath(doc\\api)andtheunixpath(doc/api)

target=target

.replace(['doc','cli'].join(path.sep),'man')

.replace(['doc','api'].join(path.sep),'man');

returntarget;

}

Right nowwe justwant to createman-pages from our documents in theapi andcli

folder.

Based on these building blocks we can create the final functionbuildMan that will

finally build our man-pages. It will make use of the functions we just created and

additionally spawnachildprocesswhichcompiles themarkdown filesusingmarked-

man.WewillusethefunctionspawnSyncfromNodecoretospawntheprocesses.As

wewritetheresulttothefilesystem,wehavetorequirethefsmodule,too:

constfs=require('fs');

constspawnSync=require('child_process').spawnSync;

Thefirstjobiscleaninguptogetanewtargetdirectorywithoutanyfilesfrompreviousbuilds.Wetheniterateoveroursourcesandgetthetargetforournewgeneratedfile.Thefile is thenwritten to theharddiskusingfs.writeFileSync.Incaseofthewebsite

indexwestoptheexecutionaswedon’twanttodousethewebsiteindexforaman-pagerightnow.Inthenextiterationwecoulddefinitelyaddamainpagefortheloungerman-pages:

functionbuildMan(){

cleanUpMan();

Object.keys(sources).forEach(type=>{

sources[type].forEach(currentFile=>{

if(type==='websiteIndex'){

return;

}

//convertmarkdowntoman-pages

constout=spawnSync('node',[

'./node_modules/marked-man/bin/marked-man',

currentFile

]);

consttarget=getTargetForManpages(currentFile,type);

//writeoutputtotargetfile

fs.writeFileSync(target,out.stdout,'utf8');

})

});

}

buildMan();

WithbuildMan();inthelastline,wekickoffthebuildprocesseverytimewerunthe

scriptwithNode.Afewmodificationstoourpackage.jsoncouldmakeitanpmscript

andrunthebuildbeforeapublish,soourusersdon’thavetocompileanythingontheir

ownaspartoftheinstallation.Thisensuresthateveryuserreallygetsthesamecontentofthepackageandmakesinstallationsfaster.

I npackage.json we modify thescripts section and add entries forbuild and

prepublish. Theprepublish entry is a special hook for npm – itwill run every

timebeforewepublishthepackagetotheregistry:

"scripts":{

"test":"mocha-Rspec",

"docs":"node./build",

"prepublish":"npmrundocs"

},

npmalsooffersanicefeatureforman-pages:npmcaninstallthemfortheusersotheyareavailable on the terminal withman<command> for Linux/Unix users. In order to do

that, we have to add another entry to ourpackage.json, theman entry in the

directorysection:

"directories":{

"man":"./man"

},

Theentrypointstoourlocalman-pagesdirectory.IfyouareonUnix/Linuxyoucantryiftheman-pagesworknow:

$npmrundocs

>lounger@1.0.0docs/home/rocko/clibook/sourcecode/documentation

>node./build

$npminstall-g.

>lounger@1.0.0prepublish/home/rocko/clibook/sourcecode/documentation

>npmrundocs

>lounger@1.0.0docs/home/rocko/clibook/sourcecode/documentation

>node./build

/home/rocko/.nvm/versions/node/v4.2.3/bin/lounger->

/home/rocko/.nvm/versions/node/v4.2.3/lib/node_modules/lounger/bin/lounger-

cli

/home/rocko/.nvm/versions/node/v4.2.3/lib

└──lounger@1.0.0

$manlounger-isonline

Figure3.Theman-pageforlounger,ourcommandlineclient

Itisalsopossibletoselectaspecificsection:

$man3lounger-isonline

$man1lounger-isonline

Forourhtml-baseddocumentationwehave toadd thehtmlspecificpartnow.Wehavecreate a folder calledwebsite in the doc folder of our module. It will contain the

templatesforthewebsitethatareusedto“frame“thedocumentoutput.

Intothefoldercalledwebsiteweputafilecalledtemplate.htmlwithsomebasic

markup,andmostimportantly,placeholders!

<!doctypehtml>

<htmllang="en">

<head>

<metacharset="utf-8">

<title>Theloungermanual&amp;documentation</title>

</head>

<body>

<divclass="wrapper">

<ahref="./">lounger</a>

<divclass="content">

__CONTENT__

</div>

</div>

<navclass="toc-container">

<divclass="tocmain-toc">

__TOC__

</div>

</nav>

</body>

</html>

Weneedtocreatesomecontentfortherootofourwebsitetowelcometheuser.Iwilljustprovide a very short example. In general the landingpage shouldgive the user an ideawhatthecommandlineclientisaboutandmaybealsodemoit.Ascreencastisagreatwaytodemosomeofthecorefeatures.Forsomeinspirationwhattoputonthesite,feelfreetovisithttp://apache.github.io/couchdb-nmo,whichisthepageforthecommandlineclientIwrotelastyear.Nexttoourtemplateindoc/websiteweaddthebespokeindexpageas

index.md:

#WelcometoLounger

LoungerisafriendlyadministrationtoolforCouchDBandPouchDB.

```

#youwillneedNode.js>4forlounger

npminstall-glounger

```

Thingsyoucandowithlounger:

```

#checkifaCouchDB/PouchDBinstanceisonline

loungerisonlinehttp://example.com

```

Thepageisminimalisticbutgivesabriefoverviewwhatloungerisabout.ItalsoshowshowourvisitorscaninstallitandgivesahintthattheyneedNode.js–noteveryonehasNode.jsinstalledandsomepeoplemayhaveneverheardofnpm.Remember:ourgoalistomakeeverythingaseasyaspossiblefornewusers–theynevergetstuck.

Inorder tobuild thewebsitewehave toaddsomecode to thebuild.js.Westartby

addinganothercleanup-functionforourwebsite files,sowecanstartwithablankslateeverytime:

functioncleanUpWebsite(){

rimraf.sync(__dirname+'/website/');

mkdirp.sync(__dirname+'/website/');

}

Our website has different targets than the man-pages. We add a new function,getTargetForWebsite.Currentlyourmarkdownfilesareprefixedwithlounger-,

whichcomeshandyfortheman-pages,butisnotveryusefulforourwebsite.Insteadwewanttoprefixthemwiththeirtype,anAPIdocumentwouldgetprefixedbyapi-.This

waywe can put pages for the API next to the ones for the CLI, whichmakes linkingeasier. The first lines of the functiongetTargetForWebsitewill take care of that

taskandreplacethelounger-prefixwithaprefixspecifictothetypeofthedocument.

Afterwereplacedtheprefix,wesetthefileendingfrom.md to.html.Thefinaltarget

folder for our compiled results will be./website, so we have to take care that the

directoriesaresetupright.ThedifferencetogetTargetForManpagesisthatweuse

thewebsite folder instead ofman and that we also take care of the path for the

doc/website/index.mdfile:

functiongetTargetForWebsite(currentFile,type){

lettarget=currentFile;

//modifiythefilenameabitforourhtmlfile:

//prefixallclifunctionswithcli-insteadoflounger-

//prefixallapifunctionswithapi-insteadoflounger-

if(type==='cli'){

target=currentFile.replace(/lounger-/,'cli-');

}

if(type==='api'){

target=currentFile.replace(/lounger-/,'api-');

}

//setthefileendingtohtml

target=target.replace(/\.md$/,'.html');

//replacethesourcedirwiththetargetdir

target=target

.replace(['doc','cli'].join(path.sep),'website')

.replace(['doc','api'].join(path.sep),'website')

.replace(['doc','website'].join(path.sep),'website');

returntarget;

}

The last item which is missing for our website is something like a table of contents.getTocForWebsitewillcreatealistingofourAPIandCLIfunctions.Laterwewill

insertthetableofcontentsintotheTOCplaceholderofthetemplate.

TheTOCitselfgetsanestedlistofthefileswecompile.Inthefutureitmightmakesenseto replace the whole system with a template engine likehandlebars, jade or

Nunjucks. I coulddefinitelywritea secondbookabout staticwebsitegenerationplus

thedifferentpossibletoolchains,butItrytokeepitsimpleandminimalisticfornowandjustuseplainES6templates.

Afterstartinganunorderedlistwith<ul>weiterateoverthetypesofoursourcesagain.

ThefilesofourwebsiteIndextypeareunwantedandwereturnearlyintheircase.For

allotherswetakethetypeofeachsectionanduseitasthefirstlistelement.Itshowsthetypeofeachsection(APIorCLI)andactsasaheading.

Afterwegotaheading,weiterateovereachfilefromthecurrentsection.Asthelinktextdiffersabitfromtheactualhyperlink,wecreateaconstantcalledfileandanadditional

linktext constant. For bothwemust get rid of thelounger- prefix.To create the

referencethatisusedashrefwemustchangethe.mdtoa.htmlending.Thelinktext

shouldnothaveafileendingatall,soweremovethe.mdendingforit:

functiongetTocForWebsite(){

lettoc='<ul>';

Object.keys(sources).forEach(type=>{

//wedon'twanttheindexinourtocfornow

if(type==='websiteIndex'){

return;

}

toc+=`<li><span>${type}</span><ul>`;

sources[type].forEach(currentFile=>{

constprefix=type==='cli'?'cli-':'api-';

constfile=path.basename(currentFile)

.replace('lounger-',prefix)

.replace(/\.md/,'.html');

constlinktext=path.basename(currentFile)

.replace('lounger-','')

.replace(/\.md/,'');

toc+=`<li><ahref="${file}">${linktext}</a></li>`;

});

toc+='</ul></li>';

});

toc+='</ul>';

returntoc;

}

We can now take the small functions we created and create the main build functionbuildWebsitefromthem.TheconstanttemplateFiledescribeswherewecanfind

ourtemplatefiletemplate.htmlwhichwecreatedinthebeginning:

consttemplateFile=__dirname+'/doc/website/template.html';

Thenext step is to replace theplaceholders in the templateswith thegenerated tableofcontentsandthedifferentcontentforeachAPIorCLImethod.

InbuildWebsitewe callcleanUpWebsite toremoveanyoutdatedfilesasafirst

step.Wereadthetemplatewithfs.readFileSyncandgetthetableofcontents.We

then iterateoverour sources.This timewedon’t spawnachildprocesswithmarked-

man,wejustusemarked,whichoutputshtml.Afterwegotthetargetforthecurrentfile

ofthewebsitewereplacetheCONTENTplaceholderwithit.TheTOCgetsreplacedwith

the tableofcontentswhich issaved intocat the topof themainfunction.Movingthe

readoperationofthetemplateandthecreationoftheTOCoutoftheloopshaspositiveeffectsontheperformanceofourbuild,aswedon’thavetocallthemforeverynewfile.Therenderedcontentfinallygetswrittentothediskusingfs.writeFileSyncagain.

The last line in the code snippet finally callsbuildWebsite in order to build our

websitewhenwerunbuild.js.

functionbuildWebsite(){

cleanUpWebsite();

consttemplate=fs.readFileSync(templateFile,'utf8');

consttoc=getTocForWebsite();

Object.keys(sources).forEach(type=>{

sources[type].forEach(currentFile=>{

//convertmarkdowntowebsitecontent

constout=spawnSync('node',[

'./node_modules/marked/bin/marked',

currentFile

]);

consttarget=getTargetForWebsite(currentFile,type);

constrendered=template

.replace('__CONTENT__',out.stdout)

.replace('__TOC__',toc);

//writeoutputtotargetfile

fs.writeFileSync(target,rendered,'utf8');

});

});

}

buildWebsite();

We can now run our build again and take a look at ourwebsite,which appears in thewebsitefolder:

$npmrundocs

Figure4.Theminimalwebsiteforloungergeneratedfromourmarkdown

The site still misses some content and also a nice stylesheet. As I already mentioned,buildingwebsitesisawholetopicforanewbookandIwanttoleaveitlikethisfornow.Feel free to add more content and styles to the website. Nevertheless we just createdsomethingthatwecandeploytoGitHubpagesoranyotherhoster.Thebestthingaboutitis that we can ship it with our command line client as additional documentation – foreveryone who can’t use or doesn’t like man-pages. We will use both types ofdocumentationinournextsectionwhenwecreateahelpsystem.

Thecodeforthissectioncanbefoundatsourcecode/documentation.

MoreHelpWegotsomedocumentation,buttherearestillsomeroughedgesinlounger.Ifwetrytoaccessacommandwhichdoesnotexist,theuserdoesn’tgetanyadvicehowtocontinue:

$loungerfoobar

Ifausercallsloungerwithnoargumentsatalltheyalsodon’tgetsupport:

$lounger

What’s missing is a help page which explains how our users can get their task done.Additionally it would be really awesome if they could open our documentation (web-pagesorman-pages)rightfromtheterminal.Ourusersshouldn’thavetocareaboutman-pagesorhave to findoutwherewehostourwebsite.Thedesiredbehaviourof loungerwouldbe:

1. TheCLI prints a general helpmessage if itwas calledwithout a command or if apassedcommanddoesn’texist.Itgivestheuserahinthowtoproceedfurthertogettheirtaskdone.

2. Itiseasytogetadditionalhelpforacommand

Westartbycreatinganewlibraryfunction,lib/help.js.Thefirstfunctionwillprint

afriendlyhelptexttotheuser.Thehelptextgetsconstructedinahelperfunction(nopunintended).

WecangetallofouravailablecommandsbycallingObject.keys(lounger.cli).

Wechaina.join(',')calltoseparateeachcommandwithacommaandaspace:

functiongetGeneralHelpMessage(){

constcommands=Object.keys(lounger.cli).join(',');

Thenextpartisatemplatestringwhichexplainshowtouselounger.Weaddallavailablecommandswhichareexposedonthecommandlineinterface.Wealsoexplainthatuserswithoutacluecanrunloungerhelptogetherwiththecommandtheyareinterestedin

togetdetailedhelpforacommand.Additionallyweprovideanexamplehowtheywouldcallthehelp.

The final line tells them which version of lounger they run, which comes handy indifferentsituations.Usuallypeopleforgetwhichversionofapackagetheyhaveinstalled.Thereason issimple:nobodycanremember theexactversionofall software theyhaveinstalledrightnow.Imagineauserwhojustreadablogarticleaboutloungerversion2.3andagreatnewfeaturetheywanttotry.Theyinstalledloungersometimeago,butsadlythecommandisjustavailablesinceversion2.3whichgotreleasedthreedaysago.Inthiscasetheygetimmediatefeedbackthattheyrunonanolderversion.

HereisthewholefunctiongetGeneralHelpMessage:

functiongetGeneralHelpMessage(){

constcommands=Object.keys(lounger.cli).join(',');

constmessage=`Usage:lounger<command>

Theavailablecommandsforloungerare:

${commands}

Youcangetmorehelponeachcommandwith:loungerhelp<command>

Example:

loungerhelpisonline

loungerv${lounger.version}onNode.js${process.version}`;

returnmessage;

}

Thenextfunctionwe’llbuildwilltrytoopentheman-pageifpossible.ItwillfallbacktothewebsiteversionforWindowsusers.

Themoduleopenerdoesagreat jobtoopenfilesondifferentoperatingsystems.This

alsoappliestohtmlfiles,aswewanttoopentheminthedefaultbrowserofthecurrentoperatingsystem.Soweinstallopenerasournextdependency:

$npminstall--saveopener

Wethenrequireopeneratthetopofhelp.js:

constopener=require('opener');

We also have to spawn theman command later and find out the absolute path for our

websitefiles:

constspawnSync=require('child_process').spawnSync;

constpath=require('path');

ThecoremoduleoscanhelpustofindoutifwearerunningonWindows:

constisWindows=require('os').platform()==='win32';

We then have to spawn theman command or open the default browser for the desired

functionality.Withstdio:inheritforthespawncommandwecanseeandinteract

withtheoutputofthespawnedprocess:

functionopenDocumentation(command){

if(isWindows){

consthtmlFile=path.resolve(__dirname+'/../website/cli-'+command+

'.html');

returnopener('file:///'+htmlFile);

}

spawnSync('man',['lounger-'+command],{stdio:'inherit'});

}

Thelasttaskisputtingallourhelperfunctionstogether,ifacommandisnotavailable,weprint the general help. If the command exists, we are opening the man-page or thebrowser,dependingontheOperatingSystem:

exports.cli=help;

functionhelp(command){

returnnewPromise((resolve,reject)=>{

if(!lounger.cli[command]){

console.log(getGeneralHelpMessage());

}else{

openDocumentation(command);

}

resolve();

});

}

Ifyouwantyoucanalsoaddsomecodetomakeitconfigurablefortheuserwhichtypeofdocumentationisopenedforthem.MaybeLinuxusersprefertheWebsiteversionofthedocs. We can already try the new help command, as it gets picked up by ourlounger.jsfile:

Figure5.Ourhelpcommandinaction

Ourfinaltaskinthissectionisthatwehavetomakesuretoprintthegeneralhelpincaseswheretheuserdoesnotenteracommandatall.Inbin/lounger-cliwerequirethe

help.jsfileandmodifythelounger.loadcall.Incasewedon’tfindthecommand

inlounger.cli,weprintthegeneralhelp:

consthelp=require('../lib/help.js');

lounger.load(parsed).then(()=>{

if(!lounger.cli[cmd]){

returnhelp.cli();

}

lounger.cli[cmd]

.apply(null,parsed.argv.remain)

.catch(errorHandler);

}).catch(errorHandler);

Congrats!Wearefinishedwiththehelpsystemnowandmadesignificantprogress!Feelfreetoplayaroundwithournewhelpsystembyenteringthesecommands:

$loungerblerg

$lounger

$loungerhelp

$loungerhelpisonline

Thecodeforthissectionislocatedat:sourcecode/help-system.

ConfigurationRequiringourusers toadd their favourite settingsas flagsbyhandevery time theyuselounger can be cumbersome.A configuration file enables our users to save the settingstheyneedeveryday.Itwouldbeniceifwecouldsupportthisfeaturethatwecoveredin[_it_supports_powerusers_configuration]tomakeourpowerusershappier.

Itiscommonpracticethatcommandlinetoolsputtheirconfigurationfilesintothehomedirectoryoftheuser.Thehome-directoryisdifferentforeveryOperatingSystem,butthemoduleosenvcanhelpustofindthecurrenthomedirectory:

$npminstall--saveosenv

Webasicallyhavetoseeifaloungerrcconfigurationexistsinourhomedirectory,and

ifnot,wehavetocreateanemptyone.Wedothatinlounger-cli,inordertokeepthe

APIfunctionsfreeof thesideeffect.This is thechangedlounger-clifilewherewe

added arequire-call forosenv and create a config file right after the argument

parsing.Wealsoaddthepathtotheconfigfiletoourparsedarguments:

#!/usr/bin/envnode

constlounger=require('../lib/lounger.js');

constpkg=require('../package.json');

constlog=require('npmlog');

constnopt=require('nopt');

consthelp=require('../lib/help.js');

constosenv=require('osenv');

constfs=require('fs');

constparsed=nopt({

'json':[Boolean]

},{'j':'--json'},process.argv,2);

consthome=osenv.home();

parsed.loungerconf=home+'/'+'.loungerrc';

if(!fs.existsSync(parsed.loungerconf)){

fs.writeFileSync(parsed.loungerconf,'');

}

The config itself will use ini-formatted config files in order to store and read settings.config-chainisamoduletoloadconfigurationswithdifferentprioritiesbasedonthe

orderweloadthem.Italsosupportsini-formattedfiles.

Let’screateafilelib/config.jsandrequireconfig-chain:

$npmi--saveconfig-chain

'usestrict';

constcc=require('config-chain');

config-chainisabletomanagemultipleconfigurations.Itwilloverrideconfiguration

settingsaccordingtotheorderweloadthem.

Forourusecasewewanttheoptionsprovidedasargumentsonthecommandlinetohavethehighestpriority.Meaning,theyoverridetheonesfromtheconfigfile.Astheloadingofthefileisdoneasync,wehavetolistentotheloadeventemittedbyconfig-chain

onceitisfinished.Fortheusecaseswherenoconfigfilewasset,wedon’ttrytoloadadditasfile.Onallerrorswerejectourpromiseandpasstheerrorobject:

exports.loadConfig=loadConfig;

functionloadConfig(nopts){

returnnewPromise((resolve,reject)=>{

letcfg;

if(!nopts.loungerconf){

cfg=cc(nopts)

.on('load',()=>{

resolve(cfg);

}).on('error',reject);

}else{

cfg=cc(nopts)

.addFile(nopts.loungerconf,'ini','config')

.on('load',()=>{

resolve(cfg);

}).on('error',reject);

}

});

};

Theconfigobjectthatisreturnedfromconfigchainhasnicegetandsetmethods.We

canevensavetheconfigbacktotheconfigurationfileafterchangingtheconfigusingthe

savemethod.FornowwehavetointegrateloadConfigintothebootstrapoflounger.

Do you remember theconfig.get method inlounger.js from the previous

chapter?Wewillreplaceitwiththeconfigobjectthatisreturnedbyourloadfunction.

Asafirststepwehavetoloadourconfiginlounger.js:

constconfig=require('./config.js');

Inlounger.loadwearegoingtoloadtheconfig:

lounger.load=functionload(opts){

returnnewPromise((resolve,reject)=>{

config

.loadConfig(opts)

.then((cfg)=>{

});

Theothercontentoflounger.loadincludingthefs.readdircallismovedintothe

callbackofthechainedthenfunction:

lounger.load=functionload(opts){

returnnewPromise((resolve,reject)=>{

config.loadConfig(opts)

.then((cfg)=>{

lounger.config=cfg;

fs.readdir(__dirname,(err,files)=>{

files.forEach((file)=>{

if(!/\.js$/.test(file)||file==='lounger.js'){

return;

}

constcmd=file.match(/(.*)\.js$/)[1];

constmod=require('./'+file);

if(mod.cli){

cli[cmd]=mod.cli;

}

if(mod.api){

api[cmd]=mod.api;

}

});

lounger.loaded=true;

resolve(lounger);

});

}).catch(reject);

});

};

Ifwenowrunloungeritwillcreateaconfigfileforusinourhomedirectory.OnOSXmyconfigisat~/.loungerrc.

Afterthefilegotcreated(don’tforgettorunloungeratleastonetime)wecansetJSONoutputtotruein~/.loungerrc:

json=true

Just try out$ lounger isonlinehttp://example.com, lounger will print

JSONnow.Wecanstilloverridetheconfigonthecommandline:

$loungerisonline--no-json

As manually editing the~/.loungerrc is not very user friendly, we will build a

loungerconfig command.The config command shouldhave the abilities to show

theconfiganditsvaluesandtosetconfigvalues.

HereistheproposedCLI:

$loungerconfigsetjsontrue

$loungerconfiggetjson

TheAPIcouldlooklikethis:

lounger.commands.set('json',true)

lounger.commands.get('json')

config-chain offers the data provided in the loaded config file as

.sources.config.data.ForJSONformattedoutputwereturn thewholeconfig if

nokeywasprovided:

constdata=lounger.config.sources.config.data;

if(lounger.config.get('json')&&!key){

resolve(data);

return;

}

IfakeywasprovidedwebuildaJSONobjectthatjustcontainsthevalueforourkey:

if(lounger.config.get('json')&&key){

resolve({[key]:data[key]});

return;

}

Givenwedon’twantJSONformattedoutputandprovidedakeywereturnthevalue:

if(key){

resolve(lounger.config.sources.config.data[key]);

return;

}

Inthelastcasewherethejsonsettingissettofalse,andnokeywasprovidedwesimplyreadtheunparsedini-file:

resolve(fs.readFileSync(lounger.config.sources));

Hereisthewholegetfunction:

functionget(key){

returnnewPromise((resolve,reject)=>{

constdata=lounger.config.sources.config.data;

if(lounger.config.get('json')&&!key){

resolve(data);

return;

}

if(lounger.config.get('json')&&key){

resolve({[key]:data[key]});

return;

}

if(key){

resolve(lounger.config.sources.config.data[key]);

return;

}

resolve(fs.readFileSync(lounger.config.sources));

});

}

Modifyingtheconfigisdonebythesetfunction.Ittakesakeyandavalue,callsseton

theconfig-chainobjectandresolvesthepromiseafterthevaluesarewrittentothedisk:

functionset(key,value){

returnnewPromise((resolve,reject)=>{

if(!key&&!value){

reject(newError('keyandvaluerequired'));

return;

}

lounger.config.set(key,value,'config');

lounger.config.on('save',()=>{

resolve();

});

lounger.config.save('config');

});

}

Asalaststepwehavetoexposebothcommands:

exports.api={

get:get,

set:set

};

WebuildtheCLIfunctionalityontopofourAPIfunctions.ThemaindifferencebetweentheAPIandCLIfunctionisthattheCLIfunctionhassideeffectsforthegetcommand:

itprintstheresulttotheconsole.Wealsoaddsomeniceerrormessagestomakeiteasiertouse forourusers.Theyget instructionshow to run thecommand if theydon’tuse itproperly:

exports.cli=cli;

functioncli(cmd,key,value){

returnnewPromise((resolve,reject)=>{

functiongetUsageError(){

consterr=newError([

'Usage:',

'',

'loungerconfigget[<key>]',

'loungerconfigset<key><value>',

].join('\n'));

err.type='EUSAGE';

returnerr;

}

if(!cmd||(cmd!=='get'&&cmd!=='set')){

consterr=getUsageError();

returnreject(err);

}

if(cmd==='get'){

returnget(key).then((result)=>{

console.log(result);

}).catch(reject);

}

if(cmd==='set'){

if(!key&&!value){

consterr=getUsageError();

returnreject(err);

}

returnset(key,value).catch(reject);

}

});

}

Great! We have a config command now! The code for this section is atsourcecode/config. We fulfilled all points from[_it_supports_powerusers] and

[_you_never_get_stuck]andarereadyforaninitialrelease!

Ourfirstrelease&releasetipsYoucanpublishOpenSourcemodulesafter registeringanaccount at thenpm registry.After registering the accountyoubasically just have to typenpmpublish topublish

yourmodule to theregistry.Beforewepublish lounger Iwant tomention that therearesomenicewaystooptimisethepublishedpackages.InthissectionIwillexplainhowtooptimiseyourpackageregardingsizeandinstallationtime.

One way to keep the installation size small is to add a.npmignore file to the root

directory.Itworkssimilartoa.gitignorefileandnpmwon’tincludethelistedfiles

anddirectoriesinthepublishedpackage.

Dependingonthetypeofthefileswhicharen’tneededwhenusingthemoduleweareabletosavealotofspaceontheharddisksofourusers.Wewillsavethemalotofbandwidth,too.

Wecansavesomespaceifweexcludeourunitandintegrationtestsandalsothesourceforthecompileddocumentation:

/test/

/docs/

build.js

.DS_Store

npm-debug.log

Anothergreatwaytospeedupinstallationtimeistoincludeallproductiondependenciesinthepublishedmodule.WecantellnpmtobundlethemwithourmodulebyaddingthemasbundleDependenciestoourpackage.json:

"bundleDependencies":[

"config-chain",

"nopt",

"npmlog",

"opener",

"osenv",

"request"

],

Bundling the dependencies reduces the installation time a lot, aswe omit all the smallHTTPrequestsforeachdependencyandtheirdependenciesduringinstallation.Withthecurrent npm 3 it reduces the installation time from 20 seconds to 5 seconds for abroadbandconnection.Bundlingthedependenciesalsomakessurethatourpackageisstillinstallableevenifamodulewasunpublished.

When we now runnpmpublish wewill publish a highly optimised version of our

package.Thecodeforthissectionisatsourcecode/first-release.

MigrationoflargeamountsofdatausingStreamsSometimeswewant toprocess largeamountsofdata.With traditionalbufferingwerunintomemoryproblemsverysoon.Thewholedatajustdoesn’tfitintothememoryofthecomputer. Streams enable us to process data in small slices.Node.js streamswork likeUnixstreamsontheterminal,whereyoupipedatafromaproducerintoaconsumerusingthepipesymbol|.

Thecatprogramprintsthecontentoffilestostdout.InthisexampleIampipingthe

output of thecat program, intotr to change all letters from ourpackage.json to

uppercaseletters:

$catsourcecode/first-release/package.json|tr'a-z''A-Z'

Itworkswithalloutputonstdout:

$echo"ishout?"|tr'a-z''A-Z'

StreamsinUnixandinNode.jsenableustocomposesmallprogramsormodulesthatdoone thingwell togetour taskdone.Theyhandlebackpressure,whichmeans thata fastproducerwillautomaticallyslowdownifitispipedintoaslowconsumer.

The most used streams in Node are theReadable, Writeable andTransform

streams.Theyarebaseclassesthatcanbeusedtobuildyourowncustomstreams.Thereare also other stream types, like theDuplex stream and thePassthrough stream

whicharen’tcoveredin thebook.TheReadablestreamisusedtoreadinputdata,the

Transform stream is usually used to modify chunks of data and theWriteable

streamsacceptsdatatowriteitsomewhere(e.g.intoafile).

TodaywewillwriteacommandthatwillusestreamsforpipingdatafromCSVfilesintoCouchDB/PouchDB.Wecouldalsowriteanimportertomigratedatafromadatabase,e.g.aPostgresorMongoDB,butwithplainCSVfileswedon’thavetoinstallanewdatabase.Theprincipleappliestobothsourcetypes,filesanddatabases.AttheendofthechapterIwillprovidea link to anexample for aNode.js streampipeline thatmigratesdata fromMongoDBtoCouchDB/PouchDB.

Thefirststream

Building streamscanbeabit tricky sometimes.Westartbycreating the filestream-

example.jsintherootdirofourmodule.

OurfirstiterationwilloutputourCSVcontentstostdout.Wewilluseittolearnhow

streamsworkandbuildourimporterontopofitlater.TodevelopweneedaCSVfile,wecansaveittotest/fixtures/test.csv:

time;location

march;austin,us

april;boston,us

october;bristol,uk

february;hermigua,es

march;hermigua,es

april;havana,cu

Luckilywedon’thavetowriteourownstreamingCSVparser:

$npminstall--savecsv-parse

Werequirethefs,andutilmodule.Theutilmoduleisusedtoinheritfromthebase

objectTransform.ThefsmoduleisneededtoreadtheCSVfilefromdisk.Wealsoneed

torequiretheCSVparser:

constparse=require('csv-parse');

constfs=require('fs');

constTransform=require('stream').Transform;

constutil=require('util');

OurcustomstreamcalledMyTransformStreaminheritsfromstream.Transform.

WesetthestreamintoobjectModetobeabletoprocesstheJSONinputfromtheCSV

parser:

functionMyTransformStream(){

Transform.call(this,{

objectMode:true

});

}

util.inherits(MyTransformStream,Transform);

ATransformstreamhastoimplementonemethod:_transform.Themethodiscalled

for every chunk of data that we are processing. In the_transform method we can

transformthechunkstosomethingnew.Thetransformeddataisthenpushedtothenextconsumerusingthis.push.Oncewearefinishedwecall thedonecallbacktosignal

thatwearefinishedwiththischunk.Rightnowwejustwanttotakealookhowachunklookslike:

MyTransformStream.prototype._transform=transform;

functiontransform(chunk,encoding,done){

console.log('chunk:',chunk);

this.push(chunk);

done();

}

AslaststepwehavetopipetheCSVfileintotheCSVparserandtheparsedoutputintoourcustomstream:

constopts={comment:'#',delimiter:';',columns:true};

constparser=parse(opts);

constinput=fs.createReadStream(__dirname+'/test/fixtures/test.csv');

input

.pipe(parser)

.pipe(newMyTransformStream());

Whenwenowrunnodestreams-example.jswegetthisoutput:

$nodestreams-example.js

chunk:{time:'march',location:'austin,us'}

chunk:{time:'april',location:'boston,us'}

chunk:{time:'october',location:'bristol,uk'}

chunk:{time:'february',location:'hermigua,es'}

chunk:{time:'march',location:'hermigua,es'}

chunk:{time:'april',location:'havana,cu'}

Every chunk is a JSON object and every time we calldone and if there is still input

produced,_transform iscalledwith thenextchunk.Wecould takeeverychunkand

post it against the CouchDB / HTTPAPI now.Wewould keep ourmemory footprintsuperlow,butwewouldalsosendalotofHTTPrequestsandthewholemigrationwould

takeverylong.AhealthycompromiseistobufferafewchunksandpostthemagainstthebulkAPIsofCouchDB/PouchDB.Thiswaywedon’tbufferallexistingdataandrunoutofmemoryandwearefinishedearlierwithourimport,aswedon’thavetosendsomanyHTTPrequests.

The code for this section can be found atsourcecode/streams/streams-

example.js.

TheTransformandWriteablestreamForournext streamwewillcreate the filestreams-bulk-example.js in theroot

directoryoflounger.ItwilltaketheobjectsfromtheCSVparsingstreamandbufferthem.Atagivenlimititwillpassthebufferedobjectstothenextconsumer.Theresultpassedtothenextconsumer is ready togetpostedagainst theCouchDB/PouchDBbulkdocsAPIendpoint. The CouchDB/PouchDB bulk API accepts an array of objects wrapped with{"docs":[]}.

Westartwiththesamesetofmodules:

constparse=require('csv-parse');

constfs=require('fs');

constTransform=require('stream').Transform;

constutil=require('util');

ThenameofourstreamwillbeTransformToBulkDocsanditwilltakeoptionsasan

object.Usingtheoptionswecanspecifytheamountofdocumentstobuffer:

functionTransformToBulkDocs(options){

if(!options){

options={};

}

if(!options.bufferedDocCount){

options.bufferedDocCount=200;

}

Theemptyarrayforthis.bufferwillbeourbuffer:

Transform.call(this,{

objectMode:true

});

this.buffer=[];

this.bufferedDocCount=options.bufferedDocCount;

}

util.inherits(TransformToBulkDocs,Transform);

Hereisthewholeconstructorfunction:

functionTransformToBulkDocs(options){

if(!options){

options={};

}

if(!options.bufferedDocCount){

options.bufferedDocCount=200;

}

Transform.call(this,{

objectMode:true

});

this.buffer=[];

this.bufferedDocCount=options.bufferedDocCount;

}

util.inherits(TransformToBulkDocs,Transform);

Inthe_transformmethodweaddeverychunktoourbuffer:

TransformToBulkDocs.prototype._transform=transform;

functiontransform(chunk,encoding,done){

this.buffer.push(chunk);

Ifthebufferhasgrownbigenoughwecallthemethodthis.push,whichweinherited

from the baseTransform stream.this.push(args) tellsNode thatwewant topass

argstothenextconsumerinourstreampipeline.Wethenemptythebufferfornewdata

thatmightarrive:

if(this.buffer.length>=this.bufferedDocCount){

this.push({docs:this.buffer});

this.buffer=[];

}

done();

}

Hereisthewhole_transformmethod:

TransformToBulkDocs.prototype._transform=transform;

functiontransform(chunk,encoding,done){

this.buffer.push(chunk);

if(this.buffer.length>=this.bufferedDocCount){

this.push({docs:this.buffer});

this.buffer=[];

}

done();

}

Thelastpartofourfileisalmostidenticaltoourfirststreamsexample:

constopts={comment:'#',delimiter:';',columns:true};

constparser=parse(opts);

constinput=fs.createReadStream(__dirname+'/test/fixtures/test.csv');

input

.pipe(parser)

.pipe(newTransformToBulkDocs());

Weaddatemporaryconsole.logtoourcodetoseeifitworks:

if(this.buffer.length>=this.bufferedDocCount){

this.push({docs:this.buffer});

console.log({docs:this.buffer});

this.buffer=[];

}

When we runnodestreams-bulk-example.js we see that… it doesn’t work!

Why?

Theproblemisthatwedon’thaveenoughdocumentstoreachthedefaultdocumentcountof 200. The same applies to the remaining documents of a set. If we have 250 initialdocuments as input, the first 200 hundreds are pushed to the next consumer, but theremaining 50 are lost. Luckily the Node.js developers were aware of the problem andprovided the_flushmethod.Themethod doesn’t have to be implemented tomake a

Transform stream work, like the_transform method. Instead we can chose to

implementit,givenweneedit.

The_flushmethodwillgetcalledat theveryendafteralldatawasconsumedby the

stream, but before the stream emits theend eventwhich signals the endof the stream.

_flushwillgetcalledattheveryendandgivenwestillhaveafewbuffereddocuments,

wepushthemtothenextconsumer:

TransformToBulkDocs.prototype._flush=flush;

functionflush(done){

this.buffer.length&&this.push({docs:this.buffer});

done();

}

That’sourfirstcustomstream!Don’tforgettoremovetheconsole.logcallweadded!

Thenextconsumerinourpipelinewilltakethecollecteddocumentsandpostthemagainstthe CouchDB / PouchDB bulk docs endpoint. As the streams are able to handlebackpressure, the other streamswillwait untilwe successfully added the documents toCouchDB/PouchDB.Theywillcontinuetopassusdatadownthepipelineonceweareabletopullinthenextcollectionofdocuments.

Thenextstreamwewillbuildaccepts thedata fromtheTransformstreamandwrites itintothedatabase.ItisaWriteablestreamandwewilladdittoourstreams-bulk-

example.jsfile:

constWritable=require('stream').Writable;

OurWritablestreamneedstoknowwheretoputthedata,sowewillneedtopassitthedatabaseurl.Asthemethodsofthestreamarecalledforeachchunkwehavetostorethepassedurlasthis.url:

functionCouchBulkImporter(options){

if(!options){

options={};

}

if(!options.url){

constmsg=[

'options.urlmustbeset',

'example:',

"newCouchBulkImporter({url:'http://localhost:5984/baseball'})"

].join('\n')

thrownewError(msg);

}

Writable.call(this,{

objectMode:true

})

//sanitiseurl,removetrailingslash

this.url=options.url.replace(/\/$/,'');

}

util.inherits(CouchBulkImporter,Writable);

ToimplementachildofaWritablestream,wehavetoimplementthe_writemethod.

Like the_transformmethodof theTransformstream, the_writemethod iscalled

foreverychunkthat ispassed to thestreamfromthepreviousproducer. Inourcasewesend the JSON chunks as JSON to the database using request. After we sent the datasuccessful to the/_bulk_docsAPIendpointwecall thedonecallbacktosignalthat

wearereadyforanewchunk:

CouchBulkImporter.prototype._write=write;

functionwrite(chunk,enc,done){

request({

json:true,

uri:this.url+'/_bulk_docs',

method:'POST',

body:chunk

},function(err,res,body){

if(err){

returndone(err);

}

if(!/^2../.test(res.statusCode)){

constmsg='CouchDBserveranswered:\nStatus:'+

res.statusCode+'\nBody:'+JSON.stringify(body);

returndone(newError(msg));

}

done();

});

}

Wealsohavetorequirerequestinourfile:

constrequest=require('request');

To use the stream we have to pipe the data into it. We update the last section ofstreams-bulk-example.js:

input

.pipe(parser)

.pipe(newTransformToBulkDocs())

.pipe(newCouchBulkImporter({url:'http://127.0.0.1:5984/travel'}));

Afterwecreatedthedatabasetravelandrunourscript,wehaveimportedtheCSV:

$curl-XPUThttp://localhost:5984/travel

{"ok":true}

$nodestreams-bulk-example.js

$curlhttp://localhost:5984/travel/_all_docs

{"total_rows":6,"offset":0,"rows":[{"id":"3444bf7c-65c0-438f-f8e8-

7f55124f1736","key":"3444bf7c-65c0-438f-f8e8-7f55124f1736","value":{"rev":"1-

37fcd2e5b40939805b8e043da44f9b1d"}},{"id":"568c3b0f-78fe-43a2-9eac-

06620cfaa595","key":"568c3b0f-78fe-43a2-9eac-06620cfaa595","value":{"rev":"1-

e047788bac9aada0564fc928642d3960"}},{"id":"5d170e63-d845-4e45-f760-

97e30cbc4b21","key":"5d170e63-d845-4e45-f760-97e30cbc4b21","value":{"rev":"1-

6f1794e4eb24b60665fe02c2624d53eb"}},{"id":"9cd34a61-8a48-4b88-afbf-

7fd6e3c9cf42","key":"9cd34a61-8a48-4b88-afbf-7fd6e3c9cf42","value":{"rev":"1-

747b11a103cc0fe31d1c546ef70d69c0"}},{"id":"cc7da504-438a-40c1-842b-

4722b63c9a37","key":"cc7da504-438a-40c1-842b-4722b63c9a37","value":{"rev":"1-

0f9e7add8b7fbb773dc7d3081475d855"}},{"id":"f09811c5-43ec-4a6d-bd3b-

b34b3674676d","key":"f09811c5-43ec-4a6d-bd3b-b34b3674676d","value":{"rev":"1-

d677c5bbbee1bbbe45f6128a9cf1fe8d"}}]}

Wecanaccessasingledocumentusingtheid:

$curlhttp://localhost:5984/travel/3444bf7c-65c0-438f-f8e8-7f55124f1736

{"time":"february","location":"hermigua,es","_id":"3444bf7c-65c0-438f-f8e8-

7f55124f1736","_rev":"1-37fcd2e5b40939805b8e043da44f9b1d"}

Looksgreat!Seemswehaveeverythinginplacetouseourlow-levelstreamingfunctionsin the command line client. The code for this section is located atsourcecode/streams/streams-bulk-example.js.

ThestreamingimportcommandForthelastpartofthischapterwewillreusethecustomstreamimplementationthatwecreated inThe Transform andWriteable stream. In the real world I would create twomodulesforourtwostreamstomakethemreusableacrossmultipleprojects,butfornowwe can copy the code for theCouchBulkImporter and the

TransformToBulkDocs streams intolib/csv.jswhichwill be the homeof our

importcommand:

constparse=require('csv-parse');

constfs=require('fs');

constTransform=require('stream').Transform;

constutil=require('util');

constWritable=require('stream').Writable;

constrequest=require('request');

constlounger=require('./lounger.js');

functionTransformToBulkDocs(options){

if(!options){

options={};

}

if(!options.bufferedDocCount){

options.bufferedDocCount=200;

}

Transform.call(this,{

objectMode:true

});

this.buffer=[];

this.bufferedDocCount=options.bufferedDocCount;

}

util.inherits(TransformToBulkDocs,Transform);

TransformToBulkDocs.prototype._transform=transform;

functiontransform(chunk,encoding,done){

this.buffer.push(chunk);

if(this.buffer.length>=this.bufferedDocCount){

this.push({docs:this.buffer});

this.buffer=[];

}

done();

}

TransformToBulkDocs.prototype._flush=flush;

functionflush(done){

this.buffer.length&&this.push({docs:this.buffer});

done();

}

functionCouchBulkImporter(options){

if(!options){

options={};

}

if(!options.url){

constmsg=[

'options.urlmustbeset',

'example:',

"newCouchBulkImporter({url:'http://localhost:5984/baseball'})"

].join('\n')

thrownewError(msg);

}

Writable.call(this,{

objectMode:true

})

//sanitiseurl,removetrailingslash

this.url=options.url.replace(/\/$/,'');

}

util.inherits(CouchBulkImporter,Writable);

CouchBulkImporter.prototype._write=write;

functionwrite(chunk,enc,done){

request({

json:true,

uri:this.url+'/_bulk_docs',

method:'POST',

body:chunk

},function(err,res,body){

if(err){

returndone(err);

}

if(!/^2../.test(res.statusCode)){

constmsg='CouchDBserveranswered:\nStatus:'+

res.statusCode+'\nBody:'+JSON.stringify(body);

returndone(newError(msg));

}

done();

});

}

Let’s think a bit about the commandwe are going to build.ACSV can have differentdelimiters, some use semicolons as a delimiter, others are using commas or tabs. The

symbolstodenoteacommentcanalsochange.Inourpreviousimplementationsweusedfixedvalues:

constopts={comment:'#',delimiter:';',columns:true};

For a real world use case the symbols for the delimiter and comment must beconfigurable.TheCSVinputisusuallyafile.

HereisapossibleCLI:

$loungercsvtransfer<file><database>[--delimiter=;][--comment=#][--

chunksize=200]

The commandcsv isopen toextensionandcanhostallCSVrelatedcommands in the

future. The command reads quite nicely and is easy to remember:lounger csv

transfer <file> <database> reads almost aslounger [do] csv

transfer[from]<file>[to]<database>.Sanedefaultshelpus toavoid

passingoptionalmodifiersatall,butincaseweneedtomodifythemwecanchangeeveryimportantaspectofourimport.

I’mnotsureifyounoticedit,butwhenweplayedwithourstreamswewouldhavehadtocreatethetargetdatabaseusingcurlinadvance.ItwouldbehandyifourCLIusersdon’thavetocreatethetargetdatabaseontheirown.Ourgoalistohelpthemtosolvetheirtaskas quickly and easily as possible, so we should automatically create databases asnecessary.

The functioncreateTargetDatabase is a helper functionwhichwrapsrequest

intoaPromise.Ifthedatabasewascreated(HTTPcode201or200)orthedatabaseexistsalready(HTTPcode412)weresolve,allotherstatesleadtorejectionofthePromise:

functioncreateTargetDatabase(url){

returnnewPromise((resolve,reject)=>{

request({

json:true,

uri:url,

method:'PUT',

body:{}

},function(er,res,body){

if(er&&(er.code==='ECONNREFUSED'||er.code==='ENOTFOUND')){

consterr=newError(

'Couldnotconnectto'+url+'.Pleasecheckifthedatabaseis

offline'

);

err.type='EUSAGE';

returnreject(err);

}

if(er){

returnreject(er);

}

constcode=res.statusCode;

if(code!==200&&code!==201&&code!==412){

constmsg='CouchDBserveranswered:\nStatus:'+

res.statusCode+'\nBody:'+JSON.stringify(body);

returnreject(newError(msg));

}

resolve();

});

});

}

In case of anECONNREFUSED orENOTFOUND error we can safely assume that the

databaseiscurrentlyofflineandasktheusertotakealookifthedatabaseisavailable.Ican’tstressenoughhowimportantpropererrorhandlingis.Takethisexample,wherewearegettingbackECONNREFUSED:

$./bin/lounger-clicsvtransfertest/fixtures/test.csv

http://127.0.0.1:1337/testimport

ERR!connectECONNREFUSED127.0.0.1:5984

ERR!Error:connectECONNREFUSED127.0.0.1:5984

ERR!atObject.exports._errnoException(util.js:870:11)

ERR!atexports._exceptionWithHostPort(util.js:893:20)

ERR!atTCPConnectWrap.afterConnect[asoncomplete](net.js:1063:14)

ERR!

ERR!

ERR!lounger:1.0.0node:v4.2.4

ERR!pleaseopenanissueincludingthislogon

http://example.com/lounger/issues

Depending on howmuch our users usedNode.js before theywould be very puzzled Iguess.Theonlywaytocontinueforthemwouldbetoaskasearchengineortoopenanissue.After receiving the issueourboring jobwouldbe toclose the issueand tell themthattheyprobablyhadatypointheirurl.

After writing thecreateTargetDatabase function we should have all our

supporting functions in place. As usual we are starting to implement the main CLIfunctions by implementing the API command which we will then wrap with our CLIfunction.

The delimiter and comment options are defined in the config file or are passed on thecommand line. To know what their values are, we have to interact withlounger.config.Toaccesslounger.configwehavetorequireit:

constlounger=require('./lounger.js');

The main API function checks if all necessary arguments were provided and appliesdefaultsifnoconfigurationwaspassedinfromtheconfigfileoronthecommandline.Wecreate the database in case it does not exist yet and delegate to the helper functionimportFromCsvFile:

exports.api={

transfer:bulkdocsImport

};

functionbulkdocsImport(file,targetDb){

returnnewPromise((resolve,reject)=>{

constopts={};

if(!file&&!targetDb){

returnreject(newError('fileand/ortargetDbargumentmissing'));

}

opts.delimiter=lounger.config.get('delimiter')||';';

opts.comment=lounger.config.get('comment')||'#';

opts.chunksize=lounger.config.get('chunksize')||200;

createTargetDatabase(targetDb)

.then(()=>{

returnimportFromCsvFile(file,targetDb,opts);

}).catch(reject);

});

}

importFromCsvFile accepts the source CSV file, url and options and creates the

streampipeline.Themaindifferencetoourpreviouscodeinstreams-example.jsis

thatwehavepropererrorhandlinginplacetocatchallerrors.

functionimportFromCsvFile(file,url,opts){

returnnewPromise((resolve,reject)=>{

constoptions={comment:opts.comment,delimiter:opts.delimiter,

columns:true};

constparser=parse(options);

constinput=fs.createReadStream(file);

input

.pipe(parser)

.on('error',reject)

.pipe(newTransformToBulkDocs({bufferedDocCount:opts.chunksize}))

.on('error',reject)

.pipe(newCouchBulkImporter({url:url}))

.on('error',reject);

});

}

TheCLIfunctionsfinallywrapsourAPImethodandaddsfriendlyerrormessages:

exports.cli=importCli;

functionimportCli(cmd,file,target){

returnnewPromise((resolve,reject)=>{

if(!cmd||cmd!=='transfer'||!file||!target){

consterr=newError(

'Usage:loungercsvtransfer<file><database>[--delimiter=;][--

comment=#][--chunksize=200]'

);

err.type='EUSAGE';

returnreject(err);

}

returnbulkdocsImport(file,target).catch(reject);

});

}

We introduced three new options,delimiter, comment andchunksize.

lounger.config enables our users to set default values using the config file. In

addition we have to take care that the options are parsed on the command line. Inbin/lounger-cliwehavetoregisterouroptionalargumentstonopt:

constparsed=nopt({

'json':[Boolean],

'delimiter':[String],

'comment':[String],

'chunksize':[Number]

},{'j':'--json'},process.argv,2);

That’s it!We created a command that is able to stream large amounts of data into ourdatabase.

If you are interested in a stream pipelinewhichwould stream data fromMongoDB toCouchDB you can take a look athttps://github.com/robertkowalski/couchbulkimporter/blob/master/examples/mongo.js.

Thecodeforthissectionisatsourcecode/streams,enjoy! �

TIPS&TRICKS

ThissectioncollectssometipsandtricksregardingNode.jsdevelopmentingeneral.

TestingWith proper unit and integration tests in place, ensure new features or bugfixes don’tintroduce regressions. There are a lot of great services that provide a hosted CIenvironment. They can test every Pull Request before it is merged, which makesreviewingcodealoteasier.ApopularserviceforhostedCIisTravisCI.TravisCIisfreeforOpenSourceprojects.

SemanticVersioningwithSemVerI would recommend to follow Semantic Versioning with SemVer (http://semver.org/).SemVer divides the version number of a release into three areas:MAJOR.MINOR.PATCH.Theversion3.5.8wouldhave3asMAJORversionlevel,5asMINOR version level and 8 as patchlevel. Given a new release includes a breakingchange,theMAJORversionnumberisbumped.AnewfeaturewouldjustneedaminorversionbumpandbugfixeswouldjustrequireabumpofthePATCHsection.

Example:Mypackage has version 3.5.7 and I add a new featurewhich does not breakbackwardscompatibility.Mynextreleasewouldbe3.6.0.

Thiswayyourusersgetanideaifareleasemightbreaktheirproductioncode(MAJOR),it contains a new feature (MINOR)or a bug fix (PATCH).Agreat tool to help you tomake the right decision for the next version bump is semantic-release(https://www.npmjs.com/package/semantic-release).

GreenkeeperKeeping track which dependencies of your project got a new version and need to getupdated can be tedious. The update itself (bumping the version number in thepackage.json) is not the most interesting task on earth, too. A new and exciting

service ishttp://greenkeeper.io.Once you registered it for your project itwill send youpullrequestswithupdatedversionsofyourdependencies.Ifyouhaveatestsuiteinplaceandeverythingis“green“youjusthavetomergethePullRequestfromtheGreenkeeper

bot.TestingandaCIServicethatautomaticallyrunsthetests,SemVerandGreenkeeperreallyshowtheirstrengthswhencombinedtogether. �

THEEND

IhopeyouenjoyedTheCLIBook!

OurjourneytosuccessfulCLIsdoesn’tendhere,itjustbegins.

I am very happy about feedback. Please send and feedback or corrections totheclibook@kowalski.gd. You can also contact me on twitter:@robinson_k – pleaserecommendthebooktoothersincaseyoulikeit. �